---
base_model: google/gemma-3-4b-it
library_name: peft
pipeline_tag: image-text-to-text
license: gemma
tags:
- gemma3
- vlm
- lora
- peft
- screenshot-to-code
- react
- code-generation
language:
- en
---

# gemma-3-4b-it-vlm-react-screenshot-to-code

A LoRA adapter for **`google/gemma-3-4b-it`** (Gemma 3 4B IT, vision-language) that turns a
**screenshot of a UI into React component code**.

Give it an image of an interface; it returns a single self-contained React component that
reproduces the layout.

👉 **Live demo:** [Reubencf/react-screenshot-to-code](https://huggingface.co/spaces/Reubencf/react-screenshot-to-code)

## Model details

| | |
|---|---|
| Base model | `google/gemma-3-4b-it` (`Gemma3ForConditionalGeneration`) |
| Adapter type | LoRA (PEFT `0.15.1`) |
| Rank / alpha | `r=16`, `lora_alpha=32`, `lora_dropout=0.0` |
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
| Adapted layers | Language model self-attention only, all 34 decoder layers (272 tensors) |
| Frozen | Full SigLIP vision tower and multimodal projector (listed in `exclude_modules`) |
| Adapter size | ~34 MB (bf16) |
| Precision | bfloat16 |

Only the language model's attention projections are adapted — the vision encoder is untouched,
so the adapter changes *how the model writes code*, not how it sees images.

> **Note on the base model.** The training run recorded its base as
> `togethercomputer/gemma-3-4b-it-VLM`, which is not a resolvable Hub repo. The bundled
> `config.json` matches `google/gemma-3-4b-it` exactly (hidden 2560, 34 text layers,
> SigLIP-27 @ 896px, vocab 262208), so `base_model_name_or_path` has been corrected to point
> there. `unsloth/gemma-3-4b-it` is an ungated mirror of the same model if you'd rather not
> request access.

## Training

Short LoRA run, recorded in `trainer_state.json`:

| | |
|---|---|
| Epochs | 1 |
| Steps | 26 (`max_steps=26`) |
| Train batch size | 1 |
| Logging / eval every | 1 / 5 steps |
| Total FLOPs | 1.44e17 |
| Start train loss | 0.986 |
| Final eval loss | **0.715** |

This is a **light adaptation, not a full fine-tune** — 26 steps at batch size 1. Expect it to
nudge output format and style toward React rather than to teach new visual reasoning. Treat
generations as a starting scaffold to edit, not production code.

## Usage

Requires `transformers >= 5.10.1` — the adapter keys use the refactored
`model.language_model.layers.*` module layout, and will not match older checkpoints.

```python
import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from peft import PeftModel
from PIL import Image

BASE = "google/gemma-3-4b-it"          # or "unsloth/gemma-3-4b-it" (ungated mirror)
ADAPTER = "Reubencf/gemma-3-4b-it-vlm-react-screenshot-to-code"

processor = AutoProcessor.from_pretrained(BASE)
model = Gemma3ForConditionalGeneration.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda"
)
model = PeftModel.from_pretrained(model, ADAPTER).merge_and_unload().eval()

image = Image.open("screenshot.png").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text":
         "Convert this screenshot into a single self-contained React component. "
         "Use Tailwind CSS classes for styling. Return only code."},
    ],
}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device, dtype=torch.bfloat16)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=1536, do_sample=True,
                         temperature=0.7, top_p=0.95)

print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
```

## Prompt format

Standard Gemma 3 chat template (`chat_template.jinja` is included). Images are inserted as
`<start_of_image>` and expand to 256 soft tokens each at 896×896.

```
<bos><start_of_turn>user
<start_of_image>Convert this screenshot into a React component.<end_of_turn>
<start_of_turn>model
```

## Limitations

- **26 training steps.** Output quality is closer to the base model than to a task-specialised one.
- Single-screenshot, single-component. No routing, state management, or multi-page flows.
- Colours, spacing, and font sizes are approximated from the image — expect to tune them.
- Text in the screenshot is transcribed by the vision encoder and can be misread at small sizes.
- Generated code is unverified. Review it before running, as you would any model output.
- Inherits all limitations and the licence of the Gemma 3 base model.

## Licence

Governed by the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). The adapter weights are
a derivative of Gemma 3 and are subject to the same terms.

### Framework versions

- PEFT 0.15.1
- Transformers 5.10.1