---
library_name: peft
license: apache-2.0
base_model: Qwen/Qwen2.5-7B-Instruct
tags:
  - condensed-matter-physics
  - graphene
  - domain-specific
  - qdora
  - chain-of-thought
  - fine-tuning
  - scientific-ai
  - base_model:adapter:Qwen/Qwen2.5-7B-Instruct
  - lora
  - sft
  - transformers
  - trl
language:
  - en
pipeline_tag: text-generation
---

# Qwen2.5-7B-GrapheneContact-QDoRA-CoT

A domain-specific language model fine-tuned for condensed matter physics, specifically graphene–metal contact resistance research. This 7B model, trained on only **64 question–answer pairs**, outperforms a 235B-parameter model with open-book access under closed-book evaluation conditions.

## Key Result

| Model | Parameters | Condition | Correct (✓) | Partial (△) | Wrong (✗) | Accuracy |
|-------|-----------|-----------|-------------|-------------|-----------|----------|
| Base 7B | 7B | Open-book | 9 | 17 | 4 | 30% |
| **FT 7B (this model)** | **7B** | **Closed-book** | **22** | **5** | **3** | **73%** |
| Qwen3-235B | 235B | Open-book | 18 | 9 | 3 | 60% |

> A fine-tuned 7B model with no access to the source paper outperforms a 33× larger model that receives the paper's content as a prompt.

## Model Details

### Model Description

- **Developed by:** Junsu Park
- **Affiliation:** Department of Physics and Photon Science, Gwangju Institute of Science and Technology (GIST), Korea
- **Base model:** [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct)
- **Fine-tuning method:** QDoRA (Quantized Weight-Decomposed Low-Rank Adaptation) + Chain-of-Thought reasoning
- **Language:** English
- **License:** Apache 2.0
- **Domain:** Condensed matter physics — graphene–metal contact resistance (TLCD/TECD devices)

### Model Sources

- **Repository:** [GitHub](https://github.com/pjspjs0987) *(coming soon)*
- **Research Article:** "Small Data, Small Model, Superior Domain Accuracy: Fine-Tuning a 7B Language Model as a Condensed Matter Physics Expert with 64 Examples"(https://www.preprints.org/manuscript/202603.1691)


## Uses

### Direct Use

This model is designed to answer domain-specific questions about graphene–metal contact resistance, including device structure (TLCD/TECD), fabrication processes, theoretical models (Landauer, BTH), and experimental results. It serves as a personalized research assistant for this specific domain.

### Out-of-Scope Use

This model is trained on a single research paper and should not be used for general scientific reasoning, other physics domains, or any task outside graphene–metal contact resistance research.

## Training Details

### Training Data

64 expert-crafted question–answer pairs from a single condensed matter physics paper, organized into five categories:

| Category | Description |
|----------|-------------|
| Structure / Geometry | TLCD vs TECD device structure, contact dimensions |
| Theory / Mechanism | Landauer model, BTH model, current crowding |
| Numerical / Experimental | Contact resistance values, TLM measurements |
| Fabrication | Metal-on-bottom process, Cr/Au deposition |
| Novelty / Contribution | First experimental verification of area-dependent Rc |

Each answer includes Chain-of-Thought (CoT) reasoning with `<think>` and `<final>` tags.

### Training Hyperparameters

| Parameter | Value |
|-----------|-------|
| Adapter | QDoRA (`use_dora=True`) |
| Rank (r) | 64 |
| Alpha (α) | 128 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Quantization | 4-bit NF4, double quantization, bfloat16 compute |
| Max steps | 80 |
| Learning rate | 5×10⁻⁴ |
| Batch size | 1 (gradient accumulation = 8, effective batch = 8) |
| Max sequence length | 2048 |
| Optimizer | paged_adamw_8bit |
| Training time | ~20 minutes |
| Training regime | bf16 mixed precision |

### Hardware

| Component | Specification |
|-----------|--------------|
| GPU | NVIDIA RTX 5090 (32 GB VRAM) |
| CPU | AMD Threadripper PRO 5975WX (32 cores) |
| RAM | 256 GB DDR4-3200 ECC |
| OS | Ubuntu |

## Evaluation

30 expert-designed questions (distinct from training set) evaluated by the domain expert (first author of the source paper) using a three-tier rubric: ✓ (correct), △ (partial), ✗ (wrong).

### Results by Category

| Category | Questions | Base 7B | FT 7B | 235B |
|----------|-----------|---------|-------|------|
| Structure / Geometry | 4 | 1 | 3 | 3 |
| Theory / Mechanism | 10 | 4 | 8 | 7 |
| Numerical / Experimental | 6 | 2 | 5 | 4 |
| Fabrication | 6 | 1 | 4 | 2 |
| Novelty / Contribution | 4 | 1 | 2 | 2 |

### Error Pattern Analysis

Each model exhibits qualitatively distinct failure modes:

- **Base 7B:** Dominated by *partial coverage (PC)* — generates relevant but insufficiently specific responses
- **235B:** Dominated by *over-extrapolation (OE)* — fills knowledge gaps with plausible but incorrect scientific details
- **FT 7B (this model):** Occasional *directional reversal (DR)* — correctly identifies relevant concepts but sometimes inverts their relationships

## How to Get Started with the Model
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "pjspjs0987/Qwen2.5-7B-GrapheneContact-QDoRA-CoT"
)

# Load fine-tuned adapter
model = PeftModel.from_pretrained(
    base_model,
    "pjspjs0987/Qwen2.5-7B-GrapheneContact-QDoRA-CoT"
)

# Inference
messages = [
    {"role": "user", "content": "Why does increasing contact area reduce contact resistance in graphene-metal interfaces?"}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Citation
```bibtex
@article{park2026small,
  title={Small Data, Small Model, Superior Domain Accuracy: Fine-Tuning a 7B Language Model as a Condensed Matter Physics Expert with 64 Examples},
  author={Park, Junsu},
  year={2026},
  institution={Gwangju Institute of Science and Technology (GIST)}
}
```

## Contact

- **Author:** Junsu Park
- **Affiliation:** GIST, Department of Physics and Photon Science
- **Hugging Face:** [pjspjs0987](https://huggingface.co/pjspjs0987)

### Framework versions

- PEFT 0.18.0