---
license: apache-2.0
language:
- en
- id
base_model: Qwen/Qwen3.5-4B-Base
tags:
- continued-pretraining
- plant-disease
- agriculture
- domain-adaptation
private: true
---

# Qwen3.5-4B-Base Continued Pretraining — Plant Disease Domain

Continued pretraining of [Qwen/Qwen3.5-4B-Base](https://huggingface.co/Qwen/Qwen3.5-4B-Base) on a curated corpus of plant disease and agricultural literature.

## Training Data

| Source | Documents | Tokens | Description |
|--------|-----------|--------|-------------|
| Journal Articles (PDF) | 4,159 | ~78M | Peer-reviewed plant pathology |
| Wikipedia Knowledge Graph | 22,452 | ~47M | Plant disease + related topics |
| BioProtection Portal (CABI) | 105 | ~200K | Curated IPM guides |
| Plant Disease Index | 4,374 | structured | Master taxonomy facts |

**Total:** ~125M tokens across {TOTAL_DOCS} documents

## Training Configuration

- **Base model:** Qwen3.5-4B-Base
- **Architecture:** Hybrid GatedDeltaNet + Attention (32 layers, 2560 dim)
- **Precision:** bfloat16
- **Optimizer:** 8-bit AdamW (lr=2e-5, cosine schedule, 5% warmup)
- **Effective batch size:** 4 (batch=1 × grad_accum=4)
- **Sequence length:** 2048
- **Gradient checkpointing:** enabled
- **Hardware:** NVIDIA A100 80GB

## Evaluation

- **Final validation PPL:** {FINAL_VAL_PPL}
- **Best validation PPL:** {BEST_VAL_PPL} at step {BEST_STEP}

## Domain Coverage

- Plant pathology (fungal, bacterial, viral diseases)
- Crop protection and IPM
- Pathogen biology and epidemiology
- Agricultural practices
- Biocontrol agents

## Limitations

- Domain-adapted base model (NOT instruction-tuned)
- Bilingual (English majority + some Indonesian)
- Best used as foundation for further SFT
- Knowledge cutoff: training data sources

## Citation

This model was created as part of the MioLineara plant disease research project.

---

*Trained: 2026-05-09*