deadbear34/qwen35-4b-plantdisease-cpt

Qwen3.5-4B-Base Continued Pretraining — Plant Disease Domain

Continued pretraining of Qwen/Qwen3.5-4B-Base on a curated corpus of plant disease and agricultural literature.

Training Data

Source	Documents	Tokens	Description
Journal Articles (PDF)	4,159	~78M	Peer-reviewed plant pathology
Wikipedia Knowledge Graph	22,452	~47M	Plant disease + related topics
BioProtection Portal (CABI)	105	~200K	Curated IPM guides
Plant Disease Index	4,374	structured	Master taxonomy facts

Total: ~125M tokens across {TOTAL_DOCS} documents

Training Configuration

• Base model: Qwen3.5-4B-Base
• Architecture: Hybrid GatedDeltaNet + Attention (32 layers, 2560 dim)
• Precision: bfloat16
• Optimizer: 8-bit AdamW (lr=2e-5, cosine schedule, 5% warmup)
• Effective batch size: 4 (batch=1 × grad_accum=4)
• Sequence length: 2048
• Gradient checkpointing: enabled
• Hardware: NVIDIA A100 80GB

Evaluation

• Final validation PPL: {FINAL_VAL_PPL}
• Best validation PPL: {BEST_VAL_PPL} at step {BEST_STEP}

Domain Coverage

• Plant pathology (fungal, bacterial, viral diseases)
• Crop protection and IPM
• Pathogen biology and epidemiology
• Agricultural practices
• Biocontrol agents

Limitations

• Domain-adapted base model (NOT instruction-tuned)
• Bilingual (English majority + some Indonesian)
• Best used as foundation for further SFT
• Knowledge cutoff: training data sources

Citation

This model was created as part of the MioLineara plant disease research project.

---

Trained: 2026-05-09