| --- |
| language: en |
| license: mit |
| pipeline_tag: text-generation |
| tags: |
| - medical |
| - gpt-neo |
| - lora |
| - peft |
| - pubmed |
| - instruction-tuned |
| datasets: |
| - ccdv/pubmed-summarization |
| base_model: EleutherAI/gpt-neo-125M |
| --- |
| |
| # GPT-Neo 125M Medical Instruction-Tuned Model |
|
|
| ## Model Overview |
|
|
| This model is an instruction-conditioned medical text generator built on top of **EleutherAI/gpt-neo-125M**. |
|
|
| It was fine-tuned using **LoRA (Low-Rank Adaptation)** with prompt-formatted biomedical abstracts from the PubMed Summarization dataset. |
|
|
| Unlike standard fine-tuned models, this version was trained using structured prompts to improve domain-specific generation quality. |
|
|
| --- |
|
|
| ## Key Improvements |
|
|
| Compared to the base fine-tuned version: |
|
|
| - Larger context window (512 tokens) |
| - Instruction-style prompt formatting |
| - Enhanced LoRA configuration (r=16) |
| - Reduced hallucination via controlled decoding |
| - Improved generation coherence for medical narratives |
|
|
| --- |
|
|
| ## Intended Use |
|
|
| This model is designed for: |
|
|
| - Medical text generation |
| - Biomedical explanation drafting |
| - Research prototyping |
| - Educational demonstrations |
| - NLP experimentation in healthcare |
|
|
| ⚠️ This model is **NOT intended for clinical use**. |
|
|
| --- |
|
|
| ## Training Details |
|
|
| | Item | Value | |
| |------|-------| |
| | Base Model | EleutherAI/gpt-neo-125M | |
| | Dataset | PubMed Summarization | |
| | Training Method | LoRA | |
| | Prompt Conditioning | Yes | |
| | Context Length | 512 | |
| | LoRA Rank | 16 | |
| | Task | Instruction-based Medical Text Generation | |
|
|
| --- |
|
|
| ## Dataset |
|
|
| Training utilized biomedical abstracts from: |
|
|
| https://huggingface.co/datasets/ccdv/pubmed-summarization |
|
|
| Prompt formatting was applied: |
|
|
| Medical report: |
| <Abstract text> |
|
|
| This improves alignment with generation tasks rather than summarization. |
|
|
| --- |
|
|
| ## Training Strategy |
|
|
| - Base model weights frozen |
| - LoRA adapters applied to attention layers |
| - Prompt-based conditioning introduced |
| - Controlled decoding parameters used during inference |
|
|
| This enables: |
|
|
| - Efficient training |
| - Low memory footprint |
| - Domain-aligned generation |
|
|
| LoRA Paper: |
| https://arxiv.org/abs/2106.09685 |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - May generate plausible but incorrect medical statements |
| - Not trained on clinical decision datasets |
| - May struggle with rare diseases |
| - No real-time knowledge updates |
|
|
| Users must verify outputs using trusted medical sources. |
|
|
| --- |
|
|
| ## Ethical Considerations |
|
|
| Allowed Uses: |
|
|
| ✔️ Research |
| ✔️ Academic projects |
| ✔️ NLP experimentation |
|
|
| Disallowed Uses: |
|
|
| ❌ Clinical decision support |
| ❌ Medical diagnosis |
| ❌ Treatment planning |
| ❌ Emergency healthcare guidance |
|
|
| This model does not replace medical professionals. |
|
|
| --- |
|
|
| ## Future Work |
|
|
| - Add ROUGE / BLEU evaluation |
| - Compare against BioGPT / ClinicalT5 |
| - Improve safety alignment |
| - Add hallucination detection layer |
| - Extend to clinical-style datasets |
|
|
| --- |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| model = AutoModelForCausalLM.from_pretrained("omniamagdy/gptneo-medical-125m") |
| tokenizer = AutoTokenizer.from_pretrained("omniamagdy/gptneo-medical-125m") |
| |
| prompt = "Medical report:\nExplain hypertension" |
| |
| inputs = tokenizer(prompt, return_tensors="pt") |
| |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=120, |
| temperature=0.6, |
| top_p=0.9, |
| repetition_penalty=1.2 |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| |
