| # Phi-3 Mini (LoRA Fine-Tuned on MITRE-STIX-CVE-ExploitDB Dataset) |
|
|
| ## Model Summary |
|
|
| This model is a fine-tuned version of **[microsoft/phi-3-mini-128k-instruct](https://huggingface.co/microsoft/phi-3-mini-128k-instruct)** using **LoRA (Low-Rank Adaptation)** and **8-bit quantization** for parameter-efficient training. |
|
|
| The fine-tuning dataset is **[jason-oneal/mitre-stix-cve-exploitdb-dataset-alpaca](https://huggingface.co/datasets/jason-oneal/mitre-stix-cve-exploitdb-dataset-alpaca)**, which contains security-related instruction-response examples (CVE, STIX, ExploitDB context). |
|
|
| The goal of this model is to act as a **cybersecurity knowledge assistant** that can answer questions about CVEs, exploits, and related security topics. |
|
|
| * **Base Model**: microsoft/phi-3-mini-128k-instruct |
| * **Fine-tuning Method**: LoRA (8-bit PEFT with bitsandbytes) |
| * **Dataset**: jason-oneal/mitre-stix-cve-exploitdb-dataset-alpaca |
| * **Languages**: English |
| * **Context Length**: 128k tokens |
|
|
| --- |
|
|
| ## Intended Uses |
|
|
| * Designed for: cybersecurity Q\&A, reasoning about vulnerabilities, exploits, and threat intelligence. |
| * Can be used for: research, learning, and prototyping of cyber threat assistants. |
|
|
| --- |
|
|
| ## Dataset |
|
|
| * **Name**: MITRE-STIX-CVE-ExploitDB Dataset (Alpaca format) |
| * **Source**: [jason-oneal/mitre-stix-cve-exploitdb-dataset-alpaca](https://huggingface.co/datasets/jason-oneal/mitre-stix-cve-exploitdb-dataset-alpaca) |
| * **Schema**: Instruction–response pairs in Alpaca format |
| * **Size Used**: Up to 5,000 training samples (subset for efficiency) |
|
|
| --- |
|
|
| ## Training Procedure |
|
|
| * **Frameworks**: Hugging Face Transformers, PEFT, bitsandbytes |
| * **Precision**: 8-bit quantization (bnb.int8) + FP16 training |
| * **Optimizer**: AdamW |
| * **Batch Size**: 4 per device |
| * **Epochs**: 1 |
| * **Learning Rate**: 3e-4 |
| * **Warmup Steps**: 50 |
| * **Max Length**: 1024 tokens |
| * **LoRA Config**: |
|
|
| * r = 16 |
| * alpha = 16 |
| * dropout = 0.05 |
| * target modules: q\_proj, k\_proj, v\_proj, o\_proj, w1, w2, dense |
|
|
| --- |
|
|
| ## Evaluation |
|
|
| * **Metric**: Training loss (did not include a validation set in this run). |
| * **Qualitative Evaluation**: The model produces meaningful responses to security-related prompts, but further fine-tuning with eval sets is recommended. |
|
|
| --- |
|
|
| ## How to Use |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline |
| |
| model_name = "sushanrai/phi3-cybersec-advisor-lora" # replace with your repo |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model = AutoModelForCausalLM.from_pretrained(model_name) |
| |
| pipe = pipeline("text-generation", model=model, tokenizer=tokenizer) |
| |
| prompt = "Explain CVE-2021-44228 in simple terms" |
| output = pipe(prompt, max_new_tokens=300, do_sample=True) |
| print(output[0]["generated_text"]) |
| ``` |
|
|
| --- |
|
|
| ## Ethical Considerations |
|
|
| * This model is trained on cybersecurity data and may produce outputs that describe exploits. |
| * Should only be used for **research, learning, and defensive security purposes**. |
| * Not intended for malicious use. |
|
|
| --- |
|
|
| ## Citation |
|
|
| If you use this model, please cite: |
|
|
| ```bibtex |
| @misc{phi3_mitre_lora_2025, |
| title={Phi-3 Mini LoRA Fine-Tuned on MITRE-STIX-CVE-ExploitDB Dataset}, |
| author={HackDMSV}, |
| year={2025}, |
| howpublished={\url{https://huggingface.co/sushanrai/phi3-cybersec-advisor-lora}} |
| } |
| ``` |
