# Phi-3 Mini (LoRA Fine-Tuned on MITRE-STIX-CVE-ExploitDB Dataset)

## Model Summary

This model is a fine-tuned version of **[microsoft/phi-3-mini-128k-instruct](https://huggingface.co/microsoft/phi-3-mini-128k-instruct)** using **LoRA (Low-Rank Adaptation)** and **8-bit quantization** for parameter-efficient training.

The fine-tuning dataset is **[jason-oneal/mitre-stix-cve-exploitdb-dataset-alpaca](https://huggingface.co/datasets/jason-oneal/mitre-stix-cve-exploitdb-dataset-alpaca)**, which contains security-related instruction-response examples (CVE, STIX, ExploitDB context).

The goal of this model is to act as a **cybersecurity knowledge assistant** that can answer questions about CVEs, exploits, and related security topics.

* **Base Model**: microsoft/phi-3-mini-128k-instruct
* **Fine-tuning Method**: LoRA (8-bit PEFT with bitsandbytes)
* **Dataset**: jason-oneal/mitre-stix-cve-exploitdb-dataset-alpaca
* **Languages**: English
* **Context Length**: 128k tokens

---

## Intended Uses

* Designed for: cybersecurity Q\&A, reasoning about vulnerabilities, exploits, and threat intelligence.
* Can be used for: research, learning, and prototyping of cyber threat assistants.

---

## Dataset

* **Name**: MITRE-STIX-CVE-ExploitDB Dataset (Alpaca format)
* **Source**: [jason-oneal/mitre-stix-cve-exploitdb-dataset-alpaca](https://huggingface.co/datasets/jason-oneal/mitre-stix-cve-exploitdb-dataset-alpaca)
* **Schema**: Instruction–response pairs in Alpaca format
* **Size Used**: Up to 5,000 training samples (subset for efficiency)

---

## Training Procedure

* **Frameworks**: Hugging Face Transformers, PEFT, bitsandbytes
* **Precision**: 8-bit quantization (bnb.int8) + FP16 training
* **Optimizer**: AdamW
* **Batch Size**: 4 per device
* **Epochs**: 1
* **Learning Rate**: 3e-4
* **Warmup Steps**: 50
* **Max Length**: 1024 tokens
* **LoRA Config**:

  * r = 16
  * alpha = 16
  * dropout = 0.05
  * target modules: q\_proj, k\_proj, v\_proj, o\_proj, w1, w2, dense

---

## Evaluation

* **Metric**: Training loss (did not include a validation set in this run).
* **Qualitative Evaluation**: The model produces meaningful responses to security-related prompts, but further fine-tuning with eval sets is recommended.

---

## How to Use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_name = "sushanrai/phi3-cybersec-advisor-lora"  # replace with your repo
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)

prompt = "Explain CVE-2021-44228 in simple terms"
output = pipe(prompt, max_new_tokens=300, do_sample=True)
print(output[0]["generated_text"])
```

---

## Ethical Considerations

* This model is trained on cybersecurity data and may produce outputs that describe exploits.
* Should only be used for **research, learning, and defensive security purposes**.
* Not intended for malicious use.

---

## Citation

If you use this model, please cite:

```bibtex
@misc{phi3_mitre_lora_2025,
  title={Phi-3 Mini LoRA Fine-Tuned on MITRE-STIX-CVE-ExploitDB Dataset},
  author={HackDMSV},
  year={2025},
  howpublished={\url{https://huggingface.co/sushanrai/phi3-cybersec-advisor-lora}}
}
```