Add model card

0d66688 verified 12 months ago

9.38 kB

	---
	base_model: mistralai/Ministral-8B-Instruct-2410
	tags:
	- unsloth
	- lora
	- qlora
	- vulnerability-detection
	- security
	- code-analysis
	- cybersecurity
	- ultival
	- peft
	- adapter
	language:
	- en
	license: apache-2.0
	library_name: peft
	pipeline_tag: text-generation
	---

	# UltiVal: Ministral-8B QLoRA Adapter for Vulnerability Detection

	This is a QLoRA adapter fine-tuned from Ministral-8B-Instruct-2410 for detecting security vulnerabilities in source code as part of the UltiVal project.

	## 🚨 Important Note

	This is a LoRA adapter, not a standalone model. You must load it together with the base model `mistralai/Ministral-8B-Instruct-2410`.

	## 📋 Model Details

	- Base Model: `mistralai/Ministral-8B-Instruct-2410`
	- Adapter Type: QLoRA (4-bit Low-Rank Adaptation)
	- Training Framework: Unsloth
	- Task: Security vulnerability detection in source code
	- Model Size: ~334MB (adapter only)
	- Context Length: 2048 tokens
	- Languages: Multi-language code analysis (Python, JavaScript, Java, C/C++, etc.)

	## 🎯 Training Configuration

	\| Parameter \| Value \|
	\|-----------\|--------\|
	\| Training Steps \| 6,000 (best checkpoint) \|
	\| Total Steps \| 6,184 \|
	\| Validation Loss \| 0.5840 (lowest achieved at step 6000) \|
	\| Final Training Loss \| 0.4081 \|
	\| Epochs \| 2 \|
	\| Learning Rate \| 2e-4 → 1.76e-7 (cosine schedule) \|
	\| Batch Size \| 8 (2 × 4 gradient accumulation) \|
	\| Sequence Length \| 2048 tokens \|
	\| LoRA Rank \| 32 \|
	\| LoRA Alpha \| 32 \|
	\| LoRA Dropout \| 0.0 \|
	\| Weight Decay \| 0.01 \|
	\| Warmup Steps \| ~5% of total steps \|

	### Target Modules
	```
	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
	```

	## 🔧 Usage

	### Option 1: Using Unsloth (Recommended)

	```python
	from unsloth import FastLanguageModel
	import torch

	# Load base model
	model, tokenizer = FastLanguageModel.from_pretrained(
	model_name="mistralai/Ministral-8B-Instruct-2410",
	max_seq_length=2048,
	dtype=None,
	load_in_4bit=True,
	)

	# Add LoRA configuration
	model = FastLanguageModel.get_peft_model(
	model,
	r=32,
	target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
	"gate_proj", "up_proj", "down_proj"],
	lora_alpha=32,
	lora_dropout=0,
	bias="none",
	use_gradient_checkpointing="unsloth",
	random_state=3407,
	)

	# Load the trained adapter
	model.load_adapter("starsofchance/Mistral-Unsloth-QLoRA-adapter")

	# Enable inference mode
	FastLanguageModel.for_inference(model)
	```

	### Option 2: Using Transformers + PEFT

	```python
	from transformers import AutoTokenizer, AutoModelForCausalLM
	from peft import PeftModel
	import torch

	# Load base model
	base_model = AutoModelForCausalLM.from_pretrained(
	"mistralai/Ministral-8B-Instruct-2410",
	torch_dtype=torch.float16,
	device_map="auto",
	load_in_4bit=True
	)

	tokenizer = AutoTokenizer.from_pretrained("mistralai/Ministral-8B-Instruct-2410")

	# Load LoRA adapter
	model = PeftModel.from_pretrained(base_model, "starsofchance/Mistral-Unsloth-QLoRA-adapter")
	```

	## 💻 Inference Example

	```python
	# Example: SQL Injection Detection
	code_snippet = '''
	def authenticate_user(username, password):
	query = "SELECT * FROM users WHERE username='" + username + "' AND password='" + password + "'"
	cursor.execute(query)
	return cursor.fetchone()
	'''

	messages = [
	{"role": "user", "content": f"Analyze this code for security vulnerabilities:\n\n{code_snippet}"}
	]

	# Tokenize and generate
	input_ids = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	return_tensors="pt"
	).to(model.device)

	outputs = model.generate(
	input_ids,
	max_new_tokens=512,
	do_sample=False,
	pad_token_id=tokenizer.eos_token_id,
	temperature=0.1
	)

	response = tokenizer.decode(outputs[0][len(input_ids[0]):], skip_special_tokens=True)
	print(response)
	```

	### Expected Output
	```
	This code contains a critical SQL injection vulnerability. The user input (username and password)
	is directly concatenated into the SQL query without any sanitization or parameterization.

	Vulnerability Type: SQL Injection (CWE-89)
	Severity: High
	Location: Line 2, query construction

	How to exploit: An attacker could input malicious SQL code like:
	- Username: `admin' --`
	- Password: `anything`

	Secure fix: Use parameterized queries:
	```python
	def authenticate_user(username, password):
	query = "SELECT * FROM users WHERE username=? AND password=?"
	cursor.execute(query, (username, password))
	return cursor.fetchone()
	```
	```

	## 🛡️ Supported Vulnerability Types

	The model is trained to detect various security vulnerabilities including:

	\| Category \| Examples \|
	\|----------\|----------\|
	\| Injection \| SQL Injection, Command Injection, LDAP Injection \|
	\| XSS \| Reflected XSS, Stored XSS, DOM-based XSS \|
	\| Authentication \| Weak passwords, Authentication bypass, Session management \|
	\| Authorization \| Privilege escalation, Access control issues \|
	\| Cryptography \| Weak encryption, Hardcoded keys, Improper random generation \|
	\| File Operations \| Path traversal, File inclusion, Unsafe deserialization \|
	\| Memory Safety \| Buffer overflow, Use after free, Memory leaks \|
	\| Web Security \| CSRF, SSRF, Insecure redirects \|

	## 📊 Performance Metrics

	### Training Progress
	- Initial Loss: 1.5544
	- Final Loss: 0.4081
	- Best Validation Loss: 0.5840 (step 6000)
	- Training Duration: ~15 hours
	- Convergence: Stable convergence with cosine learning rate schedule

	### Hardware Requirements
	- Training: NVIDIA GPU with 4-bit quantization
	- Inference: Can run on CPU or GPU (GPU recommended for speed)
	- Memory: ~6GB GPU memory for inference with 4-bit quantization

	## 📁 Repository Structure

	```
	starsofchance/Mistral-Unsloth-QLoRA-adapter/
	├── adapter_config.json # LoRA configuration
	├── adapter_model.safetensors # Trained adapter weights (~334MB)
	├── tokenizer.json # Tokenizer configuration
	├── tokenizer_config.json # Tokenizer settings
	├── special_tokens_map.json # Special tokens mapping
	└── README.md # This file
	```

	## ⚠️ Limitations

	1. Adapter Dependency: Requires the base model to function
	2. Context Window: Limited to 2048 tokens
	3. Language Coverage: Primarily trained on common programming languages
	4. False Positives: May flag secure code patterns as potentially vulnerable
	5. Novel Vulnerabilities: May not detect cutting-edge or highly obfuscated attacks
	6. Code Context: Performance depends on having sufficient code context

	## 🔄 Integration Tips

	### Batch Processing
	```python
	def analyze_multiple_files(code_files):
	results = []
	for file_path, code_content in code_files:
	# Analyze each file
	messages = [{"role": "user", "content": f"Analyze for vulnerabilities:\n\n{code_content}"}]
	# ... generate response
	results.append({"file": file_path, "analysis": response})
	return results
	```

	### Custom Prompting
	```python
	# For specific vulnerability types
	prompt = f"""
	Focus on SQL injection vulnerabilities in this code:
	{code_snippet}

	Provide:
	1. Vulnerability assessment (Yes/No)
	2. Risk level (Low/Medium/High/Critical)
	3. Specific location
	4. Remediation steps
	"""
	```

	## 📚 Training Data

	The model was fine-tuned on a curated dataset featuring:
	- Real-world vulnerabilities from CVE databases
	- Secure code patterns for contrast learning
	- Multi-language examples across different frameworks
	- Detailed explanations with remediation guidance
	- Context-rich examples showing vulnerability in realistic scenarios

	## 🎓 Model Lineage

	```
	Ministral-8B-Instruct-2410 (Mistral AI)
	↓
	QLoRA Fine-tuning (Unsloth)
	↓
	UltiVal Vulnerability Detection Adapter
	```

	## 📄 Citation

	If you use this model in your research or applications, please cite:

	```bibtex
	@misc{ultival_mistral_lora_2025,
	title={UltiVal: Ministral-8B QLoRA Adapter for Vulnerability Detection},
	author={StarsOfChance},
	year={2025},
	publisher={Hugging Face},
	url={https://huggingface.co/starsofchance/Mistral-Unsloth-QLoRA-adapter}
	}
	```

	## ⚖️ License

	This adapter inherits the license from the base model `mistralai/Ministral-8B-Instruct-2410`. Please refer to the [base model's license](https://huggingface.co/mistralai/Ministral-8B-Instruct-2410) for specific terms and conditions.

	## 🙏 Acknowledgments

	- Unsloth Team: For the efficient LoRA fine-tuning framework
	- Mistral AI: For the powerful Ministral-8B-Instruct-2410 base model
	- Hugging Face: For the model hosting and PEFT library
	- UltiVal Project: Part of ongoing research in automated vulnerability detection

	## 📞 Contact & Support

	- Issues: Report bugs or issues in the [model repository](https://huggingface.co/starsofchance/Mistral-Unsloth-QLoRA-adapter/discussions)
	- Updates: Follow for model updates and improvements
	- Community: Join discussions about vulnerability detection and code security

	---

	🔒 Security Note: This model is designed to assist in security analysis but should not be the sole method for vulnerability assessment. Always conduct comprehensive security reviews with multiple tools and expert analysis.