sainikhiljuluri2015/llama-31-cybersec

Llama 3.1 8B Cybersecurity

🎯 Model Description

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct specialized for cybersecurity threat detection and network intrusion analysis using RAFT (Retrieval Augmented Fine-Tuning) with DoRA (Weight-Decomposed Low-Rank Adaptation).

Key Features

• 🛡️ Specialized in network intrusion detection (NSL-KDD dataset)
• 📚 Trained on 50,000 RAFT-enhanced examples with 100K document knowledge base
• 🔍 Detects DoS attacks, reconnaissance, unauthorized access, and privilege escalation
• 📖 Cites sources from retrieved documents
• ✅ Admits uncertainty when information is insufficient

🚀 Quick Start

Installation

pip install transformers torch

Basic Usage (Standalone)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "sainikhiljuluri2015/llama-31-cybersec",
    torch_dtype=torch.float16,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained("sainikhiljuluri2015/llama-31-cybersec")

# Prepare input
question = "What is a SYN flood attack?"

prompt = f"""<|im_start|>system
You are a cybersecurity expert.<|im_end|>
<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant
"""

# Generate response
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=300,
    temperature=0.7,
    top_p=0.9
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Recommended Usage (with RAG/Context)

⚠️ IMPORTANT: This model was trained with RAFT and works best with retrieved context documents:

# With context documents (recommended)
context_docs = """
Document 1: [Your retrieved cybersecurity document]
Document 2: [Another relevant document]
Document 3: [Additional context]
"""

prompt = f"""<|im_start|>system
You are a cybersecurity expert. Use the provided documents to answer questions.<|im_end|>
<|im_start|>user
Context Documents:
{context_docs}

Question: {question}<|im_end|>
<|im_start|>assistant
"""

📊 Training Details

Training Data

• Dataset: NSL-KDD (Network Intrusion Detection)
• Training Examples: 50,000 RAFT-enhanced pairs
• Knowledge Base: 100,000 cybersecurity documents
• Vector Database: Pinecone (384-dim embeddings)

Training Configuration

• Method: RAFT + DoRA
• LoRA Rank: 16
• LoRA Alpha: 32
• Dropout: 0.05
• Quantization: 4-bit (training), FP16 (merged model)
• Epochs: 2
• Batch Size: 2 (effective: 16 with gradient accumulation)
• Learning Rate: 2e-4
• Training Platform: Google Colab A100 GPU

RAFT Distribution

• Oracle Mode: 19.9% (correct documents provided)
• Distractor Mode: 60.4% (incorrect documents provided)
• No-Context Mode: 19.8% (no documents provided)

🎯 Capabilities

Attack Types Detected

1. DoS Attacks: Neptune, SYN flood, Smurf, Teardrop
2. Reconnaissance/Probing: Port scanning, IP sweep, network mapping
3. Unauthorized Access (R2L): Brute force, dictionary attacks, FTP/SSH intrusions
4. Privilege Escalation (U2R): Buffer overflow, rootkit, privilege abuse

Model Behavior

• ✅ Analyzes network traffic patterns
• ✅ Provides detailed threat explanations
• ✅ Cites relevant documents when available
• ✅ Admits uncertainty when information is insufficient
• ✅ Recommends appropriate security actions

⚡ Performance

• Training Accuracy: ~95-98%
• Inference Time: ~2-3 seconds per query (A100 GPU)
• Improvement: 25-35% over baseline methods
• Model Size: ~15GB (FP16)

🚨 Limitations

1. Requires Context for Best Results: Model was trained with RAFT and performs best when provided with retrieved context documents
2. Domain-Specific: Specialized for network intrusion detection (NSL-KDD scope)
3. May Not Generalize: Limited to attack types seen during training
4. Prompt Format: Requires specific prompt structure for optimal performance

💻 Deployment

HuggingFace Inference Endpoints (Recommended)

Configuration for Auto-Scaling:

Instance Type: A100 Small (40GB)
Min Replicas: 0  # Scales to zero when not in use
Max Replicas: 1
Auto-scaling: Enabled
Cold Start: ~30-60 seconds

API Usage Example

import requests

API_URL = "https://YOUR-ENDPOINT.aws.endpoints.huggingface.cloud"
headers = {"Authorization": f"Bearer {YOUR_HF_TOKEN}"}

def query(payload):
    response = requests.post(API_URL, headers=headers, json=payload)
    return response.json()

output = query({
    "inputs": "What is a neptune DoS attack?",
    "parameters": {
        "max_new_tokens": 300,
        "temperature": 0.7,
        "top_p": 0.9
    }
})

📁 Model Architecture

• Base Architecture: Llama/Foundation-Sec 8B
• Parameters: ~8 billion
• Precision: FP16 (merged model)
• Adapter Type: DoRA (rank-16)
• Context Length: 4096 tokens

🔗 Related Resources

• Training Code: GitHub Repository
• Dataset: NSL-KDD
• Paper: RAFT - Adapting Language Model to Domain Specific RAG
• Vector Database: Pinecone

📝 Citation

If you use this model in your research, please cite:

@misc{cybersec-llm-raft-dora-llama,
  author = {Sainikhil},
  title = {Llama 3.1 8B Cybersecurity},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/sainikhiljuluri2015/llama-31-cybersec}
}

📧 Contact

For questions, issues, or collaborations:

• Open an issue on the model repository
• Connect via HuggingFace discussions

📜 License

This model is released under the Apache 2.0 License.

⚠️ Disclaimer

This model is for research and educational purposes. Always validate cybersecurity findings with professional security tools and experts. Do not rely solely on AI models for critical security decisions.

---

Built with: 🤖 Transformers • 🔥 PEFT • ⚡ DoRA • 📚 RAFT • 🎯 Pinecone