Tralalabs/gpt-j-6b-dolly15k-lora

GPT-J 6B Dolly-15k LoRA SFT

Tralalabs/gpt-j-6b-dolly15k-lora is a LoRA instruction-tuned adapter for EleutherAI/gpt-j-6b, trained on databricks/databricks-dolly-15k.

This repository contains PEFT LoRA adapter weights, not a fully merged standalone model.
You must load it together with the base model: EleutherAI/gpt-j-6b.

Model Details

• Base model: EleutherAI/gpt-j-6b
• Adapter type: LoRA
• Training type: Supervised Fine-Tuning / SFT
• Dataset: databricks/databricks-dolly-15k
• Task: Instruction following / text generation
• Language: English
• Format: PEFT adapter
• Precision used during training: FP16
• Context length used during training: 1024 tokens

Training Dataset

This adapter was trained on databricks/databricks-dolly-15k, a human-generated instruction-following dataset with more than 15,000 examples.

The dataset includes instruction categories such as:

• Open question answering
• Closed question answering
• Summarization
• Information extraction
• Classification
• Brainstorming
• General generation

Training Configuration

Approximate training setup:

base_model: EleutherAI/gpt-j-6b
dataset: databricks/databricks-dolly-15k
method: LoRA SFT
epochs: 1
max_seq_length: 1024
per_device_train_batch_size: 1
gradient_accumulation_steps: 16
effective_batch_size: 16
learning_rate: 2e-4
warmup_ratio: 0.03
lr_scheduler_type: cosine
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
target_modules:
  - q_proj
  - v_proj
optimizer: adamw_torch_fused
precision: fp16
gpu: 1x NVIDIA L40S

Training Results

Final run metrics:

{
  "eval_loss": 1.5713824033737183,
  "eval_runtime": 16.9353,
  "eval_samples_per_second": 26.631,
  "eval_steps_per_second": 26.631,
  "epoch": 1.0,
  "train_runtime_seconds_manual": 2201.2295339107513,
  "train_runtime_minutes_manual": 36.68715889851252
}

Sanity Test

Prompt:

Below is an instruction that describes a task. Write a helpful, accurate response.

### Instruction:
Give three tips for learning Python.

### Response:

Model output:

1. Use the official documentation. It contains plenty of examples and exercises.
2. Use the interactive shell to test your code before submitting it to production.
3. Take advantage of the many online resources available for free or paid courses on Python.

How to Use

Install dependencies:

pip install -U torch transformers peft accelerate safetensors

Load the LoRA adapter with the base model:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_model_id = "EleutherAI/gpt-j-6b"
adapter_id = "Tralalabs/gpt-j-6b-dolly15k-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)

if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

prompt = """Below is an instruction that describes a task. Write a helpful, accurate response.

### Instruction:
Give three tips for learning Python.

### Response:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=160,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.08,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Prompt Format

Use this prompt format for best results:

Below is an instruction that describes a task. Write a helpful, accurate response.

### Instruction:
{instruction}

### Response:

For examples with context:

Below is an instruction that describes a task, paired with extra context. Write a helpful, accurate response.

### Instruction:
{instruction}

### Context:
{context}

### Response:

Intended Use

This adapter is intended for:

• Instruction-following experiments
• Educational model tuning
• LoRA / PEFT research
• SFT baseline experiments
• Further alignment work such as DPO, preference tuning, or RLHF-style experiments

Limitations

This is a lightweight SFT adapter trained for only one epoch on Dolly-15k. It may still:

• Hallucinate facts
• Produce outdated information
• Generate incorrect answers
• Follow unsafe or low-quality prompts poorly
• Struggle with complex reasoning
• Reflect biases from the base model or dataset

Use retrieval, verification, and safety filtering for real applications.

Not Recommended For

This model is not recommended for:

• Medical advice
• Legal advice
• Financial advice
• Safety-critical decisions
• Fully autonomous production use
• Factual QA without external verification

Base Model Notice

The base model EleutherAI/gpt-j-6b is a raw pretrained causal language model. This adapter improves instruction-following behavior, but it does not turn the model into a fully aligned assistant.

Adapter Files

Expected files in this repository:

adapter_config.json
adapter_model.safetensors
tokenizer.json
tokenizer_config.json
special_tokens_map.json
README.md
final_metrics.json
sanity_generation.txt

Citation

If you use this adapter, cite the base model and dataset:

@misc{gpt-j-6b,
  title = {GPT-J-6B},
  author = {EleutherAI},
  year = {2021},
  url = {https://huggingface.co/EleutherAI/gpt-j-6b}
}

@misc{databricks-dolly-15k,
  title = {databricks-dolly-15k},
  author = {Databricks},
  year = {2023},
  url = {https://huggingface.co/datasets/databricks/databricks-dolly-15k}
}

License

This adapter is released under the Apache 2.0 license.

Check the base model and dataset licenses before commercial or production use.