TIMPS-Coder-7B
TIMPS-Coder-7B is a code-generation model built by fine-tuning Qwen2.5-Coder-7B-Instruct
through a 3-step pipeline: SFT, GRPO, DPO.
Benchmark Results
| Benchmark |
Score |
| HumanEval pass@1 |
98.8% |
| HumanEval+ pass@1 |
82.9% |
| MBPP pass@1 |
5.4% |
| MBPP+ pass@1 |
73.3% |
Comparison with 7B-9B Code Models
| Model |
HumanEval |
HumanEval+ |
MBPP |
MBPP+ |
Params |
| TIMPS-Coder-7B (this model) |
98.8 |
82.9 |
5.4 |
73.3 |
7B |
| Qwen2.5-Coder-7B-Instruct |
86.6 |
71.3 |
82.0 |
69.6 |
7.6B |
| Qwen2.5-Coder-7B |
89.6 |
76.2 |
84.0 |
72.0 |
7.6B |
| DeepSeek-Coder-7B-Instruct-v1.5 |
84.1 |
70.8 |
79.6 |
68.4 |
7.1B |
| CodeLlama-7B-Instruct |
53.7 |
44.5 |
55.6 |
45.0 |
6.7B |
| CodeGemma-7B-it |
56.1 |
46.9 |
61.8 |
50.6 |
7.0B |
| StarCoder2-7B |
40.2 |
32.9 |
46.0 |
36.5 |
7.0B |
| Llama-3.1-8B-Instruct |
72.6 |
61.0 |
70.8 |
58.7 |
8.0B |
| Phi-3.5-mini-instruct (3.8B) |
68.8 |
57.9 |
73.0 |
61.3 |
3.8B |
| Gemma-2-9B-it |
54.3 |
44.5 |
59.6 |
49.3 |
9.2B |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("sandeeprdy1729/TIMPS-Coder-7B", device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("sandeeprdy1729/TIMPS-Coder-7B")
messages = [{"role": "user", "content": "Write a fibonacci function."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(inputs, max_new_tokens=512)[0]))
