amjad-awad/mistral-7b-instruct-v0.2-bnb-4bit-books-21K-warmup-steps-15

Mistral 7b instruct

This model is a fine-tuned version of mistral-7b-instruct-v0.2-bnb-4bit on the books dataset for Automatic Essay Grading.

Robust performance on tasks involving Automatic Essay Grading to give a score and rationale

It achieves the following results on the evaluation set:

• Loss: 1.155
• Score Precision: 0.3163
• Score Recall: 0.2514
• Score F1: 0.2243
• Score Accuracy: 0.24

---

• Rationale Precision: 0.4999
• Rationale Recall: 0.5821
• Rationale F1: 0.5369

Model Details

• Base Model: Mistral 7B: https://arxiv.org/abs/2310.06825
• Fine-tuning Dataset: books: IsmaelMousa/books
• Task: Automatic Essay Grading

Training Data

The model is fine-tuned in the books dataset, curated for Automatic Essay Grading. EngSaf consists of student responses annotated with

• Questions: Typically short-answer or essay-type.
• Correct Answer: answers provided by teachers.
• Student Answers: Actual responses written by students.
• Output Label: The actual student score.
• Feedback: Explanations justifying the given scores.

Example Usage

Below is an example of how to use the model with the Hugging Face Transformers library:

import torch
from unsloth import FastLanguageModel
from transformers import AutoModelForCausalLM, AutoTokenizer

model, tokenizer = FastLanguageModel.from_pretrained(model_name="amjad-awad/mistral-7b-instruct-v0.2-bnb-4bit-books-21K-warmup-steps-15",max_seq_length=2048,load_in_4bit=True)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing=True,
    random_state=3407,
)

user_content = (
    "Provide both a score and a rationale by evaluating the student's answer strictly within the mark scheme range, "
    "grading based on how well it meets the question's requirements by comparing the student answer to the reference answer.\n"
    "Question: What is photosynthesis?\n"
    "Reference Answer: Photosynthesis is the process by which green plants and some other organisms use sunlight to synthesize nutrients from carbon dioxide and water. It generally involves the green pigment chlorophyll and generates oxygen as a by-product.\n"
    "Student Answer: Photosynthesis is how plants make their food using sunlight and carbon dioxide. It also gives off oxygen.\n"
    "Mark Scheme: {'1':'Mentions use of sunlight', '2':'Mentions carbon dioxide and water', '3':'Mentions production of oxygen', '4':'Explains synthesis of nutrients or food', '5':'Mentions chlorophyll or green pigment'}"
)

user = [
    {"role":"system", "content": "You are a grading assistant. Evaluate student answers based on the mark scheme. Respond only in JSON format with keys 'score' (int) and 'rationale' (string)."},
    {"role":"user", "content": user_content},
      ]

inputs = tokenizer.apply_chat_template(user, tokenize=True, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=128, temperature=0.2, top_k=5, do_sample=False)[0]
new_generated_ids = generated_ids[inputs["input_ids"].shape[1]:]
generated_text = tokenizer.decode(new_generated_ids, skip_special_tokens=True)

print(generated_text)

Results:

{"score": 5, "rationale": "Your answer is correct. You have accurately described the process of photosynthesis, mentioning the use of sunlight, carbon dioxide, and water, and the production of food and oxygen as by-products. Keep up the good work!"}

Training hyperparameters

The following hyperparameters were used during training:

• per_device_train_batch_size:1
• per_device_eval_batch_size:1
• gradient_accumulation_steps:8
• eval_strategy:"steps"
• save_strategy:"steps"
• eval_steps:10
• logging_dir:"./logs"
• logging_steps:10
• save_total_limit:1
• learning_rate:2e-5
• warmup_steps:15
• weight_decay:0.01
• num_train_epochs:3
• load_best_model_at_end:True
• lr_scheduler_type:"cosine"
• metric_for_best_model:"eval_loss"
• greater_is_better:False

Training results

Step	Training Loss	Validation Loss
10	3.159800	2.924874
20	2.777600	2.360129
30	2.141300	1.867057
40	1.690800	1.548438
50	1.386700	1.372941
60	1.233300	1.282304
70	1.155000	1.260744

Framework versions

• Transformers 4.51.3
• Pytorch 2.7.0
• Datasets 3.6.0
• Unsloth 2025.5.6