StefanKrsteski/Phi-3-mini-4k-instruct-DPO-EPFL

Model Card for Phi-3-mini-4k-instruct DPO

Model Details

• Model Name: Phi-3-mini-4k-instruct DPO
• Publisher: Team chatterbox, EPFL
• Model Type: Language Model, Fine-tuned with direct preference optimization (DPO)
• Training Environment: Trained on the EPFL SCITAS cluster using a 32GB GPU.

Intended Use

• Primary Applications: This model is designed as part of an AI-Tutor system.
• Intended Audience: Educators, students, and developers creating educational AI applications.

Model/Data Description

Training Data

• Datasets Used:
  • Milestone 1 Dataset: Includes 1522 unique questions with preference pairs based on the 'overall' rating, totaling 20k+ usable entries after processing.
  • Stack Exchange Dataset: Filters content from specific domains within the Stack Exchange network, using upvoted and downvoted answers to form preference pairs. Total entries after preprocessing: 54458.
  • Ultra Feedback: Utilizes responses rated on criteria like truthfulness and helpfulness to form preference pairs, with a total of 60917 entries after preprocessing.
• Preprocessing Details: Entries with identical chosen and rejected answers were removed. Datasets were formatted as JSONL where each line represents a JSON object with a "prompt", "chosen", and "rejected" response.

Training Procedure

• Configurations: (Refer to the provided training_args and trainer configuration)
• Evaluation Metrics: The primary metric for model performance is eval_loss, with the aim to minimize this value.

Evaluation Results

• Accuracies: eval/rewards/accuracies - 0.83
• Loss: eval/loss - 0.47
• Margins: eval/margins - 4.31

MT-Bench

• Single Grading Score, Overall Avg. - 8.2
• STEM Score - 9.8 (higher than GPT-4)

References

• [Include references and citations for datasets, tools, and methodologies used.]

• PEFT 0.11.1