Mistral-7B-Sunda-v1.0-GGUF

Description

This repo contains GGUF format model files for Mistral-7B-Sunda-v1.0-GGUF.

Files Provided

Name	Quant	Bits	File Size	Remark
mistral-7b-sunda.v1.0.IQ3_XXS.gguf	IQ3_XXS	3	3.02 GB	3.06 bpw quantization
mistral-7b-sunda.v1.0.IQ3_S.gguf	IQ3_S	3	3.18 GB	3.44 bpw quantization
mistral-7b-sunda.v1.0.IQ3_M.gguf	IQ3_M	3	3.28 GB	3.66 bpw quantization mix
mistral-7b-sunda.v1.0.Q4_0.gguf	Q4_0	4	4.11 GB	3.56G, +0.2166 ppl
mistral-7b-sunda.v1.0.IQ4_NL.gguf	IQ4_NL	4	4.16 GB	4.25 bpw non-linear quantization
mistral-7b-sunda.v1.0.Q4_K_M.gguf	Q4_K_M	4	4.37 GB	3.80G, +0.0532 ppl
mistral-7b-sunda.v1.0.Q5_K_M.gguf	Q5_K_M	5	5.13 GB	4.45G, +0.0122 ppl
mistral-7b-sunda.v1.0.Q6_K.gguf	Q6_K	6	5.94 GB	5.15G, +0.0008 ppl
mistral-7b-sunda.v1.0.Q8_0.gguf	Q8_0	8	7.70 GB	6.70G, +0.0004 ppl

Parameters

path	type	architecture	rope_theta	sliding_win	max_pos_embed
Nero10578/Mistral-7B-Sunda-v1.0	mistral	MistralForCausalLM	10000.0	4096	32768

Benchmarks

not available

Original Model Card

This is a fine tune of Mistral-7B-v0.1 on a very limited range of Sundanese language datasets that are available. This is a learning project for me where I just wanted to see if it's possible to teach a model a new language that it does not inherently support with just a QLora fine tune. It won't only speak sundanese but it just adds sundanese capability to the model that is to me impressive for the limited data and short amount of training time.

Datasets used: Sundanese sources from this repo that I cleaned, deduped and turned into raw corpus completion in jsonl format.

• https://github.com/w11wo/nlp-datasets

Training Hyperparameters are determined from trial and error multiple times and inspired by the Chinese LLaMa-Alpaca finetune

• https://github.com/ymcui/Chinese-LLaMA-Alpaca/wiki/Training-Details#preparation-vocabulary-expansion

• load_in_4bit: true

• adapter: qlora

• sequence_len: 1024

• lora_r: 8

• lora_alpha: 32

• lora_dropout: 0.05

• lora_target_modules: gate_proj, down_proj, up_proj, q_proj, v_proj, k_proj, o_proj

• gradient_accumulation_steps: 2

• micro_batch_size: 16

• num_epochs: 2

• optimizer: adamw_bnb_8bit

• lr_scheduler: cosine

• learning_rate: 0.0002

Hardware/Software used:

• Windows WSL2
• Axolotl
• RTX 4090 24GB

Parameters for example chat:

• temperature: 0.5
• top_p: 0.9
• top_k: 20
• repetition_penalty: 1.05

Character for example chat:

• Context: Di handap ieu obrolan dina basa Sunda antara User jeung AI. Ieu te percakapan bahasa sunda jeung AI Large Language Model nu nga bantu jelema. AI ngan ngabales ku basa sunda. AI na jago ngajawab pertanyaan nu hese jeung mere saran. AI nuturkeun kahayang User.
• Greeting: Naon anu abdi tiasa ngabantosan anjeun?

Example chat:

• AI: Naon anu abdi tiasa ngabantosan anjeun?

• You: pang jelaskeun naon balapan mobil formula 1 te

• AI: Formula 1 mangrupa salah sahiji kompetisi mobil anu diadegkeun taun 1946, anu dipingpin ku Fédération Internationale de l'Automobile (FIA). Kompetisi ieu dilaksanakeun unggal usum ti bulan Maret nepi ka Nopémber. Balapan Formula 1 biasana dilakukeun di laporan anu panjangna kurang leuwih 5-7 km, anu mana mobil-mobil anu parigel dibawa ku tim-tim anu aya di dunia.

• You: Ai fernando alonso te saha? Pang terangkeun

• AI: Fernando Alonso nyaéta saorang pembalap asal Spanyol anu lahir tanggal 29 Juli 1981 di Oviedo, Asturias, Spanyol. Anjeuna mimiti karir profesionalna di taun 1999, nalika anjeuna meunang gelar juara GP2 Series taun 2000. Dina taun 2001, Alonso mimiti maén di Formula One, nalika anjeuna ditandatanganan kontrak ku Tim Minardi. Tapi, sabada dua kali balapan, anjeuna diganti ku Alexander Wurz. Taun 2003, Alonso ganti tim ka Tim Renault, anu mana anjeuna bisa nyitak rékor anyar minangka pembalap paling umurna anu bisa meunang podium di Grand Prix Monaco. Podium éta ogé jadi hiji-hijina podium anu didamel ku Alonso di musim 2003.