--- tags: - transformers - safetensors - unsloth - gemma - finance - instruction-tuned - text-generation base_model: - unsloth/gemma-4-E4B-it datasets: - PatronusAI/financebench - next-tat/TAT-QA - AdaptLLM/ConvFinQA - BUPT-Reasoning-Lab/FinanceReasoning - Josephgflowers/Finance-Instruct-500k language: - en pipeline_tag: text-generation --- # TraceAlchemy-Gemma-4-E4B-Finance-IT BF16 `TraceAlchemy-Gemma-4-E4B-Finance-IT` is a finance-focused instruction-tuned Gemma 4 E4B model trained to improve careful financial reasoning, financial table understanding, unit and scale handling, sign and direction checks, and final-answer consistency. This repository contains the **merged BF16 model**. The LoRA adapter has already been merged into the base model weights, so this repo can be loaded directly as a full Hugging Face Transformers model. The model was fine-tuned and merged using [Unsloth](https://github.com/unslothai/unsloth). ## Model Summary This model was trained as a finance reasoning assistant with a focus on: - Financial statement reasoning - Revenue, margin, growth, and ratio calculations - SEC-style table and excerpt extraction - Unit and scale conversion, such as thousands to millions - Sign and direction reasoning - Multi-step table reasoning - Final-answer consistency checking - General finance instruction following The goal of this run was not to teach the model static finance facts. Instead, the goal was to improve the model’s behavior on finance reasoning workflows. The training data emphasizes showing calculations, checking units, avoiding unsupported assumptions, and producing clear final answers. ## Base Model - **Base model:** `unsloth/gemma-4-E4B-it` - **Fine-tuned model name:** `TraceAlchemy-Gemma-4-E4B-Finance-IT` - **Architecture class:** Gemma 4 E4B instruction model - **Training method:** LoRA supervised fine-tuning - **Final format in this repository:** merged BF16 Hugging Face model Gemma 4 E4B is treated as an effective E4B-class model. During training, the loaded parameter count was approximately 8.1B parameters including embeddings. ## What This Repository Contains This is the **merged BF16 model**, not just the LoRA adapter. That means the training flow was: ```text Base Gemma model + trained LoRA adapter → merged 16-bit model → uploaded as full BF16 model ``` The important model file is: ```text model.safetensors ``` This file contains the merged model weights. You should not expect to see: ```text adapter_model.safetensors adapter_config.json ``` Those files belong in the LoRA adapter repository, not the merged BF16 repository. ## Training Configuration | Setting | Value | |---|---:| | Base model | `unsloth/gemma-4-E4B-it` | | Training method | LoRA SFT | | Base loading during training | 8-bit | | Merge format | BF16 | | Max sequence length | 16,384 | | LoRA rank | 64 | | LoRA alpha | 64 | | Learning rate | 2e-5 | | Epochs | 1 | | Per-device batch size | 2 | | Gradient accumulation steps | 8 | | Effective batch size | 16 | | Optimizer | `adamw_8bit` | | Training examples | 10,000 | | Evaluation examples | 630 | | Total training steps | 625 | | Training runtime | ~2.93 hours on A100-class GPU | ## Training Completion The run completed successfully with the following final training output: ```text global_step: 625 epoch: 1.0 train_runtime: 10558.7348 seconds train_samples_per_second: 0.947 train_steps_per_second: 0.059 ``` The final run completed all 625 planned training steps. ## Validation and Evaluation During Training The model was evaluated every 50 training steps on a held-out evaluation set of 630 examples. Validation loss is reported as language-modeling loss on the held-out evaluation split. It is not the same thing as benchmark accuracy, but it is useful for checking whether the model is improving on examples it is not directly training on. The validation trend was strong during the run: | Step | Validation Loss | |---:|---:| | 50 | 0.765309 | | 100 | 0.409274 | | 150 | 0.318539 | | 200 | 0.286068 | | 250 | 0.271857 | | 300 | 0.263259 | | 350 | 0.257434 | The validation loss dropped from `0.765309` at step 50 to `0.257434` by step 350, showing steady improvement on held-out finance examples during training. This suggests the model was not only fitting the training examples, but also improving on the validation set. More benchmarks soon. ## Dataset Mix The training set used a 10,000-example finance-focused mixture. The data recipe was built around a real finance-data anchor from previous experimentation, then expanded with targeted synthetic examples designed to address specific finance reasoning failure modes. ## Real Finance Data | Source | Train | Eval | |---|---:|---:| | FinanceBench | 120 | 30 | | TAT-QA | 2,200 | 100 | | ConvFinQA | 1,800 | 100 | | FinanceReasoning | 750 | 100 | | Finance-Instruct-500k | 400 | 50 | | **Total real examples** | **5,270** | **380** | ## Synthetic Finance Data The synthetic examples targeted specific reasoning skills: | Synthetic Category | Train | Eval | |---|---:|---:| | Hard SEC-style extraction | 1,500 | 50 | | Final-answer consistency checks | 1,200 | 50 | | Sign, scale, and direction reasoning | 900 | 50 | | Multi-step distractor tables | 800 | 50 | | Hard unit-scale conversion | 330 | 50 | | **Total synthetic examples** | **4,730** | **250** | ## Final Dataset Size | Split | Examples | |---|---:| | Train | 10,000 | | Eval | 630 | ## Dataset Credits This model was trained using examples derived from or inspired by the following datasets: - [PatronusAI/financebench](https://huggingface.co/datasets/PatronusAI/financebench) - [next-tat/TAT-QA](https://huggingface.co/datasets/next-tat/TAT-QA) - [AdaptLLM/ConvFinQA](https://huggingface.co/datasets/AdaptLLM/ConvFinQA) - [BUPT-Reasoning-Lab/FinanceReasoning](https://huggingface.co/datasets/BUPT-Reasoning-Lab/FinanceReasoning) - [Josephgflowers/Finance-Instruct-500k](https://huggingface.co/datasets/Josephgflowers/Finance-Instruct-500k) Additional synthetic examples were generated for targeted finance reasoning skills, including SEC-style extraction, scale and unit conversion, sign and direction reasoning, multi-step table reasoning, and final-answer consistency checks. ## What Was Tested During training, the model was tested through validation loss on a held-out evaluation split containing both real and synthetic finance examples. The evaluation set included: - FinanceBench-style question answering - TAT-QA table and text reasoning - ConvFinQA conversational finance QA - FinanceReasoning-style calculation and reasoning examples - Finance-Instruct examples - Synthetic SEC-style extraction checks - Synthetic unit and scale conversion checks - Synthetic sign and direction checks - Synthetic final-answer consistency checks - Synthetic multi-step distractor table checks This validation setup was intended to check whether the model could generalize beyond the exact training examples while staying focused on finance reasoning behavior. Further external benchmark testing is planned. More benchmarks soon. ## Files in This Repository The expected main files include: ```text model.safetensors config.json tokenizer.json tokenizer_config.json processor_config.json chat_template.jinja README.md ``` The `model.safetensors` file is the full merged BF16 model. For an approximately 8.1B parameter model, a BF16 model file around 16 GB is expected because BF16 uses about 2 bytes per parameter. ## Loading the Model Example loading code: ```python import torch from transformers import AutoModelForCausalLM, AutoProcessor model_id = "trjxter/TraceAlchemy-Gemma-4-E4B-Finance-IT-bf16" processor = AutoProcessor.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", ) messages = [ { "role": "user", "content": "A financial table says amounts are in thousands. Revenue is 302,578 and gross profit is 91,842. Express revenue in millions and calculate gross margin.", } ] inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_tensors="pt", ).to(model.device) outputs = model.generate( input_ids=inputs, max_new_tokens=256, do_sample=False, ) print(processor.decode(outputs[0], skip_special_tokens=False)) ``` Depending on your local environment and installed Transformers version, Gemma 4 support may require a recent Transformers release. ## Intended Use This model is intended for finance reasoning and educational/research workflows, especially tasks involving: - Financial statement reasoning - SEC-style table and excerpt extraction - Revenue, margin, growth, and ratio calculations - Unit and scale conversion - Sign and direction checks - Final-answer consistency checking - General finance instruction following ## Limitations This model should not be treated as a source of guaranteed financial truth. It may still: - Make calculation mistakes - Misread financial tables - Mis-handle units or scales - Produce unsupported conclusions - Hallucinate details not present in the prompt - Fail on complex accounting, legal, or investment questions For real investment, accounting, legal, or business decisions, verify outputs against original filings, audited statements, and qualified professionals. ## Related Artifacts This BF16 repository is part of the TraceAlchemy Gemma finance run. Expected related repositories: - LoRA adapter: `trjxter/TraceAlchemy-Gemma-4-E4B-Finance-IT-lora` - Merged BF16 model: `trjxter/TraceAlchemy-Gemma-4-E4B-Finance-IT-bf16` - GGUF model: `trjxter/TraceAlchemy-Gemma-4-E4B-Finance-IT-gguf` ## BF16 vs Adapter vs GGUF This repo is the merged BF16 model. | Repository Type | What It Contains | Use Case | |---|---|---| | LoRA adapter | Small adapter delta weights | Use with the original base model | | Merged BF16 | Full merged model weights | Transformers/HF inference or further conversion | | GGUF | Quantized merged model files | llama.cpp, LM Studio, Ollama, local inference | This repository does not require loading the adapter separately because the adapter has already been merged into the full BF16 model weights. ## Training and Merge Process This model was fine-tuned and merged using [Unsloth](https://github.com/unslothai/unsloth). The merge flow was: ```text Base Gemma model + LoRA adapter → merged 16-bit model → saved as full BF16 Hugging Face model → Hugging Face upload ``` ## Unsloth This model was trained and merged with [Unsloth](https://github.com/unslothai/unsloth). [](https://github.com/unslothai/unsloth)