Srx7703/gemma-2-27b-financial-adapter

gemma-2-27b-it — SEC Financial QA LoRA Adapter (Phase 1)

LoRA (rank=8) adapter for google/gemma-2-27b-it, fine-tuned for SEC filing analysis on TPU v6e-8 with PyTorch/XLA SPMD FSDPv2.

Result (n=20 held-out)

Model	BERTScore F1	Δ vs base
Base gemma-2-27b-it	0.8078	—
+ this adapter	0.8361	+3.50%

Paired t = 3.64 (p <0.01), 95% CI [+0.012, +0.045], wins 16/20 held-out items.

Training data

1,060 knowledge-distilled QA pairs from 381 SEC summaries (10-K / 10-Q / 8-K) covering 69 S&P 500 companies. Knowledge distillation done with Gemini 3.1 Pro.

Hyperparameters

• LoRA rank=8, alpha=16, dropout=0.05
• Targets: q/k/v/o/gate/up/down_proj (regex-scoped to .language_model. for multimodal Gemma 4)
• bs=4 × grad_accum=2, AdamW lr=1e-4, 2 epochs, bf16
• FSDPv2 mesh (8, 1) over ("fsdp", "tensor")

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-27b-it",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("google/gemma-2-27b-it")
model = PeftModel.from_pretrained(base, "Srx7703/gemma-2-27b-financial-adapter")

prompt = "What are the principal risk factors disclosed in NVIDIA's most recent 10-K?"
inputs = tok.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)
out = model.generate(inputs, max_new_tokens=256)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

See also

• GitHub repo (full pipeline, training scripts, evaluation): https://github.com/srx7703/multi-horizon-financial-agent
• Companion adapter (Phase 2): https://huggingface.co/Srx7703/gemma-4-31b-financial-adapter

License

Use of this adapter is subject to the Gemma Terms of Use. The base model weights are not included.