README.md

---
language:
  - dna
tags:
  - genomics
  - biology
  - dna
  - masked-language-model
  - prokaryotic
  - metagenomics
  - bioinformatics
license: apache-2.0
datasets:
  - arcinstitute/opengenome2
library_name: pytorch
pipeline_tag: fill-mask
model-index:
  - name: seqlens-v2-micro-16k
    results:
      - task:
          type: text-classification
          name: Coding vs Non-coding Classification
        metrics:
          - name: Accuracy (linear probe, frozen)
            type: accuracy
            value: 0.9111
      - task:
          type: text-classification
          name: Genus Classification (50 held-out genera)
        metrics:
          - name: Accuracy (linear probe, frozen)
            type: accuracy
            value: 0.7978
---

# SeqLens v2 Micro 16K

A compact genomic language model pre-trained on prokaryotic genomes for microbial bioinformatics tasks.

## Model Description

SeqLens v2 is a bidirectional genomic language model built on BiMamba (bidirectional Mamba2 SSM) with interleaved sliding-window attention. It is designed for microbial genomics — taxonomic classification, antimicrobial resistance detection, plasmid identification, and metagenomic analysis.

The **Micro** variant is the smallest in the SeqLens v2 family, targeting high-throughput, low-latency inference.

| Property | Value |
|---|---|
| Parameters | 10.3M |
| Hidden dimension | 256 |
| Layers | 8 (BiMamba) + 2 (sliding-window attention at layers 3, 7) |
| Context length | 16,384 tokens (single nucleotide) |
| Vocabulary | A, T, G, C, N, [CLS], [SEP], [PAD], [MASK] (9 tokens) |
| Pre-training objective | Masked Language Modeling (MLM), 15% mask rate |
| Architecture | BiMamba2 + chunked sliding-window attention + SwiGLU FFN |

## Architecture Details

- **BiMamba blocks:** Bidirectional Mamba2 SSM — processes sequences in both forward and reverse directions using shared weights. Provides O(L) scaling with sequence length.
- **Sliding-window attention:** Applied every 4th layer with window size 512. Captures fine-grained local patterns (codons, motifs) that SSMs can miss.
- **Attention-weighted pooling:** Learned pooling for sequence-level embeddings (superior to mean pooling for downstream tasks).
- **SwiGLU FFN:** Gated feed-forward with 4× expansion at each layer.

## Training

### Data

Pre-trained on prokaryotic genomes from [OpenGenome2](https://huggingface.co/datasets/arcinstitute/opengenome2) (Apache 2.0):
- **GTDB v220:** 113,379 species-cluster representative genomes
- Single-nucleotide tokenization, 16,384 bp chunks
- Quality filtered: sequences with >10% N or low Shannon entropy excluded

### Hyperparameters

| Parameter | Value |
|---|---|
| Optimizer | AdamW (β₁=0.9, β₂=0.98, ε=1e-8) |
| Learning rate | 1e-3 (cosine decay to 1e-5) |
| Warmup | 500 steps |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Precision | BF16 mixed |
| Batch size | 64 effective (8 × 8 GPUs) |
| Total steps | 10,000 |
| Tokens seen | ~1.2B |

### Compute

| Resource | Value |
|---|---|
| Hardware | 8× NVIDIA A100-SXM4-80GB |
| Training time | 57 minutes |
| Framework | PyTorch 2.6.0 + mamba-ssm 2.2.4 |

## Evaluation

### Coding vs Non-coding Classification (linear probe, frozen backbone)

| Model | Params | Accuracy | F1 |
|---|---|---|---|
| **SeqLens v2 Micro** | **10M** | **0.911** | **0.911** |
| SeqLens v1 (89M) | 89M | 0.687 | 0.687 |
| 4-mer baseline | — | 0.588 | 0.588 |
| Random init | 10M | 0.596 | 0.596 |

### Genus Classification (50 held-out genera, linear probe, frozen backbone)

| Model | Params | Accuracy | F1 |
|---|---|---|---|
| 4-mer baseline | — | 0.865 | 0.838 |
| Random init | 10M | 0.826 | 0.768 |
| **SeqLens v2 Micro** | **10M** | **0.798** | **0.730** |

Note: Genus classification is composition-dominated (GC content, tetranucleotide frequencies), where k-mer baselines are expected to be competitive. The coding/non-coding task better reflects the model's learned structural and positional representations.

## Extended Benchmark Comparison

All evaluations below use the frozen linear probe protocol (frozen embeddings → LogisticRegression). Published fine-tuned baselines (e.g., ProkBERT's fine-tuned MCC scores) are not directly comparable — this is an apples-to-apples comparison across models with a fixed downstream classifier.

### ProkBERT Prokaryotic Benchmarks (accuracy)

| Task | v2_novel | v2_standard | NTv3-8M | Caduceus | SeqLens_v1 | NT-v2 | DNABERT-2 |
|---|---|---|---|---|---|---|---|
| Phage ID (L512) | 0.751 | 0.730 | 0.762 | 0.653 | 0.690 | 0.626 | 0.652 |
| Phage ID (L1024) | 0.805 | 0.775 | 0.808 | 0.650 | 0.713 | 0.659 | 0.700 |
| Phage ID (L2048) | 0.838 | 0.815 | 0.834 | 0.675 | 0.770 | 0.713 | 0.723 |
| Promoter (sigma70) | 0.653 | 0.657 | 0.674 | 0.665 | 0.628 | 0.594 | 0.591 |
| Promoter (multispecies) | 0.582 | 0.584 | 0.630 | 0.608 | 0.588 | 0.572 | 0.545 |
| Lifestyle (BASEL) | 0.705 | 0.737 | 0.753 | 0.702 | 0.735 | 0.669 | 0.700 |
| Lifestyle (E. coli, held-out) | 0.648 | 0.667 | 0.706 | 0.627 | 0.660 | 0.607 | 0.638 |
| Lifestyle (Extremophile) | 0.813 | 0.871 | 0.835 | 0.852 | 0.775 | 0.823 | 0.797 |
| **Average** | **0.724** | **0.729** | **0.751** | **0.679** | **0.695** | **0.658** | **0.668** |

NTv3-8M leads this suite (trained on 9T bp across all species, ~36× more data than our 248B prokaryotic tokens). Our v2_standard (this model) is the best of our own three recipes here — the opposite ranking from CDS/GenomicBenchmarks, where the novel recipe wins. See `PROJECT_STATE.md` §3.4 for a per-task recipe breakdown.

### GenomicBenchmarks — Eukaryotic Tasks (accuracy)

| Task | v2_novel | v2_standard | NTv3-8M | Caduceus | SeqLens_v1 | NT-v2 | DNABERT-2 |
|---|---|---|---|---|---|---|---|
| Mouse Enhancers | 0.822 | 0.826 | 0.810 | 0.744 | 0.806 | 0.802 | 0.727 |
| Coding vs Intergenic | 0.895 | 0.904 | 0.927 | 0.936 | 0.929 | 0.886 | 0.948 |
| Human vs Worm | 0.906 | 0.928 | 0.955 | 0.967 | 0.969 | 0.941 | 0.981 |
| Enhancers (Cohn) | 0.730 | 0.726 | 0.734 | 0.746 | 0.754 | 0.721 | 0.809 |
| Enhancers (Ensembl) | 0.715 | 0.729 | 0.735 | 0.747 | 0.771 | 0.749 | 0.758 |
| NonTATA Promoters | 0.830 | 0.842 | 0.852 | 0.865 | 0.864 | 0.831 | 0.891 |
| OCR (Ensembl) | 0.661 | 0.676 | 0.670 | 0.682 | 0.692 | 0.670 | 0.677 |
| **Average (excl. Regulatory)** | **0.794** | **0.805** | **0.812** | **0.813** | **0.826** | **0.800** | **0.827** |

The `human_ensembl_regulatory` task is excluded: our single-nucleotide tokenizer preserves exact sequence length, and length alone is a near-complete shortcut for this task's 3-way label (a Random Forest trained on sequence length alone reaches 91.3% accuracy). Our models score ~0.99–1.00 on this task as an artifact of that leakage, not because of learned regulatory biology. Despite training exclusively on prokaryotic genomes, our 10M models remain competitive with human/multi-species models up to 10× our size on the remaining 7 eukaryotic tasks.

Full results: `experiments/eval_reports/genomic_benchmarks/comparison.json` and `experiments/eval_reports/prokbench/comparison.json`.

## Usage

```python
import torch
from model import SeqLensForMLM
from config import SeqLensConfig, MICRO_CONFIG
from tokenizer import NucleotideTokenizer

# Load model
device = torch.device("cuda")
ckpt = torch.load("seqlens-v2-micro-16k.pt", map_location=device)
model = SeqLensForMLM(MICRO_CONFIG).to(device).to(torch.bfloat16)
model.load_state_dict(ckpt["model"])
model.eval()

# Tokenize a DNA sequence
tokenizer = NucleotideTokenizer(max_len=16384)
seq = "ATGCGATCGATCG..." # your DNA sequence
token_ids = torch.tensor([tokenizer.encode(seq)], dtype=torch.long).to(device)

# Get sequence-level embeddings (for classification tasks)
with torch.no_grad():
    embeddings = model.get_embeddings(token_ids, pool="attention")  # (1, 256)

# Or get per-position predictions (MLM)
with torch.no_grad():
    output = model(token_ids)
    logits = output["logits"]  # (1, L, 9)
```

## Model Family

| Variant | Params | Layers | Dim | Context | Status |
|---|---|---|---|---|---|
| **Micro** | 10M | 8 | 256 | 16K | ✅ Released |
| Base | ~100M | 12 | 512 | 32K | In development |
| Large | ~400M | 24 | 768 | 64K | Planned |

## Limitations

- Pre-trained on prokaryotic genomes only — may underperform on eukaryotic tasks
- 16K context may truncate long contigs; longer variants planned
- Current model trained for ~1.2B tokens; extended training may improve performance
- The `human_ensembl_regulatory` GenomicBenchmarks task is excluded from our averages — it's a confirmed sequence-length artifact of single-nucleotide tokenization, not a genuine capability (see Extended Benchmark Comparison above)

## Citation

```bibtex
@misc{seqlens-v2-2026,
  title={SeqLens v2: Compact Genomic Language Models for Microbial Bioinformatics},
  author={SeqSight Team},
  year={2026},
  url={https://huggingface.co/seqSight/seqlens-v2-micro-16k}
}
```

## License

Apache 2.0