PoreBERT-DNA-VQI-110M-530

A BERT-style foundation model for nanopore DNA sequencing signal representation learning.

PoreBERT-DNA-VQI-110M-530 learns contextual representations from discretized nanopore electrical signal tokens generated by a Vector Quantization (VQ) tokenizer.

The model is pretrained with a Masked Language Modeling (MLM) objective, following the BERT architecture paradigm, and is designed for downstream nanopore sequencing applications.

---

Model Summary

Property	Description
Model Type	BERT Encoder
Domain	Nanopore DNA sequencing
Parameters	~110M
Training Objective	Masked Language Modeling (MLM)
Input	Discrete nanopore signal tokens
Tokenizer	VQ-based tokenizer
Vocabulary	Minimal vocabulary tokenizer
Development Release	530

---

Model Name Explanation

PoreBERT-DNA-VQI-110M-530

PoreBERT

Nanopore sequencing foundation model based on the BERT encoder architecture.

DNA

The model is trained on nanopore DNA sequencing electrical signal representations.

VQ

Vector Quantization tokenizer.

The raw nanopore electrical signal is first converted into discrete tokens through a neural codec based on vector quantization.

I

Minimal vocabulary tokenizer.

The VQI tokenizer is designed with a compact vocabulary to provide efficient signal token representation while preserving important sequencing information.

110M

Approximately 110 million model parameters.

530

Internal development release identifier.

---

Architecture Overview

The complete pipeline is:

Nanopore Electrical Signal

          |
          v

PoreCodec
(CNN Encoder + Vector Quantization)

          |
          v

Discrete Signal Tokens

          |
          v

PoreBERT-DNA-VQI-110M-530

          |
          v

Downstream Applications

The BERT model itself operates on token sequences rather than raw electrical signals.

---

Tokenizer

This model uses the following tokenizer:

ShuaiAnwo/pore-codec-rsq742c12a-511

The tokenizer converts nanopore electrical signals into discrete token IDs.

Tokenizer repository:

https://huggingface.co/ShuaiAnwo/pore-codec-rsq742c12a-511

The workflow is:

Raw signal

    |
    v

Pore Codec Tokenizer

    |
    v

Token IDs

    |
    v

PoreBERT

---

Model Architecture Details

Configuration:

Parameter	Value
Architecture	Transformer Encoder
Hidden Size	768
Transformer Layers	12
Attention Heads	12
Intermediate Size	3076
Maximum Sequence Length	1536
Vocabulary Size	2560
Parameters	~110M

The architecture follows the standard BERT-style encoder design:

Token Embedding
        |
Position Embedding
        |
LayerNorm + Dropout
        |
12 Transformer Encoder Layers
        |
LayerNorm
        |
Contextual Token Representation

---

Pretraining Objective

The model is pretrained using Masked Language Modeling.

During training:

1. Token sequences are generated by the VQ tokenizer.
2. A fraction of tokens are randomly masked.
3. The model predicts the original tokens using bidirectional context.

Example:

Input:

A B [MASK] D E


Prediction:

C

Training configuration:

Parameter	Value
MLM Probability	0.25
Optimizer	AdamW
Precision	BF16
Sequence Length	1536

---

Usage

PoreBERT-DNA-VQI-110M-530 operates on discrete signal tokens generated by the corresponding PoreCodec VQ tokenizer.

Pipeline:

Nanopore Raw Signal
        |
        v
PoreCodec VQ Tokenizer
        |
        v
Discrete Token IDs
        |
        v
PoreBERT
        |
        v
Contextual Signal Embeddings

Quick Start

import numpy as np
import torch

from transformers import AutoFeatureExtractor, AutoModel


# Load PoreCodec tokenizer and PoreBERT model

codec_name = "ShuaiAnwo/pore-codec-rsq742c12a-511"
bert_name = "ShuaiAnwo/PoreBERT-DNA-VQI-110M-530"


codec = AutoModel.from_pretrained(
    codec_name,
    trust_remote_code=True,
).eval()


feature_extractor = AutoFeatureExtractor.from_pretrained(
    codec_name,
    trust_remote_code=True,
)


bert = AutoModel.from_pretrained(
    bert_name,
).eval()



# Example nanopore electrical signal

raw_signal = np.random.normal(
    70,
    8,
    1855,
).astype(np.float32)



with torch.no_grad():

    # Raw signal -> VQ token IDs

    signal = feature_extractor(
        raw_signal,
        return_tensors="pt",
    )["signal"]


    token_ids = codec.encode_signal(
        signal,
        layer=2,
    )


    # Token IDs -> PoreBERT contextual embeddings

    outputs = bert(
        input_ids=token_ids,
    )


    embeddings = outputs.last_hidden_state



print("Embedding shape:", embeddings.shape)

Output:

Embedding shape:
(batch_size, sequence_length, 768)

The generated embeddings can be used for downstream nanopore sequencing tasks, including:

• Basecalling
• Modified base detection
• Signal representation learning
• Read-level embedding

---

Training Data

The model was pretrained on nanopore DNA sequencing signal token sequences.

The training data pipeline:

Nanopore signal

        |
        v

PoreCodec tokenizer

        |
        v

Discrete token sequence

        |
        v

Masked Language Modeling

        |
        v

PoreBERT

---

Gradient Checkpointing

This model supports gradient checkpointing during training.

Gradient checkpointing reduces GPU memory usage by recomputing intermediate activations during backward propagation.

Benefits:

• Lower GPU memory consumption
• Ability to train longer sequences
• Larger models on limited hardware

Trade-off:

• Increased training computation time

---

Limitations

• The model does not directly accept raw nanopore electrical signals.
• Raw signal must first be converted into tokens using the corresponding tokenizer.
• Performance may depend on tokenizer quality and training data distribution.

---

Citation

Coming soon.

---

License

Please refer to the LICENSE file for usage conditions.