Instructions to use ArchSpace-Collection/SiameseNorm-DepthAttention with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ArchSpace-Collection/SiameseNorm-DepthAttention with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ArchSpace-Collection/SiameseNorm-DepthAttention")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("ArchSpace-Collection/SiameseNorm-DepthAttention", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ArchSpace-Collection/SiameseNorm-DepthAttention with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ArchSpace-Collection/SiameseNorm-DepthAttention"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ArchSpace-Collection/SiameseNorm-DepthAttention",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ArchSpace-Collection/SiameseNorm-DepthAttention

SGLang

How to use ArchSpace-Collection/SiameseNorm-DepthAttention with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ArchSpace-Collection/SiameseNorm-DepthAttention" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ArchSpace-Collection/SiameseNorm-DepthAttention",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ArchSpace-Collection/SiameseNorm-DepthAttention" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ArchSpace-Collection/SiameseNorm-DepthAttention",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ArchSpace-Collection/SiameseNorm-DepthAttention with Docker Model Runner:
```
docker model run hf.co/ArchSpace-Collection/SiameseNorm-DepthAttention
```

OLMo 3 with SiameseNorm and DepthAttention

This repository contains the 1B checkpoints from the four-stage OLMo 3 training pipeline with SiameseNorm and DepthAttention.

Checkpoints

Checkpoint	Hub subfolder	Context length
Stage 1 pretraining	`olmo3/1b/stage1`	8,192
Stage 2 mid-training	`olmo3/1b/stage2`	8,192
Stage 3 long-context training	`olmo3/1b/stage3`	65,536
Stage 4 Think SFT	`olmo3/1b/stage4/think`	65,536
Stage 4 Instruct SFT	`olmo3/1b/stage4/instruct`	65,536

Stage 3 and Stage 4 apply YaRN only to Full-attention layers. Sliding-window attention layers retain the original RoPE and a 4,096-token window.

Loading

Select one checkpoint through subfolder. SDPA is the recommended and release-validated BF16 inference backend:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "ArchSpace-Collection/SiameseNorm-DepthAttention"
subfolder = "olmo3/1b/stage4/instruct"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    subfolder=subfolder,
    trust_remote_code=True,
    fix_mistral_regex=False,
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    subfolder=subfolder,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
)

fix_mistral_regex=False is intentional and preserves the tokenizer behavior used during training.

The eager backend can also load these checkpoints. For BF16 generation, SDPA is recommended because eager cache partitioning can introduce small rounding differences when the leading logits are nearly tied; in that narrow case, greedy generation can select a different token. This is a numerical backend/cache-partition effect, not a checkpoint conversion or weight-integrity problem.

The repository root contains the shared custom modeling code required by Transformers remote-code loading. Each checkpoint subfolder also contains a self-contained copy of its configuration, tokenizer, modeling code, and weights.

Architecture

16 transformer layers
hidden size 2,048
intermediate size 8,192
16 query heads and 16 key/value heads
128-dimensional attention heads
3:1 sliding-window/full-attention pattern
4,096-token sliding window
reordered RMSNorm, SiameseNorm, and DepthAttention

The Hugging Face implementation is intended for inference and generation. Exact continuation of the native distributed training objective should use the accompanying MindSpeed/Megatron training pipeline.

Downloads last month: -; Downloads are not tracked for this model. How to track