communitylabs/Llama-3.1-8B-cascadia-int4

Llama 3.1 8B Instruct — Cascadia int4 shards

Built with Llama.

Pre-exported Cascadia inference shards for Llama 3.1 8B Instruct, so operators can deploy without running an export themselves.

These are OpenVINO IR artifacts for Cascadia's ov-runtime engine. They are not loadable by transformers — see Using these below.

Presets

Preset	Path	Stages	Layers	Size
2	int4/stages-2	2	32	4.1 GB

Each preset is a separate export: a dense model's layer split is fixed at export time, so a 2-stage tree cannot serve a 4-node pipeline. Pick the one matching your fleet.

int4/stages-2/
  pipeline_config.json     # model geometry: layers, heads, rope, arch tag
  stage_0/  stage_1/       # each: openvino_model.xml/.bin + stage_config.json
  tokenizer/               # tokenizer.json + configs

cascadia.json at the repo root is the machine-readable index (sizes, checksums, export version) that Cascadia's model registry reads.

Using these

hf download communitylabs/Llama-3.1-8B-cascadia-int4 --local-dir ./llama-3.1-8b
cascadia worker --model ./llama-3.1-8b/int4/stages-2 --engine ov-runtime

The worker takes a local path, never a HuggingFace id — Cascadia workers never download or convert models at serve time.

Provenance

Exported with Cascadia's tools/export_shards.py, export_version: v5_canonical_inputs, int4 via NNCF (group size 128). Source weights: unsloth/Meta-Llama-3.1-8B-Instruct, an unmodified redistribution of meta-llama/Llama-3.1-8B-Instruct.

Quantization is lossy. For anything quality-sensitive, benchmark against the source model rather than assuming parity.

License

Llama 3.1 Community License. A copy of the Agreement is included as LICENSE, the required attribution as NOTICE, and Meta's Acceptable Use Policy as USE_POLICY.md.

Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Your use of these artifacts is subject to that Agreement and the Acceptable Use Policy, including the additional commercial terms that apply above 700 million monthly active users. These shards are a derivative work of the Llama Materials.