docs/APPLE_MLX_DEPLOYMENT.md

Apple MLX Deployment Guide

Supported deployment target

The validated target is an Apple M1 Mac with 16 GB unified memory. The model also runs on newer Apple Silicon systems with sufficient memory.

The 4-bit release is the recommended choice for 16 GB because macOS, Python, model weights, Metal allocations, image tensors and KV cache all share the same memory pool.

1. Prerequisites

Confirm Apple Silicon:

uname -s
uname -m
sysctl -n machdep.cpu.brand_string

Expected:

Darwin
arm64
Apple M1, M2, M3, M4 or later Apple Silicon

Install Python if needed. The validated conversion used Python 3.14, but use a current native arm64 Python compatible with the pinned MLX packages.

2. Download the repository

python3 -m pip install --upgrade "huggingface-hub[cli]"
hf auth login

cd ~/Downloads
hf download \
  AXONVERTEX-AI-RESEARCH/Shieldstral-1.0-3B-MLX-4bit \
  --local-dir Shieldstral-1.0-3B-MLX-4bit

cd Shieldstral-1.0-3B-MLX-4bit

The .cache/huggingface directory created by --local-dir is local download metadata. Remove it after download when a clean directory is preferred:

rm -rf .cache/huggingface

3. Install dependencies

./scripts/install.sh
source .venv/bin/activate

The installer creates a local virtual environment and installs the pinned MLX-VLM stack.

4. Verify release integrity

./scripts/verify_checksums.sh

Then load the model and perform one deterministic classification:

./scripts/validate_model.sh

Expected outcome:

PASS: model loaded
PASS: deterministic classification returned yes or no

5. Start the server

./scripts/start_server.sh

Defaults:

HOST=127.0.0.1
PORT=18190
MAX_KV_SIZE=16384
KV_BITS=8
TOP_LOGPROBS_K=20

The command binds only to localhost. Keep this default unless remote access is intentionally configured and protected.

Lower-memory mode

MAX_KV_SIZE=8192 ./scripts/start_server.sh

Minimal test mode

MAX_KV_SIZE=4096 ./scripts/start_server.sh

Different port

PORT=18210 ./scripts/start_server.sh

Bind to a Tailscale interface

Determine the Tailscale IPv4 address:

TAILSCALE_IP="$(tailscale ip -4)"

Start:

HOST="$TAILSCALE_IP" PORT=18190 ./scripts/start_server.sh

Do not bind to 0.0.0.0 on an untrusted network without authentication, firewalling and transport controls.

6. Health and smoke tests

From a second terminal:

cd ~/Downloads/Shieldstral-1.0-3B-MLX-4bit
source .venv/bin/activate

./scripts/health.sh
./scripts/smoke_test.sh

Run the technical-report protocol cases:

./scripts/run_paper_alignment.sh

Run all 13 regression cases:

./scripts/run_evals.sh

7. Background service with launchd

Create a logs directory:

mkdir -p "$HOME/Library/Logs/shieldstral-mlx"

Create ~/Library/LaunchAgents/ai.axonvertex.shieldstral-mlx.plist and replace /ABSOLUTE/PATH:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>ai.axonvertex.shieldstral-mlx</string>
  <key>ProgramArguments</key>
  <array>
    <string>/bin/zsh</string>
    <string>/ABSOLUTE/PATH/Shieldstral-1.0-3B-MLX-4bit/scripts/start_server.sh</string>
  </array>
  <key>WorkingDirectory</key>
  <string>/ABSOLUTE/PATH/Shieldstral-1.0-3B-MLX-4bit</string>
  <key>EnvironmentVariables</key>
  <dict>
    <key>HOST</key><string>127.0.0.1</string>
    <key>PORT</key><string>18190</string>
    <key>MAX_KV_SIZE</key><string>16384</string>
    <key>KV_BITS</key><string>8</string>
  </dict>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
  <key>StandardOutPath</key>
  <string>/Users/REPLACE_ME/Library/Logs/shieldstral-mlx/server.out.log</string>
  <key>StandardErrorPath</key>
  <string>/Users/REPLACE_ME/Library/Logs/shieldstral-mlx/server.err.log</string>
</dict>
</plist>

Load it:

launchctl bootout "gui/$(id -u)" ~/Library/LaunchAgents/ai.axonvertex.shieldstral-mlx.plist 2>/dev/null || true
launchctl bootstrap "gui/$(id -u)" ~/Library/LaunchAgents/ai.axonvertex.shieldstral-mlx.plist
launchctl kickstart -k "gui/$(id -u)/ai.axonvertex.shieldstral-mlx"

Check:

launchctl print "gui/$(id -u)/ai.axonvertex.shieldstral-mlx"
curl -fsS http://127.0.0.1:18190/health | python -m json.tool

8. Memory monitoring

Use Activity Monitor or:

ps -o pid,rss,%cpu,command -ax | grep -E 'mlx_vlm.server|Python' | grep -v grep
memory_pressure

If macOS memory pressure becomes yellow or red:

1. reduce MAX_KV_SIZE;
2. reduce simultaneous requests;
3. close memory-heavy applications;
4. restart the server after large multimodal workloads.

9. Operational guidance

• Keep max_tokens=1 for classification.
• Keep temperature=0.0 for deterministic scoring.
• Request top_logprobs=20 so both yes and no are normally available.
• Use one policy question per request for category-specific moderation.
• Use a broad query only when the instruction explicitly defines the combined policy scope.
• Treat the score as a policy signal, not as sole authority for destructive or irreversible action.

10. Troubleshooting

MistralCommonBackend has no attribute vocab

This release intentionally omits tekken.json from the runtime artifact and uses tokenizer.json, forcing the MLX-compatible TokenizersBackend. Do not add tekken.json back to the model directory.

Model loads but server is killed

Reduce context:

MAX_KV_SIZE=4096 ./scripts/start_server.sh

Port already in use

lsof -nP -iTCP:18190 -sTCP:LISTEN
PORT=18191 ./scripts/start_server.sh

Checksum failure after hf download --local-dir

The checksum manifest excludes .cache/huggingface. Run the provided verifier from the model root. It ignores local cache metadata.