ho22joshua/hep-chat-entrypoint

HEP Chat Entrypoint

This repository contains ready-to-run inference entry points for HEP chat models. Models are listed in models.json, so new trained models can be added without rewriting the app, terminal chat, batch runner, or download helper.

The default configured model is:

hep-qwen2.5-7b-lora16-sigbg-irred-red-step2200

Human Quick Start

The default human workflow is terminal chat:

python -m pip install -r requirements.txt
python diagnose_system.py
python terminal_chat.py

On an Apple Silicon MacBook GPU:

python terminal_chat.py --device mps

Fast local mode uses shorter deterministic replies and skips chat history so each turn has less prompt to process:

python terminal_chat.py --device mps --fast

MPS does not support bfloat16. This repo loads LoRA adapters on CPU first and then moves the model to MPS in a supported dtype.

Inside the chat, use /clear to reset history and /exit to quit.

Apple Silicon Arm64 Environment

If diagnostics reports platform.machine as x86_64 on an Apple Silicon Mac, create an arm64 virtual environment. The repo ignores .venv-arm64/ so it can live beside the code:

/Library/Frameworks/Python.framework/Versions/3.11/bin/python3 -m venv .venv-arm64
.venv-arm64/bin/python -m pip install --upgrade pip setuptools wheel
.venv-arm64/bin/python -m pip install -r requirements.txt
.venv-arm64/bin/python diagnose_system.py
.venv-arm64/bin/python terminal_chat.py --device mps --fast

To activate it for a shell session:

source .venv-arm64/bin/activate
python terminal_chat.py --device mps --fast

Code Agent Quick Start

The default code-agent workflow is non-interactive batch inference:

python -m pip install -r requirements.txt
python diagnose_system.py
python run_questions.py --questions questions.txt --output outputs.jsonl --device auto

For faster local batch inference:

python run_questions.py --questions questions.txt --output outputs.jsonl --device mps --fast

Code agents should execute model inference themselves with run_questions.py, not drive terminal_chat.py or the Gradio UI. The question file is one question per non-empty line. For blank-line separated multi-line questions:

python run_questions.py --questions questions.txt --input-format blocks --output outputs.jsonl

The output is JSONL. The first record is run metadata unless --no-metadata is set. Answer records include question, answer, model_name, base_model, adapter, model_mode, device, and created_at fields. See AGENTS.md for the short operational guide intended for Claude, Codex, Copilot, and similar code agents.

Models

List configured models:

python list_models.py

Run a specific configured model:

python run_questions.py --model hep-qwen2.5-7b-lora16-sigbg-irred-red-step2200 --questions questions.txt --output outputs.jsonl
python terminal_chat.py --model hep-qwen2.5-7b-lora16-sigbg-irred-red-step2200

To add a model, edit models.json:

{
  "models": {
    "my-new-hep-model": {
      "base_model": "BASE_MODEL_REPO",
      "adapter": "OPTIONAL_LORA_ADAPTER_REPO",
      "description": "Short description for humans and agents."
    }
  }
}

If a model does not use a LoRA adapter, omit adapter and run it with --model-mode base or MODEL_MODE=base.

Validation and Benchmarking

Run a single end-to-end smoke test:

python smoke_test.py --device mps --fast --text

Benchmark load time, latency, and output tokens per second:

python benchmark.py --device mps --fast --questions examples/questions.txt

Both commands emit JSON by default. Use smoke tests after changing model entries or Python environments, and benchmark when comparing Mac MPS, CUDA, CPU, or future quantized backends.

Copy-Paste Cluster Launch

Run these commands inside a GPU allocation, not on a login node:

mkdir -p ${SCRATCH:-$PWD}/hf_home
export HF_HOME=${SCRATCH:-$PWD}/hf_home
export HF_HUB_CACHE=$HF_HOME/hub

git clone https://huggingface.co/ho22joshua/hep-chat-entrypoint
cd hep-chat-entrypoint

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

python download_models.py
python terminal_chat.py

If you already cloned the repo:

cd hep-chat-entrypoint
git pull

mkdir -p ${SCRATCH:-$PWD}/hf_home
export HF_HOME=${SCRATCH:-$PWD}/hf_home
export HF_HUB_CACHE=$HF_HOME/hub

python -m pip install -r requirements.txt --upgrade
python download_models.py
python terminal_chat.py

Run the baseline Qwen2.5-7B-Instruct model without the LoRA adapter:

python terminal_chat.py --model-mode base

Run the fine-tuned HEP LoRA adapter, which is the default:

python terminal_chat.py --model-mode adapter

Compare baseline and adapter answers side by side:

python compare_chat.py

compare_chat.py loads both the baseline and adapter model at the same time, so it needs substantially more memory than single-model chat.

Inside the chat, use /clear to reset history and /exit to quit.

If Hugging Face returns 429 Too Many Requests, wait a few minutes and rerun:

python download_models.py
python terminal_chat.py

The download helper uses one worker and retries, which is friendlier to the Hub from shared clusters.

Run Locally

git clone https://huggingface.co/ho22joshua/hep-chat-entrypoint
cd hep-chat-entrypoint
pip install -r requirements.txt
python terminal_chat.py

Terminal chat works best with a CUDA GPU. On an Apple Silicon MacBook, use the PyTorch MPS backend:

python -m pip install -r requirements.txt
python terminal_chat.py --device mps

DEVICE=auto and --device auto also detect MPS automatically when CUDA is not available. CPU-only mode is possible but very slow:

ALLOW_CPU=1 python app.py

To launch the Gradio app instead of terminal chat:

python app.py

Terminal Chat

For clusters, the terminal interface is often simpler than running a web server:

python terminal_chat.py

Commands inside the chat:

• /clear: reset conversation history
• /exit: quit

CPU-only fallback:

python terminal_chat.py --allow-cpu

Explicit Apple Silicon GPU mode:

python terminal_chat.py --device mps

Fast Apple Silicon GPU mode:

python terminal_chat.py --device mps --fast

Useful generation knobs:

python terminal_chat.py --max-new-tokens 200 --temperature 0.0 --no-history

Baseline-only mode:

python terminal_chat.py --model-mode base

Side-by-side comparison mode:

python compare_chat.py --max-new-tokens 200 --temperature 0.0

Predownload the model cache without starting a chat:

python download_models.py

Predownload every configured model:

python download_models.py --all

Cluster Setup

On shared clusters, put the Hugging Face cache on scratch or project storage before starting the app:

export HF_HOME=/path/to/scratch/hf_home
export HF_HUB_CACHE=$HF_HOME/hub
python app.py

Replace /path/to/scratch/hf_home with a real writable directory. Do not use the placeholder path literally.

On Perlmutter, a typical setup is:

mkdir -p $SCRATCH/hf_home
export HF_HOME=$SCRATCH/hf_home
export HF_HUB_CACHE=$HF_HOME/hub

python terminal_chat.py

If $SCRATCH is not set, use a writable project directory instead:

mkdir -p /global/cfs/projectdirs/atlas/joshua/hf_home
export HF_HOME=/global/cfs/projectdirs/atlas/joshua/hf_home
export HF_HUB_CACHE=$HF_HOME/hub

python terminal_chat.py

Check that the cache path is valid before loading the model:

echo $HF_HOME
test -w "$HF_HOME" && echo writable

If you need the Gradio server to listen on all interfaces for tunneling:

GRADIO_SERVER_NAME=0.0.0.0 python app.py

For faster Gradio defaults:

FAST_MODE=1 DEVICE=mps python app.py

Do not load the 7B model on login nodes. Start an interactive GPU allocation or batch job first.

Storage and Memory

Run diagnostics before loading the model:

python diagnose_system.py

If Torch import is broken and you only want Python, disk, RAM, and package metadata:

python diagnose_system.py --skip-torch

To test requirement logic without changing the machine, use simulation flags:

python diagnose_system.py --simulate-ram-gb 16 --simulate-disk-free-gb 20
python diagnose_system.py --simulate-ram-gb 32 --simulate-disk-free-gb 25 --simulate-mps
python diagnose_system.py --simulate-ram-gb 64 --simulate-disk-free-gb 50 --simulate-cuda-vram-gb 24

The diagnostics output is JSON so code agents can parse it. Important fields:

• summary.can_try_mps: this Python environment can try Apple Silicon GPU
• summary.can_try_cuda: this Python environment can try NVIDIA CUDA GPU
• summary.meets_minimum_disk: cache disk has enough free space to try one 7B model
• summary.meets_recommended_disk: cache disk has comfortable free space
• summary.meets_minimum_ram: system RAM meets the minimum for 7B inference
• summary.meets_recommended_ram: system RAM is in the comfortable range
• disk.cache_dir: where Hugging Face model files are expected
• disk.cache_size_gb: current size of that model cache
• packages: installed versions of Torch, Transformers, PEFT, NumPy, and related packages
• simulation.enabled: whether any reported resource values were synthetic
• memory.detected_total_ram_gb and disk.detected_free_gb: real values when simulation is used

Minimum requirements for the default 7B model:

• Disk: 25 GB free for one clean model cache and Python environment
• Apple Silicon Mac: 32 GB unified memory
• NVIDIA CUDA GPU: 16 GB VRAM
• CPU-only: 32 GB system RAM, but expect very slow inference
• Python: 3.10+ with the packages in requirements.txt

Recommended requirements:

• Disk: 50 GB free for cache growth, partial downloads, and future model variants
• Apple Silicon Mac: 64 GB unified memory
• NVIDIA CUDA GPU: 24 GB VRAM
• CPU-only: 64 GB system RAM, still slow
• On clusters: 75 GB+ in scratch/cache storage if trying multiple models

Why disk usage is this high:

• Default 7B base model: about 15 GB
• LoRA adapter: about 100 MB
• Python environment: commonly 2-8 GB on Mac, 5-15 GB with CUDA packages
• Hugging Face cache can keep partial downloads, locks, snapshots, and old revisions

Why memory usage is this high:

• The current backend runs the default 7B model in fp16/bf16-style Transformers weights.
• The base model is roughly 14 GB before runtime overhead.
• KV cache grows with prompt length and generated tokens.
• Apple Silicon uses unified memory, shared by the OS, Python, model weights, and GPU execution.

Performance guidance:

• Use --fast for local Mac runs.
• Keep terminal chat open across multiple questions to avoid reloading weights.
• Reduce --max-new-tokens for faster replies and lower memory pressure.
• Avoid compare_chat.py on memory-constrained machines; it loads two models.
• On Apple Silicon, prefer an arm64 Python environment. If diagnostics show platform.machine as x86_64, recreate the environment with an arm64 Python/Conda install for best local GPU performance.

Fixing Install Issues

If Gradio fails with ImportError: cannot import name 'HfFolder' from 'huggingface_hub', reinstall with the pinned requirements:

pip install -r requirements.txt

or explicitly:

pip install "huggingface_hub<1.0" "gradio==4.44.1"

Turn This Into A Hugging Face Space

Hugging Face currently requires a paid account feature for Gradio Spaces on cpu-basic, and this model needs GPU hardware for actual chat anyway.

After enabling Space hosting/GPU hardware, create a Gradio Space and upload these files:

hf repo create ho22joshua/hep-chat --type space --sdk gradio --public --exist-ok
hf upload ho22joshua/hep-chat . . --repo-type space

Then set the Space hardware to a GPU such as l4x1, a10g-small, or t4-medium.

OpenAI-Compatible Endpoint

For a production-style endpoint, use vLLM:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --enable-lora \
  --lora-modules hep=ho22joshua/hep-qwen2.5-7b-lora16-sigbg-irred-red-step2200 \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype bfloat16

Call it with:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hep",
    "messages": [
      {"role": "user", "content": "For H to AA to photons, what Standard Model backgrounds should be considered?"}
    ],
    "max_tokens": 300
  }'