Instructions to use l3lab/ntpctx-llama3-8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use l3lab/ntpctx-llama3-8b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="l3lab/ntpctx-llama3-8b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("l3lab/ntpctx-llama3-8b")
model = AutoModelForMultimodalLM.from_pretrained("l3lab/ntpctx-llama3-8b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use l3lab/ntpctx-llama3-8b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "l3lab/ntpctx-llama3-8b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "l3lab/ntpctx-llama3-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/l3lab/ntpctx-llama3-8b

SGLang

How to use l3lab/ntpctx-llama3-8b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "l3lab/ntpctx-llama3-8b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "l3lab/ntpctx-llama3-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "l3lab/ntpctx-llama3-8b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "l3lab/ntpctx-llama3-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use l3lab/ntpctx-llama3-8b with Docker Model Runner:
```
docker model run hf.co/l3lab/ntpctx-llama3-8b
```

ntpctx-llama3-8b / README.md

wellecks

Create README.md

b6e07aa verified almost 2 years ago

1.32 kB

license: mit
base_model: meta-llama/Meta-Llama-3-8B

miniCTX: Neural Theorem Proving with (Long-)Contexts

File-tuned context model based on miniCTX: Neural Theorem Proving with (Long-)Contexts.

Base language model: Llama 3 8B
Data: ntp-mathlib-instruct-context

It is specifically finetuned for Lean 4 tactic prediction given proof states and optional file contexts.

Example input

/- You are proving a theorem in Lean 4.
You are given the following information:
- The file contents up to the current tactic, inside [CTX]...[/CTX]
- The current proof state, inside [STATE]...[/STATE]

Your task is to generate the next tactic in the proof.
Put the next tactic inside [TAC]...[/TAC]
-/
[CTX]
import Mathlib.Data.Nat.Prime

theorem test_thm (m n : Nat) (h : m.Coprime n) : m.gcd n = 1 := by
  
[/CTX]
[STATE]
m n : ℕ
h : Nat.Coprime m n
⊢ Nat.gcd m n = 1
[/STATE]
[TAC]

Example output

rw [Nat.Coprime] at h
[/TAC]

Citation

Please cite:

@misc{hu2024minictx,
  author = {Jiewen Hu and Thomas Zhu and Sean Welleck},
  title = {miniCTX: Neural Theorem Proving with (Long-)Contexts},
  year = {2024},
  eprint={2408.03350},
  archivePrefix={arXiv},
}