Instructions to use ornith-ai/Ornith-1.0-35B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ornith-ai/Ornith-1.0-35B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ornith-ai/Ornith-1.0-35B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ornith-ai/Ornith-1.0-35B")
model = AutoModelForMultimodalLM.from_pretrained("ornith-ai/Ornith-1.0-35B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ornith-ai/Ornith-1.0-35B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ornith-ai/Ornith-1.0-35B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ornith-ai/Ornith-1.0-35B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ornith-ai/Ornith-1.0-35B

SGLang

How to use ornith-ai/Ornith-1.0-35B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ornith-ai/Ornith-1.0-35B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ornith-ai/Ornith-1.0-35B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ornith-ai/Ornith-1.0-35B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ornith-ai/Ornith-1.0-35B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ornith-ai/Ornith-1.0-35B with Docker Model Runner:
```
docker model run hf.co/ornith-ai/Ornith-1.0-35B
```

VLLM load {1 x a100 80 VRAM} ( ERROR 35b ) vs transformers {1 x a100 80 VRAM} ( load model OK 9b and 35b)

#18

by tepirale - opened Jun 29

Discussion

tepirale

Jun 29

•

edited Jun 30

First, congratulations on the models, thank you very much for releasing them.
Now, while testing models 9b and 35b, I've only had problems loading the model 35B in VLLM, but I haven't had any problems with the Transformers library. It's only with VLLM that I'm having trouble.

I've also tested it with DFLASH 9b (model deepreinforce-ai/Ornith-1.0-9B ) and it works quickly.

===== METRICS 1 =====
Tokens de OUTPUT: 2698
TTFT : 3.716 s
TITIME GENERATION: 11.346 s
TIME TOTAL: 15.062 s
Speed: 237.79 tok/s

===== METRICS 2 =====
Tokens de OUTPUT: 52430
TTFT : 8.141 s
TIME GENERATION: 1569.540 s
TIME TOTAL: 1577.681 s
Speed: 33.40 tok/s

1.1 I want to add that with vllm I can't make it reason (thinkg)

The model (deepreinforce-ai/Ornith-1.0-35B) loads only up to 70 GB of V-RAM and then displays the error shown below.

I'll provide you with my python code, serve_ornith_vllm.sh code file and how I run it.

!uv pip install --system --force-reinstall -U --pre vllm \
  --extra-index-url https://wheels.vllm.ai/nightly/cu129 \
  --extra-index-url https://download.pytorch.org/whl/cu129 \
  --index-strategy unsafe-best-match

!python -c "import vllm; print(vllm.__version__)"
0.23.1rc1.dev578+g72f639927

code serve_ornith_vllm.sh

#!/usr/bin/env bash
# ---------------------------------------------------------------------------
# serve_ornith_vllm.sh  (versión DETACHED)
# Levanta Ornith-1.0 con vLLM en segundo plano (OpenAI-compatible).
#
# Igual que el serve de Gemma 4: nohup + disown -> el servidor sobrevive aunque
# cierres la shell o termine este script. Guarda el PID y espera a que el
# endpoint /health responda antes de devolverte el control.
#
# Ornith es un modelo de razonamiento (<think>...</think>) con tool-calling
# estilo Qwen3, así que activamos --reasoning-parser qwen3 y --tool-call-parser
# qwen3_xml para obtener reasoning_content y tool_calls.
#
# Uso:
#   chmod +x serve_ornith_vllm.sh stop_ornith.sh
#   ./serve_ornith_vllm.sh           # arranca en background y espera al health
#   ./stop_ornith.sh                 # lo apaga
#
# El número de GPUs (tensor-parallel-size) se DETECTA AUTOMÁTICAMENTE.
#
# Variables de entorno configurables (con sus valores por defecto):
#   MODEL          deepreinforce-ai/Ornith-1.0-9B
#   SERVED_NAME    Ornith-1.0-9B
#   TP_SIZE        (auto-detectado)  -> fuerza un valor para sobreescribir
#   PORT           8000
#   HOST           0.0.0.0
#   MAX_LEN        262144   (bajalo si te quedas sin VRAM/KV cache, p.ej. 16384)
#   GPU_UTIL       0.90
#   API_KEY        EMPTY    (token que exigira el server; cliente debe enviarlo)
#   LOG            ornith_vllm.log
#   PIDFILE        ornith_server.pid
#   SHOW_PROGRESS  1        (=1 muestra el log en vivo al cargar; =0 silencioso)
#
# Ejemplos:
#   ./serve_ornith_vllm.sh                       # usa TODAS las GPUs visibles
#   TP_SIZE=2 ./serve_ornith_vllm.sh             # fuerza solo 2 GPUs
#   MAX_LEN=16384 ./serve_ornith_vllm.sh         # recorta contexto (recomendado en 1 GPU)
#   SHOW_PROGRESS=0 ./serve_ornith_vllm.sh       # arranque silencioso
# ---------------------------------------------------------------------------
set -euo pipefail

# ---------------------------------------------------------------------------
# Deteccion automatica de GPUs
# Prioridad: 1) TP_SIZE manual  2) CUDA_VISIBLE_DEVICES  3) nvidia-smi
# ---------------------------------------------------------------------------
detect_gpus() {
    if [[ -n "${CUDA_VISIBLE_DEVICES:-}" ]]; then
        echo "${CUDA_VISIBLE_DEVICES}" | tr ',' '\n' | grep -cE '^[0-9]+$' || echo 0
        return
    fi
    if command -v nvidia-smi >/dev/null 2>&1; then
        nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | grep -c . || echo 0
        return
    fi
    echo 0
}

if [[ -n "${TP_SIZE:-}" ]]; then
    echo "[info] TP_SIZE forzado manualmente: ${TP_SIZE}"
else
    NUM_GPUS="$(detect_gpus)"
    if [[ "${NUM_GPUS}" -lt 1 ]]; then
        echo "[error] No se detecto ninguna GPU NVIDIA." >&2
        echo "        vLLM requiere GPU. Verifica los drivers con 'nvidia-smi'." >&2
        echo "        Si quieres forzar un valor de todos modos: TP_SIZE=1 ./serve_ornith_vllm.sh" >&2
        exit 1
    fi
    TP_SIZE="${NUM_GPUS}"
    echo "[info] GPUs detectadas: ${NUM_GPUS} -> tensor-parallel-size=${TP_SIZE}"

    if (( TP_SIZE > 1 )) && (( (TP_SIZE & (TP_SIZE - 1)) != 0 )); then
        echo "[warn] ${TP_SIZE} no es potencia de 2; si vLLM falla al cargar," >&2
        echo "       fuerza una potencia de 2 (p.ej. TP_SIZE=2 o TP_SIZE=4)." >&2
    fi
fi

# ---------------------------------------------------------------------------
# Configuracion del modelo / servidor
# ---------------------------------------------------------------------------
# MODEL="${MODEL:-deepreinforce-ai/Ornith-1.0-9B}"
# SERVED_NAME="${SERVED_NAME:-Ornith-1.0-9B}"
MODEL="${MODEL:-deepreinforce-ai/Ornith-1.0-35B}"
SERVED_NAME="${SERVED_NAME:-Ornith-1.0-35B}"
PORT="${PORT:-8000}"
HOST="${HOST:-0.0.0.0}"
MAX_LEN="${MAX_LEN:-262144}"
GPU_UTIL="${GPU_UTIL:-0.90}"
API_KEY="${API_KEY:-EMPTY}"
LOG="${LOG:-ornith_vllm.log}"
PIDFILE="${PIDFILE:-ornith_server.pid}"
SHOW_PROGRESS="${SHOW_PROGRESS:-1}"

# Evita arrancar dos veces sobre el mismo PIDFILE
if [ -f "$PIDFILE" ] && kill -0 "$(cat "$PIDFILE")" 2>/dev/null; then
    echo "[error] Ya hay un servidor corriendo (PID $(cat "$PIDFILE"), $PIDFILE)." >&2
    echo "        Apagalo primero con ./stop_ornith.sh" >&2
    exit 1
fi

echo "============================================================"
echo "  Sirviendo: ${MODEL}"
echo "  Nombre expuesto: ${SERVED_NAME}"
echo "  GPUs (TP): ${TP_SIZE}   |   Contexto max: ${MAX_LEN}"
echo "  Endpoint: http://${HOST}:${PORT}/v1   |   Log: ${LOG}"
echo "============================================================"

export VLLM_API_KEY="${API_KEY}"

# ─── Chat template oficial de Ornith (dispara el modo <think>) ───
TEMPLATE="${TEMPLATE:-ornith_chat_template.jinja}"
# TEMPLATE_URL="${TEMPLATE_URL:-https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B/resolve/main/chat_template.jinja}"
TEMPLATE_URL="${TEMPLATE_URL:-https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B/resolve/main/chat_template.jinja}"
if [ ! -f "$TEMPLATE" ]; then
  echo "Descargando chat template de Ornith..."
  curl -sfL -o "$TEMPLATE" "$TEMPLATE_URL" \
    || { echo "ERROR: no se pudo descargar el template"; exit 1; }
fi
TEMPLATE_PATH="$(pwd)/$TEMPLATE"



# ---------------------------------------------------------------------------
# Arranque DETACHED (sobrevive al cierre de la shell)
# ---------------------------------------------------------------------------
nohup vllm serve "${MODEL}" \
    --served-model-name "${SERVED_NAME}" \
    --tensor-parallel-size "${TP_SIZE}" \
    --host "${HOST}" --port "${PORT}" \
    --max-model-len "${MAX_LEN}" \
    --gpu-memory-utilization "${GPU_UTIL}" \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3 \
    --chat-template "${TEMPLATE_PATH}" \
    --default-chat-template-kwargs '{"enable_thinking": true}' \
    --trust-remote-code \
    --attention-backend TRITON_ATTN \
    # --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.5-9B-DFlash", "num_speculative_tokens": 8, "draft_tensor_parallel_size": 1}' \
    # --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8, "draft_tensor_parallel_size": 1}' \
     --limit-mm-per-prompt '{"image":4,"video":0}' \
     --max-num-batched-tokens 4176 \
     --max-num-seqs 10 \
    --api-key "${API_KEY}" \
    > "${LOG}" 2>&1 &

SERVER_PID=$!
echo "${SERVER_PID}" > "${PIDFILE}"
disown
echo "Servidor lanzado (PID ${SERVER_PID}, guardado en ${PIDFILE}). Logs: ${LOG}"

# ---------------------------------------------------------------------------
# Progreso de carga en vivo (opcional). tqdm de vLLM escribe en el log.
# ---------------------------------------------------------------------------
TAIL_PID=""
if [ "$SHOW_PROGRESS" = "1" ]; then
    echo "── Progreso de carga (SHOW_PROGRESS=1) ───────────────────────"
    tail -f "$LOG" &
    TAIL_PID=$!
fi
stop_tail() { [ -n "$TAIL_PID" ] && kill "$TAIL_PID" 2>/dev/null || true; TAIL_PID=""; }

# ---------------------------------------------------------------------------
# Espera a que cargue el modelo (/health no requiere API key)
# ---------------------------------------------------------------------------
[ "$SHOW_PROGRESS" = "1" ] || echo "Esperando a que el modelo cargue (puede tardar varios minutos)..."
until curl -sf "http://localhost:${PORT}/health" > /dev/null 2>&1; do
    if ! kill -0 "$SERVER_PID" 2>/dev/null; then
        stop_tail
        echo "ERROR: el servidor murió durante el arranque. Últimas líneas:"
        tail -n 40 "$LOG"
        rm -f "$PIDFILE"
        exit 1
    fi
    sleep 3
done

stop_tail
[ "$SHOW_PROGRESS" = "1" ] && echo "──────────────────────────────────────────────────────────────"

echo "OK: servidor listo en http://localhost:${PORT}/v1 (sigue corriendo en segundo plano)"
echo "  Modelo expuesto:    ${SERVED_NAME}"
echo "  Apaga el servidor:  ./stop_ornith.sh   (o: kill \$(cat ${PIDFILE}))"

ERRORS LOAD MODEL,

root@d2d57e80eb10:/workspace# MAX_LEN=256000 SHOW_PROGRESS=1 ./serve_ornith_vllm.sh
[info] GPUs detectadas: 1 -> tensor-parallel-size=1
============================================================
  Sirviendo: deepreinforce-ai/Ornith-1.0-35B
  Nombre expuesto: Ornith-1.0-35B
  GPUs (TP): 1   |   Contexto max: 256000
  Endpoint: http://0.0.0.0:8000/v1   |   Log: ornith_vllm.log
============================================================
Servidor lanzado (PID 4980, guardado en ornith_server.pid). Logs: ornith_vllm.log
── Progreso de carga (SHOW_PROGRESS=1) ───────────────────────
(APIServer pid=4980) INFO 06-29 22:26:50 [api_utils.py:339] 
(APIServer pid=4980) INFO 06-29 22:26:50 [api_utils.py:339]        █     █     █▄   ▄█
(APIServer pid=4980) INFO 06-29 22:26:50 [api_utils.py:339]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.23.1rc1.dev578+g72f639927
(APIServer pid=4980) INFO 06-29 22:26:50 [api_utils.py:339]   █▄█▀ █     █     █     █  model   deepreinforce-ai/Ornith-1.0-35B
(APIServer pid=4980) INFO 06-29 22:26:50 [api_utils.py:339]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=4980) INFO 06-29 22:26:50 [api_utils.py:339] 
(APIServer pid=4980) INFO 06-29 22:26:50 [api_utils.py:273] non-default args: {'model_tag': 'deepreinforce-ai/Ornith-1.0-35B', 'chat_template': '/workspace/ornith_chat_template.jinja', 'default_chat_template_kwargs': {'enable_thinking': True}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_xml', 'host': '0.0.0.0', 'api_key': ['EMPTY'], 'model': 'deepreinforce-ai/Ornith-1.0-35B', 'trust_remote_code': True, 'max_model_len': 256000, 'served_model_name': ['Ornith-1.0-35B'], 'attention_backend': 'TRITON_ATTN', 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.9, 'enable_prefix_caching': True, 'limit_mm_per_prompt': {'image': 4, 'video': 0}, 'max_num_batched_tokens': 4176, 'max_num_seqs': 10}
(APIServer pid=4980) INFO 06-29 22:26:51 [model.py:601] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=4980) INFO 06-29 22:26:51 [model.py:1731] Using max model len 256000
(APIServer pid=4980) INFO 06-29 22:26:51 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=4176.
(APIServer pid=4980) WARNING 06-29 22:26:51 [config.py:555] Mamba cache mode is set to 'align' for Qwen3_5MoeForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=4980) INFO 06-29 22:26:51 [config.py:575] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=4980) INFO 06-29 22:26:51 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=4980) INFO 06-29 22:26:51 [kernel.py:278] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=4980) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=5322) INFO 06-29 22:27:19 [core.py:114] Initializing a V1 LLM engine (v0.23.1rc1.dev578+g72f639927) with config: model='deepreinforce-ai/Ornith-1.0-35B', speculative_config=None, tokenizer='deepreinforce-ai/Ornith-1.0-35B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=256000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=Ornith-1.0-35B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4176], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=5322) INFO 06-29 22:27:23 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.25.0.2:55593 backend=nccl
(EngineCore pid=5322) INFO 06-29 22:27:23 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=5322) INFO 06-29 22:27:24 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=5322) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=5322) INFO 06-29 22:27:37 [gpu_model_runner.py:5175] Starting to load model deepreinforce-ai/Ornith-1.0-35B...
(EngineCore pid=5322) INFO 06-29 22:27:37 [cuda.py:542] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=5322) INFO 06-29 22:27:37 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=5322) INFO 06-29 22:27:37 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=5322) INFO 06-29 22:27:37 [unquantized.py:260] Using TRITON Unquantized MoE backend out of potential backends: ['FlashInfer TRTLLM', 'FlashInfer CUTLASS', 'TRITON', 'BATCHED_TRITON'].
(EngineCore pid=5322) INFO 06-29 22:27:37 [cuda.py:423] Using AttentionBackendEnum.TRITON_ATTN backend.
(EngineCore pid=5322) <frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore pid=5322) <frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore pid=5322) INFO 06-29 22:27:38 [weight_utils.py:849] Filesystem type for checkpoints: OVERLAY. Checkpoint size: 65.40 GiB. Available RAM: 819.81 GiB.
(EngineCore pid=5322) INFO 06-29 22:27:38 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (OVERLAY) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:   0% Completed | 0/16 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   6% Completed | 1/16 [00:00<00:10,  1.41it/s]
Loading safetensors checkpoint shards:  12% Completed | 2/16 [00:02<00:14,  1.05s/it]
Loading safetensors checkpoint shards:  19% Completed | 3/16 [00:03<00:16,  1.26s/it]
Loading safetensors checkpoint shards:  25% Completed | 4/16 [00:04<00:15,  1.26s/it]
Loading safetensors checkpoint shards:  31% Completed | 5/16 [00:06<00:14,  1.29s/it]
Loading safetensors checkpoint shards:  38% Completed | 6/16 [00:07<00:13,  1.35s/it]
Loading safetensors checkpoint shards:  44% Completed | 7/16 [00:08<00:11,  1.32s/it]
Loading safetensors checkpoint shards:  50% Completed | 8/16 [00:10<00:10,  1.31s/it]
Loading safetensors checkpoint shards:  56% Completed | 9/16 [00:11<00:09,  1.29s/it]
Loading safetensors checkpoint shards:  62% Completed | 10/16 [00:12<00:07,  1.22s/it]
Loading safetensors checkpoint shards:  69% Completed | 11/16 [00:13<00:05,  1.17s/it]
Loading safetensors checkpoint shards:  75% Completed | 12/16 [00:14<00:04,  1.18s/it]
Loading safetensors checkpoint shards:  81% Completed | 13/16 [00:15<00:03,  1.14s/it]
Loading safetensors checkpoint shards:  88% Completed | 14/16 [00:16<00:02,  1.13s/it]
Loading safetensors checkpoint shards:  94% Completed | 15/16 [00:18<00:01,  1.17s/it]
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:18<00:00,  1.04s/it]
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:18<00:00,  1.18s/it]
(EngineCore pid=5322) 
(EngineCore pid=5322) INFO 06-29 22:27:57 [default_loader.py:430] Loading weights took 18.99 seconds
(EngineCore pid=5322) INFO 06-29 22:27:57 [unquantized.py:325] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=5322) INFO 06-29 22:27:58 [gpu_model_runner.py:5272] Model loading took 65.53 GiB memory and 20.152452 seconds
(EngineCore pid=5322) INFO 06-29 22:27:58 [interface.py:773] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=5322) INFO 06-29 22:27:58 [interface.py:797] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=5322) INFO 06-29 22:27:58 [gpu_model_runner.py:6288] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=5322) INFO 06-29 22:28:08 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/3d8ba1d8ca/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=5322) INFO 06-29 22:28:08 [backends.py:1148] Dynamo bytecode transform time: 7.68 s
(EngineCore pid=5322) INFO 06-29 22:28:10 [backends.py:378] Cache the graph of compile range (1, 4176) for later use
(EngineCore pid=5322) INFO 06-29 22:28:45 [backends.py:393] Compiling a graph for compile range (1, 4176) takes 36.87 s
(EngineCore pid=5322) INFO 06-29 22:28:49 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/763aa92bca22b1ea43b1dc3ba7d0785276fbc62241cac5791179cfb672fdc771/rank_0_0/model
(EngineCore pid=5322) INFO 06-29 22:28:49 [monitor.py:53] torch.compile took 48.81 s in total
(EngineCore pid=5322) WARNING 06-29 22:30:12 [fused_moe.py:1106] Using default MoE config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.11/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=256,N=512,device_name=NVIDIA_A100_80GB_PCIe.json
(EngineCore pid=5322) INFO 06-29 22:30:18 [monitor.py:81] Initial profiling/warmup run took 89.70 s
(EngineCore pid=5322) INFO 06-29 22:30:27 [gpu_model_runner.py:6500] Profiling CUDA graph memory: PIECEWISE=5 (largest=16), FULL=4 (largest=8)
(EngineCore pid=5322) INFO 06-29 22:30:35 [gpu_model_runner.py:6605] Estimated CUDA graph memory: 0.17 GiB total
(EngineCore pid=5322) INFO 06-29 22:30:36 [gpu_worker.py:515] Available KV cache memory: 3.56 GiB
(EngineCore pid=5322) INFO 06-29 22:30:36 [gpu_worker.py:530] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9000 is equivalent to --gpu-memory-utilization=0.8979 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9021. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231] EngineCore failed to start.
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231] Traceback (most recent call last):
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]   File "/usr/local/lib/python3.11/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]     return func(*args, **kwargs)
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core.py", line 966, in __init__
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]     super().__init__(
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core.py", line 133, in __init__
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]   File "/usr/local/lib/python3.11/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]     return func(*args, **kwargs)
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core.py", line 294, in _initialize_kv_caches
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]     kv_cache_configs = get_kv_cache_configs(
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]                        ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/core/kv_cache_utils.py", line 2101, in get_kv_cache_configs
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]     _check_enough_kv_cache_memory(
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/core/kv_cache_utils.py", line 760, in _check_enough_kv_cache_memory
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231]     raise ValueError(
(EngineCore pid=5322) ERROR 06-29 22:30:36 [core.py:1231] ValueError: To serve at least one request with the model's max seq len (256000), (5.02 GiB KV cache is needed, which is larger than the available KV cache memory (3.56 GiB). Based on the available memory, the estimated maximum model length is 179520. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_model_len` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.
(EngineCore pid=5322) Process EngineCore:
(EngineCore pid=5322) Traceback (most recent call last):
(EngineCore pid=5322)   File "/usr/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=5322)     self.run()
(EngineCore pid=5322)   File "/usr/lib/python3.11/multiprocessing/process.py", line 108, in run
(EngineCore pid=5322)     self._target(*self._args, **self._kwargs)
(EngineCore pid=5322)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core.py", line 1235, in run_engine_core
(EngineCore pid=5322)     raise e
(EngineCore pid=5322)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core
(EngineCore pid=5322)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=5322)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5322)   File "/usr/local/lib/python3.11/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=5322)     return func(*args, **kwargs)
(EngineCore pid=5322)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5322)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core.py", line 966, in __init__
(EngineCore pid=5322)     super().__init__(
(EngineCore pid=5322)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core.py", line 133, in __init__
(EngineCore pid=5322)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=5322)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5322)   File "/usr/local/lib/python3.11/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=5322)     return func(*args, **kwargs)
(EngineCore pid=5322)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5322)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core.py", line 294, in _initialize_kv_caches
(EngineCore pid=5322)     kv_cache_configs = get_kv_cache_configs(
(EngineCore pid=5322)                        ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=5322)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/core/kv_cache_utils.py", line 2101, in get_kv_cache_configs
(EngineCore pid=5322)     _check_enough_kv_cache_memory(
(EngineCore pid=5322)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/core/kv_cache_utils.py", line 760, in _check_enough_kv_cache_memory
(EngineCore pid=5322)     raise ValueError(
(EngineCore pid=5322) ValueError: To serve at least one request with the model's max seq len (256000), (5.02 GiB KV cache is needed, which is larger than the available KV cache memory (3.56 GiB). Based on the available memory, the estimated maximum model length is 179520. Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_model_len` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.
[rank0]:[W629 22:30:37.966267986 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=4980) Traceback (most recent call last):
(APIServer pid=4980)   File "/usr/local/bin/vllm", line 10, in <module>
(APIServer pid=4980)     sys.exit(main())
(APIServer pid=4980)              ^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/cli/main.py", line 95, in main
(APIServer pid=4980)     args.dispatch_function(args)
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/cli/serve.py", line 148, in cmd
(APIServer pid=4980)     uvloop.run(run_server(args))
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/uvloop/__init__.py", line 92, in run
(APIServer pid=4980)     return runner.run(wrapper())
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
(APIServer pid=4980)     return self._loop.run_until_complete(task)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=4980)     return await main
(APIServer pid=4980)            ^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 692, in run_server
(APIServer pid=4980)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 706, in run_server_worker
(APIServer pid=4980)     async with build_async_engine_client(
(APIServer pid=4980)   File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=4980)     return await anext(self.gen)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=4980)     async with build_async_engine_client_from_engine_args(
(APIServer pid=4980)   File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=4980)     return await anext(self.gen)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=4980)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=4980)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=4980)     return cls(
(APIServer pid=4980)            ^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=4980)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=4980)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=4980)     return func(*args, **kwargs)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core_client.py", line 132, in make_async_mp_client
(APIServer pid=4980)     return AsyncMPClient(*client_args)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=4980)     return func(*args, **kwargs)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core_client.py", line 963, in __init__
(APIServer pid=4980)     super().__init__(
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core_client.py", line 573, in __init__
(APIServer pid=4980)     with launch_core_engines(
(APIServer pid=4980)   File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=4980)     next(self.gen)
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/utils.py", line 1213, in launch_core_engines
(APIServer pid=4980)     wait_for_engine_startup(
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/utils.py", line 1272, in wait_for_engine_startup
(APIServer pid=4980)     raise RuntimeError(
(APIServer pid=4980) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
ERROR: el servidor murió durante el arranque. Últimas líneas:
(APIServer pid=4980)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 706, in run_server_worker
(APIServer pid=4980)     async with build_async_engine_client(
(APIServer pid=4980)   File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=4980)     return await anext(self.gen)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=4980)     async with build_async_engine_client_from_engine_args(
(APIServer pid=4980)   File "/usr/lib/python3.11/contextlib.py", line 210, in __aenter__
(APIServer pid=4980)     return await anext(self.gen)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=4980)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=4980)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=4980)     return cls(
(APIServer pid=4980)            ^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=4980)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=4980)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=4980)     return func(*args, **kwargs)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core_client.py", line 132, in make_async_mp_client
(APIServer pid=4980)     return AsyncMPClient(*client_args)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=4980)     return func(*args, **kwargs)
(APIServer pid=4980)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core_client.py", line 963, in __init__
(APIServer pid=4980)     super().__init__(
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/core_client.py", line 573, in __init__
(APIServer pid=4980)     with launch_core_engines(
(APIServer pid=4980)   File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
(APIServer pid=4980)     next(self.gen)
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/utils.py", line 1213, in launch_core_engines
(APIServer pid=4980)     wait_for_engine_startup(
(APIServer pid=4980)   File "/usr/local/lib/python3.11/dist-packages/vllm/v1/engine/utils.py", line 1272, in wait_for_engine_startup
(APIServer pid=4980)     raise RuntimeError(
(APIServer pid=4980) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
root@d2d57e80eb10:/workspace#

Transformers OK

from transformers import AutoProcessor, AutoModelForImageTextToText
model_name = "deepreinforce-ai/Ornith-1.0-9B"
model_name = "deepreinforce-ai/Ornith-1.0-35B"


processor = AutoProcessor.from_pretrained(model_name)
tok= processor.tokenizer
model = AutoModelForImageTextToText.from_pretrained(
    model_name, dtype="auto", device_map="auto",
)

tepirale

Jun 30

nohup vllm serve "${MODEL}"
...
--kv-cache-dtype bfloat16
...

tepirale changed discussion status to closed Jun 30

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment