Upload USAGE_GUIDE.md with huggingface_hub

USAGE_GUIDE.md

@@ -0,0 +1,181 @@
+# Qwen3-0.6B GGUF Models
+
+This directory contains Qwen3-0.6B language model converted to GGUF format with multiple quantization levels optimized for CPU inference.
+
+## Available Model Variants
+
+| Variant | File Size | Use Case | Quality | Speed |
+|---------|-----------|----------|---------|-------|
+| **FP16** | 1.4 GB | Maximum accuracy, baseline reference | Highest | Slowest |
+| **Q8_0** | 767 MB | High-quality CPU inference | Very High | Medium |
+| **Q5_K_M** | 526 MB | Balanced quality and performance | High | Fast |
+| **Q4_K_M** | 462 MB | Edge devices, fastest inference | Good | Fastest |
+
+### Quantization Recommendations
+
+- **Q4_K_M**: Best for edge devices, mobile, or when speed is critical. Minimal quality loss for most tasks.
+- **Q5_K_M**: Recommended for production use. Excellent balance of quality and resource efficiency.
+- **Q8_0**: Use when quality is paramount and you have sufficient memory. Close to FP16 performance.
+- **FP16**: Reference model for validation and quality comparison. Use for benchmarking.
+
+## Model Specifications
+
+- **Architecture**: Qwen3 Causal Language Model
+- **Parameters**: 0.6B (Non-embedding: 0.44B)
+- **Layers**: 28
+- **Attention**: Grouped Query Attention (GQA) - 16 heads for Q, 8 heads for KV
+- **Context Length**: 32,768 tokens (40,960 in config)
+- **Vocabulary Size**: 151,936 tokens
+- **Special Features**:
+  - Thinking/non-thinking mode with `<think>...</think>` tags
+  - Multilingual support (100+ languages)
+  - Tool calling capabilities
+  - Enhanced reasoning
+
+## Usage Instructions
+
+### 1. Using llama.cpp CLI
+
+```bash
+# Basic text generation
+/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf -p "Hello, I am" -n 512 --temp 0.7
+
+# Interactive chat mode
+/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf --interactive-first --reverse-prompt "User:"
+
+# With GPU offloading (if available)
+/path/to/llama-cli -m qwen3-0.6b-q5_k_m.gguf -ngl 28 -p "Explain quantum computing"
+```
+
+### 2. Using Ollama
+
+Create a `Modelfile`:
+
+```dockerfile
+FROM ./qwen3-0.6b-q5_k_m.gguf
+
+PARAMETER temperature 0.7
+PARAMETER top_k 40
+PARAMETER top_p 0.9
+
+TEMPLATE """<|im_start|>system
+You are a helpful AI assistant.<|im_end|>
+<|im_start|>user
+{{ .Prompt }}<|im_end|>
+<|im_start|>assistant
+"""
+
+PARAMETER stop "<|im_start|>"
+PARAMETER stop "<|im_end|>"
+```
+
+Import and run:
+
+```bash
+# Create the model
+ollama create qwen3-0.6b -f Modelfile
+
+# Run the model
+ollama run qwen3-0.6b "What is machine learning?"
+```
+
+### 3. Using Python (llama-cpp-python)
+
+```python
+from llama_cpp import Llama
+
+# Initialize model
+llm = Llama(
+    model_path="qwen3-0.6b-q5_k_m.gguf",
+    n_ctx=2048,
+    n_threads=4,
+    n_gpu_layers=0,
+    verbose=False
+)
+
+# Generate text
+output = llm(
+    "Write a short poem about AI:",
+    max_tokens=256,
+    temperature=0.7,
+    stop=["<|im_end|>"]
+)
+
+print(output['choices'][0]['text'])
+```
+
+## Performance Characteristics
+
+### Inference Speed (approximate, CPU-dependent)
+
+| Model | Tokens/sec (2-core) | Tokens/sec (8-core) | Memory Usage |
+|-------|---------------------|---------------------|--------------|
+| Q4_K_M | ~15-20 | ~40-60 | ~800 MB |
+| Q5_K_M | ~12-18 | ~35-50 | ~900 MB |
+| Q8_0 | ~10-15 | ~30-40 | ~1.2 GB |
+| FP16 | ~8-12 | ~25-35 | ~1.8 GB |
+
+*Note: Actual performance depends on CPU architecture, cache size, and prompt complexity.*
+
+### Recommended CPU Configurations
+
+- **Minimum**: 2 cores, 2GB RAM - Use Q4_K_M
+- **Recommended**: 4 cores, 4GB RAM - Use Q5_K_M
+- **Optimal**: 8+ cores, 8GB RAM - Use Q8_0 or FP16
+
+## Context Length Management
+
+The model supports up to 32,768 tokens but uses less memory with smaller contexts:
+
+```bash
+# Short context (faster, less memory)
+llama-cli -m qwen3-0.6b-q5_k_m.gguf --ctx-size 2048
+
+# Long context (slower, more memory)
+llama-cli -m qwen3-0.6b-q5_k_m.gguf --ctx-size 32768
+```
+
+## Multilingual Support
+
+Qwen3-0.6B supports 100+ languages including English, Chinese, Spanish, French, German, Japanese, Korean, and many more.
+
+## Troubleshooting
+
+### Model fails to load
+- **Solution**: Use smaller quantization (Q4_K_M) or reduce context size
+
+### Slow generation
+- **Solution**: Increase thread count, use Q4_K_M, or reduce batch size
+
+### Poor quality outputs
+- **Solution**: Use higher quantization (Q8_0 or FP16), adjust temperature
+
+## Conversion Details
+
+- **Source**: Qwen/Qwen3-0.6B from Hugging Face Hub
+- **Conversion Tool**: llama.cpp convert_hf_to_gguf.py
+- **Base Format**: FP16 (converted from BF16)
+- **Quantization Tool**: llama-quantize
+- **Validated**: All models tested for loading and inference
+
+## License
+
+This model follows the Apache 2.0 license from the original Qwen3-0.6B model.
+
+**Original Model**: https://huggingface.co/Qwen/Qwen3-0.6B
+
+## Support & Resources
+
+- **llama.cpp Documentation**: https://github.com/ggerganov/llama.cpp
+- **Ollama Documentation**: https://ollama.ai/
+- **Qwen3 Model Card**: https://huggingface.co/Qwen/Qwen3-0.6B
+
+## Version Information
+
+- **Conversion Date**: 2024-12-17
+- **llama.cpp Version**: Build 7451 (669696e00)
+- **GGUF Version**: V3 (latest)
+
+---
+
+**Note**: Performance metrics are approximate and will vary based on hardware. Test different quantization levels to find the optimal balance for your use case.