Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
tags:
|
| 4 |
+
- neural-rendering
|
| 5 |
+
- real-time-graphics
|
| 6 |
+
- game-engine
|
| 7 |
+
- pytorch
|
| 8 |
+
- onnx
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
# Uruk Neural Renderer
|
| 12 |
+
|
| 13 |
+
A multi-model neural rendering pipeline designed for real-time game graphics, trained on NVIDIA B200 GPUs. The Uruk system uses a modular workstream architecture where specialized models handle different aspects of the rendering pipeline β from world modeling and scene remapping to cinematic rendering and runtime optimization.
|
| 14 |
+
|
| 15 |
+
## Architecture Overview
|
| 16 |
+
|
| 17 |
+
The Uruk Neural Renderer is organized into a multi-workstream pipeline, where each workstream trains a specialized model family. A policy-compliant orchestrator manages the 4-stage training lifecycle: **Smoke** (bug-catching), **Calibration** (hyperparameter tuning), **Production** (full training with early stopping), and **Distillation** (teacher-to-student compression).
|
| 18 |
+
|
| 19 |
+
The flagship **V2-Ultra** model uses a 32-channel input (including a 12-channel G-buffer with material IDs, depth, and normals) and achieves **94.6% material accuracy** β enabling physically-correct lighting decisions based on ground-truth geometry rather than screen-space inference.
|
| 20 |
+
|
| 21 |
+
## Models
|
| 22 |
+
|
| 23 |
+
### V2-Ultra (Neural Renderer V2)
|
| 24 |
+
|
| 25 |
+
The primary neural renderer with a 6.9M-parameter student model and 37.4M-parameter teacher. Trained with a 2-stage curriculum on the Frontier dataset.
|
| 26 |
+
|
| 27 |
+
| File | Description | Size |
|
| 28 |
+
|---|---|---|
|
| 29 |
+
| `v2_ultra/v2_ultra_global_best.pt` | Global best checkpoint (student) | 26.4 MB |
|
| 30 |
+
| `v2_ultra/v2_ultra_best_stage1.pt` | Best Stage 1 (foundation) checkpoint | 79.3 MB |
|
| 31 |
+
| `v2_ultra/v2_ultra_best_stage2.pt` | Best Stage 2 (fine-tune) checkpoint | 79.3 MB |
|
| 32 |
+
| `onnx/uruk_v2_ultra_best.onnx` | ONNX export for deployment | 26.4 MB |
|
| 33 |
+
|
| 34 |
+
### V2-Optimized (Reconstruction-First Approach)
|
| 35 |
+
|
| 36 |
+
An improved training run that initializes from V1 weights and uses a 3-stage curriculum (rendering, material, optional GAN) with MS-SSIM loss and quality gates.
|
| 37 |
+
|
| 38 |
+
| File | Description | Size |
|
| 39 |
+
|---|---|---|
|
| 40 |
+
| `v2_optimized/v2opt_global_best.pt` | Global best checkpoint | 26.4 MB |
|
| 41 |
+
|
| 42 |
+
### Workstream Production Models
|
| 43 |
+
|
| 44 |
+
Best checkpoints from the policy-compliant orchestrator production runs.
|
| 45 |
+
|
| 46 |
+
| File | Workstream | Description | Size |
|
| 47 |
+
|---|---|---|---|
|
| 48 |
+
| `workstreams/ws2_learned_world/ws2_best.pt` | WS2 β Learned World Model (Family D) | Learns world dynamics and physics | 59.4 MB |
|
| 49 |
+
| `workstreams/ws3_world_authoring/ws3_best.pt` | WS3 β World Authoring (Family B) | Procedural world generation from embeddings | 103.3 MB |
|
| 50 |
+
| `workstreams/ws4_world_remapper/ws4_best.pt` | WS4 β World Remapper (Family C) | Scene-to-scene transformation | 259.3 MB |
|
| 51 |
+
| `workstreams/ws4_world_remapper_v2/ws4_v2_best.pt` | WS4 v2 β World Remapper (optimized rerun) | Completed all 500 epochs | 259.3 MB |
|
| 52 |
+
| `workstreams/ws5_cinematic_renderer/ws5_frontier_best.pt` | WS5 β Cinematic Renderer (Family I) | Rich G-buffer rendering (19-ch input, 10.1M params) | 116.2 MB |
|
| 53 |
+
| `workstreams/ws6_runtime_optimization/ws6_best.pt` | WS6 β Runtime Optimization (Family G) | Inference speed optimization | 4.4 MB |
|
| 54 |
+
| `workstreams/ws6_runtime_optimization_v2/ws6_v2_best.pt` | WS6 v2 β Runtime Optimization (rerun) | Completed all 500 epochs | 13.7 MB |
|
| 55 |
+
| `workstreams/ws7_scene_to_world_v2/ws7_v2_best.pt` | WS7 v2 β Scene to World | Graph-based scene understanding | 43.7 MB |
|
| 56 |
+
|
| 57 |
+
### Additional Models
|
| 58 |
+
|
| 59 |
+
| File | Description | Size |
|
| 60 |
+
|---|---|---|
|
| 61 |
+
| `npc_director_v2/best.pt` | NPC Director v2 β behavioral AI for NPC state management (99.57% state accuracy) | 15.0 MB |
|
| 62 |
+
| `animation_director/best.pt` | Animation Director β procedural animation control | 8.0 MB |
|
| 63 |
+
| `ws8_structure_generator/best_model.pt` | WS8 β Structure Generator | 32.4 MB |
|
| 64 |
+
| `onnx/npc_director.onnx` | NPC Director ONNX export | 5.1 MB |
|
| 65 |
+
|
| 66 |
+
### Distillation Students
|
| 67 |
+
|
| 68 |
+
| File | Description | Size |
|
| 69 |
+
|---|---|---|
|
| 70 |
+
| `distillation/ws5_student_best.pt` | WS5 Cinematic Renderer distilled student (best val_loss: 0.236) | 38.7 MB |
|
| 71 |
+
|
| 72 |
+
## Training Infrastructure
|
| 73 |
+
|
| 74 |
+
All models were trained on a single **NVIDIA B200** GPU (180 GB VRAM) on RunPod. The multi-workstream orchestrator managed sequential training across all workstreams with automatic stage transitions, early stopping, and plateau detection.
|
| 75 |
+
|
| 76 |
+
## Usage
|
| 77 |
+
|
| 78 |
+
```python
|
| 79 |
+
import torch
|
| 80 |
+
|
| 81 |
+
# Load a checkpoint
|
| 82 |
+
checkpoint = torch.load("v2_ultra/v2_ultra_global_best.pt", map_location="cpu")
|
| 83 |
+
model_state = checkpoint["model_state_dict"]
|
| 84 |
+
|
| 85 |
+
# For ONNX inference
|
| 86 |
+
import onnxruntime as ort
|
| 87 |
+
session = ort.InferenceSession("onnx/uruk_v2_ultra_best.onnx")
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
## License
|
| 91 |
+
|
| 92 |
+
Apache 2.0
|