jrcedeno commited on
Commit
0ed9146
Β·
verified Β·
1 Parent(s): c76a96f

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +92 -0
README.md ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - neural-rendering
5
+ - real-time-graphics
6
+ - game-engine
7
+ - pytorch
8
+ - onnx
9
+ ---
10
+
11
+ # Uruk Neural Renderer
12
+
13
+ A multi-model neural rendering pipeline designed for real-time game graphics, trained on NVIDIA B200 GPUs. The Uruk system uses a modular workstream architecture where specialized models handle different aspects of the rendering pipeline β€” from world modeling and scene remapping to cinematic rendering and runtime optimization.
14
+
15
+ ## Architecture Overview
16
+
17
+ The Uruk Neural Renderer is organized into a multi-workstream pipeline, where each workstream trains a specialized model family. A policy-compliant orchestrator manages the 4-stage training lifecycle: **Smoke** (bug-catching), **Calibration** (hyperparameter tuning), **Production** (full training with early stopping), and **Distillation** (teacher-to-student compression).
18
+
19
+ The flagship **V2-Ultra** model uses a 32-channel input (including a 12-channel G-buffer with material IDs, depth, and normals) and achieves **94.6% material accuracy** β€” enabling physically-correct lighting decisions based on ground-truth geometry rather than screen-space inference.
20
+
21
+ ## Models
22
+
23
+ ### V2-Ultra (Neural Renderer V2)
24
+
25
+ The primary neural renderer with a 6.9M-parameter student model and 37.4M-parameter teacher. Trained with a 2-stage curriculum on the Frontier dataset.
26
+
27
+ | File | Description | Size |
28
+ |---|---|---|
29
+ | `v2_ultra/v2_ultra_global_best.pt` | Global best checkpoint (student) | 26.4 MB |
30
+ | `v2_ultra/v2_ultra_best_stage1.pt` | Best Stage 1 (foundation) checkpoint | 79.3 MB |
31
+ | `v2_ultra/v2_ultra_best_stage2.pt` | Best Stage 2 (fine-tune) checkpoint | 79.3 MB |
32
+ | `onnx/uruk_v2_ultra_best.onnx` | ONNX export for deployment | 26.4 MB |
33
+
34
+ ### V2-Optimized (Reconstruction-First Approach)
35
+
36
+ An improved training run that initializes from V1 weights and uses a 3-stage curriculum (rendering, material, optional GAN) with MS-SSIM loss and quality gates.
37
+
38
+ | File | Description | Size |
39
+ |---|---|---|
40
+ | `v2_optimized/v2opt_global_best.pt` | Global best checkpoint | 26.4 MB |
41
+
42
+ ### Workstream Production Models
43
+
44
+ Best checkpoints from the policy-compliant orchestrator production runs.
45
+
46
+ | File | Workstream | Description | Size |
47
+ |---|---|---|---|
48
+ | `workstreams/ws2_learned_world/ws2_best.pt` | WS2 β€” Learned World Model (Family D) | Learns world dynamics and physics | 59.4 MB |
49
+ | `workstreams/ws3_world_authoring/ws3_best.pt` | WS3 β€” World Authoring (Family B) | Procedural world generation from embeddings | 103.3 MB |
50
+ | `workstreams/ws4_world_remapper/ws4_best.pt` | WS4 β€” World Remapper (Family C) | Scene-to-scene transformation | 259.3 MB |
51
+ | `workstreams/ws4_world_remapper_v2/ws4_v2_best.pt` | WS4 v2 β€” World Remapper (optimized rerun) | Completed all 500 epochs | 259.3 MB |
52
+ | `workstreams/ws5_cinematic_renderer/ws5_frontier_best.pt` | WS5 β€” Cinematic Renderer (Family I) | Rich G-buffer rendering (19-ch input, 10.1M params) | 116.2 MB |
53
+ | `workstreams/ws6_runtime_optimization/ws6_best.pt` | WS6 β€” Runtime Optimization (Family G) | Inference speed optimization | 4.4 MB |
54
+ | `workstreams/ws6_runtime_optimization_v2/ws6_v2_best.pt` | WS6 v2 β€” Runtime Optimization (rerun) | Completed all 500 epochs | 13.7 MB |
55
+ | `workstreams/ws7_scene_to_world_v2/ws7_v2_best.pt` | WS7 v2 β€” Scene to World | Graph-based scene understanding | 43.7 MB |
56
+
57
+ ### Additional Models
58
+
59
+ | File | Description | Size |
60
+ |---|---|---|
61
+ | `npc_director_v2/best.pt` | NPC Director v2 β€” behavioral AI for NPC state management (99.57% state accuracy) | 15.0 MB |
62
+ | `animation_director/best.pt` | Animation Director β€” procedural animation control | 8.0 MB |
63
+ | `ws8_structure_generator/best_model.pt` | WS8 β€” Structure Generator | 32.4 MB |
64
+ | `onnx/npc_director.onnx` | NPC Director ONNX export | 5.1 MB |
65
+
66
+ ### Distillation Students
67
+
68
+ | File | Description | Size |
69
+ |---|---|---|
70
+ | `distillation/ws5_student_best.pt` | WS5 Cinematic Renderer distilled student (best val_loss: 0.236) | 38.7 MB |
71
+
72
+ ## Training Infrastructure
73
+
74
+ All models were trained on a single **NVIDIA B200** GPU (180 GB VRAM) on RunPod. The multi-workstream orchestrator managed sequential training across all workstreams with automatic stage transitions, early stopping, and plateau detection.
75
+
76
+ ## Usage
77
+
78
+ ```python
79
+ import torch
80
+
81
+ # Load a checkpoint
82
+ checkpoint = torch.load("v2_ultra/v2_ultra_global_best.pt", map_location="cpu")
83
+ model_state = checkpoint["model_state_dict"]
84
+
85
+ # For ONNX inference
86
+ import onnxruntime as ort
87
+ session = ort.InferenceSession("onnx/uruk_v2_ultra_best.onnx")
88
+ ```
89
+
90
+ ## License
91
+
92
+ Apache 2.0