--- license: other pipeline_tag: image-to-image tags: - android - neural-graphics - gaming - graphics language: - en --- # Neural Super Sampling (NSS)
⚡ NSS v1 is now available
• New architecture and trained weights.
• Higher image quality compared to NSS v0.1, with comparable runtime performance.
• Three quality modes (High, Mid, and Low) let you balance image quality and performance.
• Updated example scenarios using fragment shaders.
Neural Super Sampling (NSS) is an innovative, efficient network for temporal super sampling on mobile devices. Content rendered at a low resolution can be upscaled to a higher resolution, resulting in up to 50% GPU savings. With our [retraining tools](https://github.com/arm/neural-graphics-model-gym) content creators and game studios can build derivatives of the model suited to artwork style and performance requirements. ## 🎥 Neural Super Sampling Demo ## Model Details Neural Super Sampling (NSS) is a parameter prediction model for real-time temporal super sampling developed by Arm, optimized for execution on Neural Accelerators (NX) in mobile GPUs. It enables high-resolution rendering at a lower compute cost by reconstructing high-quality output frames from low-resolution temporal inputs. NSS is particularly suited for mobile gaming, XR, and other power-constrained graphics use cases. NSS v1 provides three quality modes that allow you to balance image quality and runtime performance: * `high`: highest runtime cost but best-quality option. Checks the current frame and depth/motion detail more thoroughly, uses a larger image filter for clean-up, and samples previous frames more accurately. * `low`: lowest runtime cost but lowest-quality option. Uses lighter current-frame and depth/motion checks, and samples previous frames less accurately. May show more flicker or motion artifacts around fine detail and moving objects. * `mid`: balanced option. Similar to `low` but samples previous frames more accurately, like `high`. For a more detailed explanation of these quality modes, refer to the [Neural Graphics Model Gym documentation](https://github.com/arm/neural-graphics-model-gym/blob/main/docs/nss/nss_configuration.md).
Developed ByArm Limited
Model TypeTemporal image super sampling
LicenseOther
RepositoryNeural Graphics Model Gym
BlogHow Neural Super Sampling Works
PaperMobile Neural Super Sampling
Quickstart with ML Extensions for Vulkan®ML Extensions for Vulkan® Learning Path
Quickstart with Unreal®NSS integration into Unreal® Engine Learning Path
NSS is under active development with regular updates planned. As we increase the size and diversity of the training dataset we expect to see significant quality improvements. Follow Arm to stay up to date on the latest releases. The model is released under Arm's permissive [AI Model Community License](https://huggingface.co/Arm/neural-super-sampling/blob/main/Arm_AI_Model_Community_License_v1_0_PRE-1154.pdf), designed to foster innovation in the graphics industry and provide differentiation to content creators. It allows NSS to be retrained on datasets captured from your own content. A [plugin](https://github.com/arm/neural-graphics-data-capture-for-unreal) for capturing data in Unreal Engine is available to capture your own content for use in (re)training or evaluation in the [Neural Graphics Model Gym](https://github.com/arm/neural-graphics-model-gym). ------------------------------------------------------------------------ ## Integration NSS can be directly integrated into graphics pipelines using ML Extensions for Vulkan. The recommended way of integrating the model into a graphics pipeline is by using the [VGF Library](https://github.com/arm/ai-ml-sdk-vgf-library/tree/main) from the ML SDK for Vulkan. ### Integration in Game Engines NSS has been integrated into Unreal Engine via the [NSS Plugin for Unreal Engine](https://github.com/arm/neural-graphics-for-unreal/). See the [Learning Path](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/nss-unreal/) for step-by-step instructions on how to use NSS in Unreal Engine. It is also available to use as part of the [Neural Graphics SDK for Game Engines](https://github.com/arm/neural-graphics-sdk-for-game-engines). This is designed to help integrate NSS across a wide variety of game engines. ------------------------------------------------------------------------ ## Getting Started This repository contains pre-trained weights and compiled NSS models in VGF format, ready for integration with Vulkan applications. These VGF models are unshaped, so will require developers to shape-specialize them before use. An example of how to do this can be seen in the [Neural Graphics SDK for Game Engines](https://github.com/arm/neural-graphics-sdk-for-game-engines/blob/f8df12935ef5743d7c7979751cef0c9a9c2f4e9d/sdk/src/backends/vk/ffx_vk.cpp#L137). See the included ML SDK for Vulkan [scenarios](https://huggingface.co/Arm/neural-super-sampling/tree/main/scenario) for the simplest way to evaluate the model. The scenarios include the necessary pre- and post-processing fragment shaders along with a single frame worth of input data. They demonstrate full execution of NSS on a Vulkan compute-capable system. The Emulation Layer can be used to implement ML Extensions for Vulkan where it is not supported by the native Vulkan driver. Three different scenarios for high, mid, and low quality modes of NSS are provided. For demonstration purposes these work on a fixed size input, performing upsampling from 540p to 1080p. ### 1. Clone the Repository These instructions will work on both Windows and Linux. On Windows, we strongly recommend the use of PowerShell over `cmd.exe` ("Command Prompt"). 1. Ensure that Git LFS is installed. See [the project's website](https://git-lfs.com/) for more details. 2. Clone the NSS model repository from Hugging Face: ```bash git clone https://huggingface.co/Arm/neural-super-sampling cd neural-super-sampling git lfs pull ``` ### 2. Install the Scenario Runner and Emulation Layers 1. Follow the steps in the Python documentation's ["Create a new virtual environment"](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#create-a-new-virtual-environment) guide. We will assume below that you installed this environment into a `.venv` subdirectory. 2. Follow the steps in the Python documentation's ["Activate a virtual environment"](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#activate-a-virtual-environment) guide. *Note for Windows users:* It is not necessary to determine the location of the Python interpreter as described in this guide. For reference, however, the correct PowerShell command to do this is `gcm python` rather than the given `where python`. 3. Install Arm's Scenario Runner and Emulation Layer into your new virtual environment: ```bash pip install ai-ml-sdk-scenario-runner ai-ml-emulation-layer-for-vulkan ``` 4. The Emulation Layer package provides a helper, that prints the commands to set environment variables for your current platform and virtual environment: * Enter the following command: ```bash emulation_layer --help ``` Copy and run the commands it prints in the same shell you will use for scenario-runner. On Linux this will set LD_LIBRARY_PATH, VK_LAYER_PATH, and VK_INSTANCE_LAYERS. On Windows it will set the equivalent PowerShell $env:VK_LAYER_PATH and $env:VK_INSTANCE_LAYERS variables. For more help, the Emulation Layer's documentation explains how its environment variables should be configured on [Linux](https://github.com/arm/ai-ml-emulation-layer-for-vulkan/blob/main/README.md#usage-on-linux) and on [Windows](https://github.com/arm/ai-ml-emulation-layer-for-vulkan/blob/main/README.md#usage-on-windows). ### 3. Run the Scenario 1. Change your working directory to the `scenario` folder: ```bash cd scenario ``` 2. Execute the Scenario Runner with your scenario of choice: ```bash scenario-runner --scenario configs/960x540_1920x1080_high_fragment.json --output out_high scenario-runner --scenario configs/960x540_1920x1080_mid_fragment.json --output out_mid scenario-runner --scenario configs/960x540_1920x1080_low_fragment.json --output out_low ``` 3. Use [RenderDoc](https://renderdoc.org/) to view these images. Output images are encoded as `B10G11R11_UFLOAT`. This format is common for framebuffers but not widely supported by image viewers. ------------------------------------------------------------------------ ## Training and Evaluation For background on NSS architecture and training read our blog: [How Neural Super Sampling Works](https://community.arm.com/arm-community-blogs/b/mobile-graphics-and-gaming-blog/posts/how-arm-neural-super-sampling-works) Training and evaluation details, including model architecture code, training pipeline, test configurations and dataset capture plugins, are available at: - Model training code: - Examples and tutorials: - Sample dataset: - Dataset capture plugin: ## 🔎 Model Explorer VGF Extension The [VGF extension to Model Explorer](https://github.com/arm/vgf-adapter-model-explorer) provides a simple interface to visualize and analyse the NSS model and any other VGF model. ![Model Explorer screenshot](resources/model-explorer-screenshot.png) ## Biases, Risks, Limitations and Out-of-Scope Use - Not suited for non-temporal tasks such as a standalone image upsampling - Requires accurate motion vectors and frame history for stable output - May underperform in extremely low framerate scenarios (<10 FPS) with fast camera movement - Padding of the input is needed if input dimensions are not divisible by 8 ### Recommendations For ultra-low-FPS use cases, reduce the camera speed, acceleration, or both so that the relative motion between frames mimics the application running at a higher frame rate. ## License - The license for the model source code can be found [here](https://github.com/arm/neural-graphics-model-gym/blob/main/LICENSES/Apache-2.0.txt). - The license for the content of this repository can be found [here](https://huggingface.co/Arm/neural-super-sampling/blob/main/Arm_AI_Model_Community_License_v1_0_PRE-1154.pdf) ## More Information 🧑‍🔬 More technical details about the model can be found in the [NSS Guide](https://developer.arm.com/documentation/111009/latest/). 👩🏽‍💻 Our [Neural Graphics Development Kit](https://developer.arm.com/mobile-graphics-and-gaming/neural-technology) contains engine plugins, model training tools, code examples and extensive developer documentation. 🙋🏻‍♀️ For questions or feedback please [start a discussion](https://huggingface.co/Arm/neural-super-sampling/discussions) ------------------------------------------------------------------------ ## Trademark Notice * Arm® is a registered trademark of Arm Limited (or its subsidiaries) in the US and/or elsewhere. * Python® is a registered trademark of the Python Software Foundation. * Unreal® is a trademark or registered trademark of Epic Games, Inc. in the United States of America and elsewhere. * Vulkan and the Vulkan logo are registered trademarks of the [Khronos Group Inc.](https://www.khronos.org/legal/trademarks). * Windows® is a trademark of the Microsoft group of companies.