Directory structure: └── performance/ ├── device-tensor.md ├── index.md ├── mobile-performance-tuning.md ├── olive.md ├── transformers-optimization.md ├── model-optimizations/ │ ├── float16.md │ ├── graph-optimizations.md │ ├── index.md │ ├── ort-format-model-runtime-optimization.md │ ├── ort-format-models.md │ └── quantization.md └── tune-performance/ ├── index.md ├── iobinding.md ├── logging_tracing.md ├── memory.md ├── profiling-tools.md ├── threading.md └── troubleshooting.md ================================================ FILE: docs/performance/device-tensor.md ================================================ --- title: Device tensors description: Custom device memory usage to reduce copies parent: Performance nav_order: 6 --- # Using device tensors in ONNX Runtime Using device tensors can be a crucial part in building efficient AI pipelines, especially on heterogenous memory systems. A typical example of such systems is any PC with a dedicated GPU. While a [recent GPU](https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889) itself has a memory bandwidth of about 1TB/s, the interconnect [PCI 4.0 x16](https://en.wikipedia.org/wiki/PCI_Express) to the CPU can often be the limiting factor with only ~32GB/s. Therefore it is often best to keep data local to the GPU as much as possible or hide slow memory traffic behind computation as the GPU is able to execute compute and PCI memory traffic simultaneously. A typical use case for these scenarios where memory is already local to the inference device would be a GPU accelerated video processing of an encoded video stream which can be decoded with GPU decoders. Another common case are iterative networks like diffusion networks or large language models for which intermediate tensors do not have to be copied back to CPU. Tile based inference for high resolution images is another use-case where custom memory management is important to reduce GPU idle times during PCI copies. Rather than doing sequential processing of each tile it is possible to overlap PCI copies and processing on the GPU and pipeline work in that matter. Image of sequential PCI->Processing->PCI and another image of it being interleaved. ## CUDA CUDA in ONNX Runtime has two custom memory types. `"CudaPinned"` and `"Cuda"` memory where [CUDA pinned](https://developer.nvidia.com/blog/how-optimize-data-transfers-cuda-cc/) is actually CPU memory which is directly accessible by the GPU allowing for fully asynchronous up and download of memory using [`cudaMemcpyAsync`](https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__MEMORY.html#group__CUDART__MEMORY_1g85073372f776b4c4d5f89f7124b7bf79). Normal CPU tensors only allow for a synchronous downloads from GPU to CPU while CPU to GPU copies can always be executed asynchronous. Allocating a tensor using the `Ort::Sessions`'s allocator is very straight forward using the [C++ API](https://onnxruntime.ai/docs/api/c/struct_ort_1_1_value.html#a5d35080239ae47cdbc9e505666dc32ec) which directly maps to the C API. ```c++ Ort::Session session(ort_env, model_path_cstr, session_options); Ort::MemoryInfo memory_info_cuda("Cuda", OrtArenaAllocator, /*device_id*/0, OrtMemTypeDefault); Ort::Allocator gpu_allocator(session, memory_info_cuda); auto ort_value = Ort::Value::CreateTensor( gpu_allocator, shape.data(), shape.size(), ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT16); ``` External allocated data can also be wrapped to an `Ort::Value` without copying it: ```c++ Ort::MemoryInfo memory_info_cuda("Cuda", OrtArenaAllocator, device_id, OrtMemTypeDefault); std::array shape{1, 4, 64, 64}; size_t cuda_buffer_size = 4 * 64 * 64 * sizeof(float); void *cuda_resource; CUDA_CHECK(cudaMalloc(&cuda_resource, cuda_buffer_size)); auto ort_value = Ort::Value::CreateTensor( memory_info_cuda, cuda_resource, cuda_buffer_size, shape.data(), shape.size(), ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT); ``` These allocated tensors can then be used as [I/O Binding](../performance/tune-performance/iobinding.md) to eliminate copy ops on the network and move the responsibility to the user. With such IO bindings more performance tunings are possible: - due to the fixed tensor address, a CUDA graph can be captured to reduce CUDA launch latency on CPU - due to either having fully asynchronous downloads to pinned memory or eliminating memory copies by using device local tensor, CUDA can run [fully asynchronous via a run option](../execution-providers/CUDA-ExecutionProvider.md#performance-tuning) on its given stream To set the custom compute stream for CUDA, refer to the V2 option API exposing the `Ort[CUDA|TensorRT]ProviderOptionsV2*`opaque struct pointer and the function `Update[CUDA|TensorRT]ProviderOptionsWithValue(options, "user_compute_stream", cuda_stream);` to set it's stream member. More details can be found in each execution provider doc. If you want to verify your optimizations Nsight System helps to correlate CPU API and GPU execution of CUDA operations. This will allow to verify as well if the desired synchronizations were made and no async operation falls back to synchronous execution. It is also used in [this](https://www.nvidia.com/gtc/session-catalog/?search=S62336#/session/1695978753458001R4wk) GTC talk explaining optimal usage of device tensors. ### Python API The Python API supports the same performance opportunities as the above mentioned C++ API. [Device tensors](https://onnxruntime.ai/docs/api/python/api_summary.html#data-on-device) can be allocated as shown here. Besides this the `user_compute_stream` can be set through this [API](https://onnxruntime.ai/docs/api/python/api_summary.html#onnxruntime.InferenceSession.set_providers) ```python sess = onnxruntime.InferenceSession("model.onnx", providers=["TensorrtExecutionProvider"]) option = {} s = torch.cuda.Stream() option["user_compute_stream"] = str(s.cuda_stream) sess.set_providers(["TensorrtExecutionProvider"], [option]) ``` Enabling asynchronous execution in python is possible through the same [run option](https://onnxruntime.ai/docs/api/python/api_summary.html#runoptions) as on the C++ API. ## DirectML Achieving the same behavior is possible through DirectX resources. To run asynchronous processing, it is crucial to do the same management of execution streams as needed with CUDA. For DirectX, this means managing the device and its command queue, which is possible through the C API. Details of how to set the compute command queue are documented with the usage of [`SessionOptionsAppendExecutionProvider_DML1`](../execution-providers/DirectML-ExecutionProvider.md#usage). If separate command queues are used for copy and compute, it is possible to overlap PCI copies and execution as well as make execution asynchronous. ```c++ #include Ort::MemoryInfo memory_info_dml("DML", OrtDeviceAllocator, device_id, OrtMemTypeDefault); std::array shape{1, 4, 64, 64}; void *dml_resource; size_t d3d_buffer_size = 4 * 64 * 64 * sizeof(float); const OrtDmlApi *ort_dml_api; Ort::ThrowOnError(Ort::GetApi().GetExecutionProviderApi( "DML", ORT_API_VERSION, reinterpret_cast(&ort_dml_api))); // Create d3d_buffer using D3D12 APIs Microsoft::WRL::ComPtr d3d_buffer = ...; // Create the dml resource from the D3D resource. ort_dml_api->CreateGPUAllocationFromD3DResource(d3d_buffer.Get(), &dml_resource); Ort::Value ort_value(Ort::Value::CreateTensor(memory_info_dml, dml_resource, d3d_buffer_size, shape.data(), shape.size(), ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT)); ``` A [single file sample](https://github.com/ankan-ban/HelloOrtDml/blob/main/Main.cpp) can be found on GitHub which shows how to manage and create copy and execution command queues. ### Python API Although allocating DirectX inputs from Python might not be a major use case, the API is available. This can prove to be very beneficial, especially for intermediate network caches, such as key-value caching in large language models (LLMs). ```python import onnxruntime as ort import numpy as np session = ort.InferenceSession("model.onnx", providers=["DmlExecutionProvider"]) cpu_array = np.zeros((1, 4, 512, 512), dtype=np.float32) dml_array = ort.OrtValue.ortvalue_from_numpy(cpu_array, "dml") binding = session.io_binding() binding.bind_ortvalue_input("data", dml_array) binding.bind_output("out", "dml") # if the output dims are known we can also bind a preallocated value # binding.bind_ortvalue_output("out", dml_array_out) session.run_with_iobinding(binding) ``` ================================================ FILE: docs/performance/index.md ================================================ --- title: Performance has_children: true nav_order: 8 --- # ONNX Runtime Performance ================================================ FILE: docs/performance/mobile-performance-tuning.md ================================================ --- title: Tune Mobile Performance (ORT <1.10 only) parent: Performance nav_exclude: true --- {::options toc_levels="2" /} # **IMPORTANT: THIS INFORMATION ONLY APPLIES TO ONNX RUNTIME VERSION 1.10 AND EARLIER. PLEASE USE A NEWER VERSION.** # ONNX Runtime Mobile Performance Tuning Learn how different optimizations affect performance, and get suggestions for performance testing with ORT format models. ONNX Runtime Mobile can be used to execute ORT format models using NNAPI (via the NNAPI Execution Provider (EP)) on Android platforms, and CoreML (via the CoreML EP) on iOS platforms. First, please review the introductory details in [using NNAPI with ONNX Runtime Mobile](../execution-providers/NNAPI-ExecutionProvider.md) and [using CoreML with ONNX Runtime](../execution-providers/CoreML-ExecutionProvider.md). **IMPORTANT NOTE:** The examples on this page refer to the NNAPI EP for brevity. The information equally applies to the CoreML EP, so any reference to 'NNAPI' below can be substituted with 'CoreML'.
Support for creating a CoreML-aware ORT format model, similar to [creating an NNAPI-aware ORT format model](#3-creating-an-nnapi-aware-ort-format-model), was added in ONNX Runtime version 1.9. ## Contents {: .no_toc} * TOC {:toc} ## 1. ONNX Model Optimization Example ONNX Runtime applies optimizations to the ONNX model to improve inferencing performance. These optimizations occur prior to exporting an ORT format model. See the [graph optimization](./model-optimizations/graph-optimizations.md) documentation for further details of the available optimizations. It is important to understand how the different optimization levels affect the nodes in the model, as this will determine how much of the model can be executed using NNAPI or CoreML. *Basic* The _basic_ optimizations remove redundant nodes and perform constant folding. Only ONNX operators are used by these optimizations when modifying the model. *Extended* The _extended_ optimizations replace one or more standard ONNX operators with custom internal ONNX Runtime operators to boost performance. Each optimization has a list of EPs that it is valid for. It will only replace nodes that are assigned to that EP, and the replacement node will be executed using the same EP. *Layout* _Layout_ optimizations may be hardware specific and involve internal conversions between the NCHW image layout used by ONNX and NHWC or NCHWc formats. They are enabled with an optimization level of 'all'. - For ONNX Runtime versions prior to 1.8 layout optimizations should not be used when creating ORT format models. - For ONNX Runtime version 1.8 or later layout optimizations may be enabled, as the hardware specific optimizations are automatically disabled. ### Outcome of optimizations when creating an optimized ORT format model Below is an example of the changes that occur in _basic_ and _extended_ optimizations when applied to the MNIST model with only the CPU EP enabled. The optimization level is specified when [creating the ORT format model](./model-optimizations/ort-format-models.md#optimization-level). - At the _basic_ level we combine the Conv and Add nodes (the addition is done via the 'B' input to Conv), we combine the MatMul and Add into a single Gemm node (the addition is done via the 'C' input to Gemm), and constant fold to remove one of the Reshape nodes. - `python /tools/python/convert_onnx_models_to_ort.py --optimization_level basic /dir_with_mnist_onnx_model` - At the _extended_ level we additionally fuse the Conv and Relu nodes using the internal ONNX Runtime FusedConv operator. - `python /tools/python/convert_onnx_models_to_ort.py --optimization_level extended /dir_with_mnist_onnx_model` ![Changes to nodes from basic and extended optimizations](../../images/mnist_optimization.png) ### Outcome of executing an optimized ORT format model with the NNAPI EP If the NNAPI EP is registered at runtime, it is given an opportunity to select the nodes in the loaded model that it can execute. When doing so it will group as many nodes together as possible to minimize the overhead of copying data between the CPU and NNAPI to execute the nodes. Each group of nodes can be considered as a sub-graph. The more nodes in each sub-graph, and the fewer sub-graphs, the better the performance will be. For each sub-graph, the NNAPI EP will create an [NNAPI model](https://developer.android.com/ndk/guides/neuralnetworks#model) that replicates the processing of the original nodes. It will create a function that executes this NNAPI model and performs any required data copies between CPU and NNAPI. ONNX Runtime will replace the original nodes in the loaded model with a single node that calls this function. If the NNAPI EP is not registered, or can not process a node, the node will be executed using the CPU EP. Below is an example for the MNIST model comparing what happens to the ORT format models at runtime if the NNAPI EP is registered. As the _basic_ level optimizations result in a model that only uses ONNX operators, the NNAPI EP is able to handle the majority of the model as NNAPI can execute the Conv, Relu and MaxPool nodes. This is done with a single NNAPI model as all the nodes NNAPI can handle are connected. We would expect performance gains from using NNAPI with this model, as the overhead of the device copies between CPU and NNAPI for a single NNAPI node is likely to be exceeded by the time saved executing multiple operations at once using NNAPI. The _extended_ level optimizations introduce the custom FusedConv nodes, which the NNAPI EP ignores as it will only take nodes that are using ONNX operators that NNAPI can handle. This results in two nodes using NNAPI, each handling a single MaxPool operation. The performance of this model is likely to be adversely affected, as the overhead of the device copies between CPU and NNAPI (which are required before and after each of the two NNAPI nodes) is unlikely to be exceeded by the time saved executing a single MaxPool operation each time using NNAPI. Better performance may be obtainable by not registering the NNAPI EP so that all nodes in the model are executed using the CPU EP. ![Changes to nodes by NNAPI EP depending on the optimization level the model was created with](../../images/mnist_optimization_with_nnapi.png) ## 2. Initial Performance Testing The best optimization settings will differ by model. Some models may perform better with NNAPI, some models may not. As the performance will be model specific you must run performance tests to determine the best combination for your model. It is suggested to run performance tests: - with NNAPI enabled and an ORT format model created with _basic_ level optimization - with NNAPI disabled and an ORT format model created with _extended_ or _all_ level optimization - use _all_ for ONNX Runtime version 1.8 or later, and _extended_ for previous versions For most scenarios it is expected that one of these two approaches will yield the best performance. If using an ORT format model with _basic_ level optimizations and NNAPI yields equivalent or better performance, it _may_ be possible to further improve performance by creating an NNAPI-aware ORT format model. The difference with this model is that the higher level optimizations are only applied to nodes that can not be executed using NNAPI. Whether any nodes fall into this category is model dependent. ## 3. Creating an NNAPI-aware ORT format model An NNAPI-aware ORT format model will keep all nodes from the ONNX model that can be executed using NNAPI, and allow _extended_ optimizations to be applied to any remaining nodes. For our MNIST model that would mean that after the _basic_ optimizations are applied, the nodes in the red shading are kept as-is, and nodes in the green shading could have _extended_ optimizations applied to them. ![Show nodes that are preserved as NNAPI can execute them, and nodes that are considered by extended optimizations](../../images/nnapi_aware_ort_format_model.png) To create an NNAPI-aware ORT format model please follow these steps. 1. Create a 'full' build of ONNX Runtime with the NNAPI EP by [building ONNX Runtime from source](../build/inferencing.md#cpu). This build can be done on any platform, as the NNAPI EP can be used to create the ORT format model without the Android NNAPI library as there is no model execution in this process. When building add `--use_nnapi --build_shared_lib --build_wheel` to the build flags if any of those are missing. Do NOT add the `--minimal_build` flag. - Windows : ``` \build.bat --config RelWithDebInfo --use_nnapi --build_shared_lib --build_wheel --parallel ``` - Linux: ``` /build.sh --config RelWithDebInfo --use_nnapi --build_shared_lib --build_wheel --parallel ``` **NOTE**: For **ONNX Runtime version 1.10 and earlier**, if you have previously done a minimal build with reduced operator kernels you will need to run `git reset --hard` to make sure any operator kernel exclusions are reversed prior to performing the 'full' build. If you do not, you may not be able to load the ONNX format model due to missing kernels. 2. Install the python wheel from the build output directory. - Windows : This is located in `build/Windows///dist/.whl`. - Linux : This is located in `build/Linux//dist/.whl`. The package name will differ based on your platform, python version, and build parameters. `` is the value from the `--config` parameter from the build command. ``` pip install -U build\Windows\RelWithDebIfo\RelWithDebIfo\dist\onnxruntime_noopenmp-1.7.0-cp37-cp37m-win_amd64.whl ``` 3. Create an NNAPI-aware ORT format model by running `convert_onnx_models_to_ort.py` as per the [standard instructions](./model-optimizations/ort-format-models.md), with NNAPI enabled (`--use_nnapi`), and the optimization level set to _extended_ or _all_ (e.g. `--optimization_level extended`). This will allow higher level optimizations to run on any nodes that NNAPI can not handle. ``` python /tools/python/convert_onnx_models_to_ort.py --use_nnapi --optimization_level extended /models ``` The python package from your 'full' build with NNAPI enabled must be installed for `--use_nnapi` to be a valid option This ORT model created can be used with a minimal build that includes the NNAPI EP. ================================================ FILE: docs/performance/olive.md ================================================ --- title: End to end optimization with Olive description: Hardware-aware model optimization tool parent: Performance nav_order: 5 --- # Olive - hardware-aware model optimization tool [Olive](https://github.com/microsoft/Olive) is an easy-to-use hardware-aware model optimization tool that composes industry-leading techniques across model compression, optimization, and compilation. It works with ONNX Runtime as an E2E inference optimization solution. Given a model and targeted hardware, Olive composes the best suitable optimization techniques to output the most efficient model(s) and runtime configurations for inferencing with ONNX Runtime, while taking a set of constraints such as accuracy and latency into consideration. Techniques Olive has integrated include ONNX Runtime Transformer optimizations, ONNX Runtime performance tuning, HW-dependent tunable post training quantization, quantize aware training, and more. Olive is the recommended tool for model optimization for ONNX Runtime. **Examples:** 1. [BERT optimization on CPU (with post training quantization)](https://github.com/microsoft/Olive/blob/main/examples/bert/bert_ptq_cpu.json) 2. [BERT optimization on CPU (with quantization aware training)](https://github.com/microsoft/Olive/blob/main/examples/bert/bert_qat_customized_train_loop_cpu.json) For more details, pls refer to [Olive repo](https://github.com/microsoft/Olive) and [Olive documentation](https://microsoft.github.io/Olive). ================================================ FILE: docs/performance/transformers-optimization.md ================================================ --- title: Transformers optimizer description: Transformer model optimization tool to use with ONNX Runtime parent: Performance nav_order: 4 --- # Transformer Model Optimization Tool Overview {: .no_toc } While ONNX Runtime automatically applies most optimizations while loading transformer models, some of the latest optimizations that have not yet been integrated into ONNX Runtime. These additional optimizations can be applied using the [transformer optimization tool](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/python/tools/transformers) to tune models for the best performance. This optimization tool provides an offline capability to optimize transformer models in scenarios where ONNX Runtime does not apply the optimization at load time. This tool can be helpful when: * ONNX Runtime does not yet have transformer-specific graph optimization enabled * The model can be converted to use float16 to boost performance using mixed precision on GPUs with Tensor Cores (like V100 or T4) * The model has inputs with dynamic axis, which blocks some optimizations from being applied by ONNX Runtime due to shape inference. * Experimenting with disabling or enabling some fusions to evaluate impact on performance or accuracy. **Usage:** 1. [Install ONNX Runtime](#1-install-onnx-runtime) 2. [Convert the transformer model to ONNX](#2-convert-a-transformer-model-to-onnx) 3. [Run the model optimizer tool](#3-run-the-model-optimizer-tool) 4. [Benchmark and profile the model](#4-benchmark-and-profile-the-model) ## Supported models For the list of models that have been tested with the optimizer, please refer to [this page](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/README.md#supported-models). Most optimizations require exact match of a subgraph. Any layout change in the subgraph might cause some optimization to not work. Note that different versions of training or export tool might lead to different graph layouts. It is recommended to use the latest released version of PyTorch and Transformers. ### Limitations * Due to the CUDA implementation of the Attention kernel in ONNX Runtime, the maximum number of attention heads is 1024. * Normally, due to GPU memory constraints, the maximum supported sequence length is 4096 for Longformer and 1024 for other types of models. --- ## 1. Install ONNX Runtime First you need install onnxruntime or onnxruntime-gpu package for CPU or GPU inference. To use onnxruntime-gpu, it is required to install CUDA and cuDNN and add their bin directories to PATH environment variable. See [Python installation instructions](./../install/index.md#python-installs). ## 2. Convert a transformer model to ONNX To convert the transformer model to ONNX, use [torch.onnx](https://pytorch.org/docs/stable/onnx.html) or [tensorflow-onnx](https://github.com/onnx/tensorflow-onnx). * Huggingface transformers has a [notebook](https://github.com/huggingface/notebooks/blob/master/examples/onnx-export.ipynb) shows an example of exporting a pretrained model to ONNX. * For tf2onnx, please refer to this [BERT tutorial](https://github.com/onnx/tensorflow-onnx/blob/master/tutorials/BertTutorial.ipynb). ### GPT-2 Model conversion Converting the GPT-2 model from PyTorch to ONNX is not straightforward when past state is used. The tool [convert_to_onnx](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/models/gpt2/convert_to_onnx.py) can help. You can use commands like the following to convert a pre-trained PyTorch GPT-2 model to ONNX for given precision (float32, float16): ```bash python -m onnxruntime.transformers.models.gpt2.convert_to_onnx -m gpt2 --model_class GPT2LMHeadModel --output gpt2.onnx -p fp32 python -m onnxruntime.transformers.models.gpt2.convert_to_onnx -m distilgpt2 --model_class GPT2LMHeadModel --output distilgpt2.onnx -p fp16 --use_gpu --optimize_onnx --auto_mixed_precision ``` The tool will also verify whether the ONNX model and corresponding PyTorch model generate the same outputs given the same random inputs. ### Longformer Model conversion Requirement: Linux OS (e.g. Ubuntu 18.04 or 20.04) and a Python environment with PyTorch 1.9.* like the following: ```bash conda create -n longformer python=3.8 conda activate longformer pip install torch==1.9.1+cpu torchvision==0.10.1+cpu torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html pip install onnx transformers==4.18.0 onnxruntime numpy ``` Next, build the source of torch extensions: ```bash cd onnxruntime/python/tools/transformers/models/longformer/torch_extensions python setup.py install ``` It will generate a PyTorch extension file like "build/lib.linux-x86_64-3.8/longformer_attention.cpython-38-x86_64-linux-gnu.so" under the directory. Finally, convert longformer model to ONNX model like the following: ```bash cd .. python convert_to_onnx.py -m longformer-base-4096 ``` The exported ONNX model can only run on GPU right now. ## 3. Run the model optimizer tool For all Optimizer options, please see [Github](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/README.md#optimizer-options). In your Python code, you can use the optimizer like the following: ```python from onnxruntime.transformers import optimizer optimized_model = optimizer.optimize_model("bert.onnx", model_type='bert', num_heads=12, hidden_size=768) optimized_model.convert_float_to_float16() optimized_model.save_model_to_file("bert_fp16.onnx") ``` You can also use command line. Example of optimizing a BERT-large model to use mixed precision (float16): ```console python -m onnxruntime.transformers.optimizer --input bert_large.onnx --output bert_large_fp16.onnx --num_heads 16 --hidden_size 1024 --float16 ``` You can also download the latest script files from [here](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/). Then run it like the following: ```console python optimizer.py --input bert.onnx --output bert_opt.onnx --model_type bert ``` ### BERT Model Verification If your BERT model has three inputs (like input_ids, token_type_ids and attention_mask), a script compare_bert_results.py can be used to do a quick verification. The tool will generate some fake input data, and compare results from both the original and optimized models. If outputs are all close, it is safe to use the optimized model. Example of verifying models optimized for CPU: ```console python -m onnxruntime.transformers.compare_bert_results --baseline_model original_model.onnx --optimized_model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 --samples 100 ``` For GPU, please append --use_gpu to the command. ## 4. Benchmark and profile the model ### Benchmarking The bash script [run_benchmark.sh](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/run_benchmark.sh) can be used for running benchmarks. You can modify the bash script to choose your options (models, batch sizes, sequence lengths, target device, etc) before running. The bash script will call benchmark.py script to measure inference performance of OnnxRuntime, PyTorch or PyTorch+TorchScript on pretrained models of Huggingface Transformers. #### Benchmark.py If you use run_benchmark.sh, you need not use benchmark.py directly. You can skip this section if you do not want to know the details. Below is example to run benchmark.py on pretrained model bert-base-cased on GPU. ```console python -m onnxruntime.transformers.benchmark -g -m bert-base-cased -o -v -b 0 python -m onnxruntime.transformers.benchmark -g -m bert-base-cased -o python -m onnxruntime.transformers.benchmark -g -m bert-base-cased -e torch python -m onnxruntime.transformers.benchmark -g -m bert-base-cased -e torchscript ``` The first command will generate ONNX models (both before and after optimizations), but not run performance tests since batch size is 0. The other three commands will run performance test on each of three engines: OnnxRuntime, PyTorch and PyTorch+TorchScript. If you remove -o parameter, optimizer script is not used in benchmark. If your GPU (like V100 or T4) has TensorCore, you can append `-p fp16` to the above commands to enable mixed precision. In some decoder-only(e.g GPT2) based generative models, you can enable [strict mode](../execution-providers/CUDA-ExecutionProvider.md#enable_skip_layer_norm_strict_mode) for SkipLayerNormalization Op on CUDA EP to achieve better accuracy. However, the performance will drop a bit. If you want to benchmark on CPU, you can remove -g option in the commands. Note that our current benchmark on GPT2 and DistilGPT2 models has disabled past state from inputs and outputs. By default, ONNX model has only one input (input_ids). You can use -i parameter to test models with multiple inputs. For example, we can add "-i 3" to command line to test a bert model with 3 inputs (input_ids, token_type_ids and attention_mask). This option only supports OnnxRuntime right now. ### Performance Test bert_perf_test.py can be used to check the BERT model inference performance. Below are examples: ```console python -m onnxruntime.transformers.bert_perf_test --model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 ``` For GPU, please append --use_gpu to the command. After test is finished, a file like perf_results_CPU_B1_S128_.txt or perf_results_GPU_B1_S128_.txt will be output to the model directory. ### Profiling profiler.py can be used to run profiling on a transformer model. It can help figure out the bottleneck of a model, and CPU time spent on a node or subgraph. Examples commands: ```console python -m onnxruntime.transformers.profiler --model bert.onnx --batch_size 8 --sequence_length 128 --samples 1000 --dummy_inputs bert --thread_num 8 --kernel_time_only python -m onnxruntime.transformers.profiler --model gpt2.onnx --batch_size 1 --sequence_length 1 --past_sequence_length 128 --samples 1000 --dummy_inputs gpt2 --use_gpu python -m onnxruntime.transformers.profiler --model longformer.onnx --batch_size 1 --sequence_length 4096 --global_length 8 --samples 1000 --dummy_inputs longformer --use_gpu ``` Result file like onnxruntime_profile__.json will be output to current directory. Summary of nodes, top expensive nodes and results grouped by operator type will be printed to console. Benchmark results can be found [here](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/README.md#benchmark-results). ================================================ FILE: docs/performance/model-optimizations/float16.md ================================================ --- title: Float16 and mixed precision models grand_parent: Performance parent: Model optimizations nav_order: 2 redirect_from: /docs/performance/float16 --- # Create Float16 and Mixed Precision Models {: .no_toc } Converting a model to use float16 instead of float32 can decrease the model size (up to half) and improve performance on some GPUs. There may be some accuracy loss, but in many models the new accuracy is acceptable. Tuning data is not needed for float16 conversion, which can make it preferable to quantization. ## Contents {: .no_toc } * TOC placeholder {:toc} ## Float16 Conversion Convert a model to float16 by following these steps: 1. Install onnx and [onnxconverter-common](https://github.com/microsoft/onnxconverter-common) `pip install onnx onnxconverter-common` 2. Use the `convert_float_to_float16` function in python. ```python import onnx from onnxconverter_common import float16 model = onnx.load("path/to/model.onnx") model_fp16 = float16.convert_float_to_float16(model) onnx.save(model_fp16, "path/to/model_fp16.onnx") ``` ### Float16 Tool Arguments If the converted model does not work or has poor accuracy, you may need to set additional arguments. ```python convert_float_to_float16(model, min_positive_val=1e-7, max_finite_val=1e4, keep_io_types=False, disable_shape_infer=False, op_block_list=None, node_block_list=None) ``` - `model`: The ONNX model to convert. - `min_positive_val`, `max_finite_val`: Constant values will be clipped to these bounds. `0.0`, `nan`, `inf`, and `-inf` will be unchanged. - `keep_io_types`: Whether model inputs/outputs should be left as float32. - `disable_shape_infer`: Skips running onnx shape/type inference. Useful if shape inference is crashing, shapes/types are already present in the model, or types are not needed (types are used to determine where cast ops are needed for unsupported/blocked ops). - `op_block_list`: List of op types to leave as float32. By default uses the list from `float16.DEFAULT_OP_BLOCK_LIST`. This list has ops that are not supported for float16 in ONNX Runtime. - `node_block_list`: List of node names to leave as float32. **NOTE**: Blocked ops will have have casts inserted around them to/from float16/float32. Currently, if two blocked ops are next to each other, the casts will still be inserted, creating a redundant pair. ORT will optimize this pair out at runtime, so the results will remain at full-precision. ## Mixed Precision If float16 conversion is giving poor results, you can convert most of the ops to float16 but leave some in float32. The `auto_mixed_precision.auto_convert_mixed_precision` tool finds a minimal set of ops to skip while retaining a certain level of accuracy. You will need to provide a sample input for the model. Since the CPU version of ONNX Runtime doesn't support float16 ops and the tool needs to measure the accuracy loss, **the mixed precision tool must be run on a device with a GPU**. ```python from onnxconverter_common import auto_mixed_precision import onnx model = onnx.load("path/to/model.onnx") # Assuming x is the input to the model feed_dict = {'input': x.numpy()} model_fp16 = auto_convert_mixed_precision(model, feed_dict, rtol=0.01, atol=0.001, keep_io_types=True) onnx.save(model_fp16, "path/to/model_fp16.onnx") ``` ### Mixed Precision Tool Arguments ```python auto_convert_mixed_precision(model, feed_dict, validate_fn=None, rtol=None, atol=None, keep_io_types=False) ``` - `model`: The ONNX model to convert. - `feed_dict`: Test data used to measure the accuracy of the model during conversion. Format is similar to InferenceSession.run (map of input names to values) - `validate_fn`: A function accepting two lists of numpy arrays (the outputs of the float32 model and the mixed-precision model, respectively) that returns `True` if the results are sufficiently close and `False` otherwise. Can be used instead of or in addition to `rtol` and `atol`. - `rtol`, `atol`: Absolute and relative tolerances used for validation. See [numpy.allclose](https://numpy.org/doc/stable/reference/generated/numpy.allclose.html) for more information. - `keep_io_types`: Whether model inputs/outputs should be left as float32. The mixed precision tool works by converting clusters of ops to float16. If a cluster fails, it is split in half and both clusters are tried independently. A visualization of the cluster sizes is printed as the tool works. ================================================ FILE: docs/performance/model-optimizations/graph-optimizations.md ================================================ --- title: Graph optimizations parent: Model optimizations grand_parent: Performance nav_order: 3 redirect_from: - /docs/how-to/graph-optimizations - /docs/performance/graph-optimizations --- # Graph Optimizations in ONNX Runtime {: .no_toc } ONNX Runtime provides various graph optimizations to improve performance. Graph optimizations are essentially graph-level transformations, ranging from small graph simplifications and node eliminations to more complex node fusions and layout optimizations. Graph optimizations are divided in several categories (or *levels*) based on their complexity and functionality. They can be performed either *online* or *offline*. In online mode, the optimizations are done before performing the inference, while in offline mode, the runtime saves the optimized graph to disk. ONNX Runtime provides Python, C#, C++, and C APIs to enable different optimization levels and to choose between offline vs. online mode. Below we provide details on the optimization levels, the online/offline mode, and the various APIs to control them. ## Contents {: .no_toc } * TOC placeholder {:toc} ## Graph Optimization Levels Graph optimizations are divided into three levels: 1. Basic 2. Extended 3. Layout Optimizations The optimizations belonging to one level are performed after the optimizations of the previous level have been applied (e.g., extended optimizations are applied after basic optimizations have been applied). **All optimizations are enabled by default.** ### Basic Graph Optimizations These are semantics-preserving graph rewrites which remove redundant nodes and redundant computation. They run before graph partitioning and thus apply to all the execution providers. Available basic graph optimizations are as follows: * Constant Folding: Statically computes parts of the graph that rely only on constant initializers. This eliminates the need to compute them during runtime. * Redundant node eliminations: Remove all redundant nodes without changing the graph structure. The following such optimizations are currently supported: * Identity Elimination * Slice Elimination * Unsqueeze Elimination * Dropout Elimination * Semantics-preserving node fusions : Fuse/fold multiple nodes into a single node. For example, Conv Add fusion folds the Add operator as the bias of the Conv operator. The following such optimizations are currently supported: * Conv Add Fusion * Conv Mul Fusion * Conv BatchNorm Fusion * Relu Clip Fusion * Reshape Fusion ### Extended Graph Optimizations These optimizations include complex node fusions. They are run after graph partitioning and are only applied to the nodes assigned to the CPU or CUDA or ROCm execution provider. Available extended graph optimizations are as follows: | Optimization | Execution Provider | Comment | |---------------------------------|--------------------|-----------------------------------------------------------------------------| | GEMM Activation Fusion | CPU | | | Matmul Add Fusion | CPU | | | Conv Activation Fusion | CPU | | | GELU Fusion | CPU, CUDA, ROCm | | | Layer Normalization Fusion | CPU, CUDA, ROCm | | | BERT Embedding Layer Fusion | CPU, CUDA, ROCm | Fuse BERT embedding layer, layer normalization and attention mask length | | Attention Fusion* | CPU, CUDA, ROCm | | | Skip Layer Normalization Fusion | CPU, CUDA, ROCm | Fuse bias of fully connected layer, skip connection and layer normalization | | Bias GELU Fusion | CPU, CUDA, ROCm | Fuse bias of fully connected layer and GELU activation | | GELU Approximation* | CUDA, ROCm | Disabled by default. Enable with [kOrtSessionOptionsEnableGeluApproximation](https://cs.github.com/microsoft/onnxruntime/blob/175acf08f470db0bb2e4b8eefe55cdeb87c8b132/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h?q=kOrtSessionOptionsEnableGeluApproximation#L52) |
Approximations (click to expand) To optimize performance of [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)), approximation is used in GELU Approximation and Attention Fusion for CUDA and ROCm execution provider. The impact on accuracy is negligible based on our evaluation: F1 score for a BERT model on SQuAD v1.1 is almost same (87.05 vs 87.03).
### Layout Optimizations These optimizations change the data layout for applicable nodes to achieve higher performance improvements. They are run after graph partitioning and are only applied to nodes assigned to CPU execution provider. Available layout optimizations are as follows: * NCHWc Optimizer: Optimizes the graph by using NCHWc layout instead of NCHW layout. ## Online/Offline Mode All optimizations can be performed either online or offline. In online mode, when initializing an inference session, we also apply all enabled graph optimizations before performing model inference. Applying all optimizations each time we initiate a session can add overhead to the model startup time (especially for complex models), which can be critical in production scenarios. This is where the offline mode can bring a lot of benefit. In offline mode, after performing graph optimizations, ONNX Runtime serializes the resulting model to disk. Subsequently, we can reduce startup time by using the already optimized model and disabling all optimizations. **Notes**: * When running in offline mode, make sure to use the exact same options (e.g., execution providers, optimization level) and hardware as the target machine that the model inference will run on (e.g., you cannot run a model pre-optimized for a GPU execution provider on a machine that is equipped only with CPU). * When layout optimizations are enabled, the offline mode can only be used on compatible hardware to the environment when the offline model is saved. For example, if model has layout optimized for AVX2, the offline model would require CPUs that support AVX2. ## Usage ### Levels ONNX Runtime defines the `GraphOptimizationLevel` enum to determine which of the aforementioned optimization levels will be enabled. Choosing a level enables the optimizations of that level, as well as the optimizations of all preceding levels. For example, enabling Extended optimizations, also enables Basic optimizations. The mapping of these levels to the enum is as follows: * GraphOptimizationLevel::ORT_DISABLE_ALL -> Disables all optimizations * GraphOptimizationLevel::ORT_ENABLE_BASIC -> Enables basic optimizations * GraphOptimizationLevel::ORT_ENABLE_EXTENDED -> Enables basic and extended optimizations * GraphOptimizationLevel::ORT_ENABLE_ALL -> Enables all available optimizations including layout optimizations ### Offline mode To enable serialization of the optimized model to disk, set the SessionOptions option `optimized_model_filepath`. #### Python API Example ```python import onnxruntime as rt sess_options = rt.SessionOptions() # Set graph optimization level sess_options.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_EXTENDED # To enable model serialization after graph optimization set this sess_options.optimized_model_filepath = "" session = rt.InferenceSession("", sess_options) ``` #### C API Example ```c const OrtApi* Ort::g_api = OrtGetApi(ORT_API_VERSION); OrtEnv* env; g_ort->CreateEnv(ORT_LOGGING_LEVEL_WARNING, "test", &env); OrtSessionOptions* session_options; g_ort->CreateSessionOptions(&session_options) // Set graph optimization level g_ort->SetSessionGraphOptimizationLevel(session_options, ORT_ENABLE_EXTENDED); // To enable model serialization after graph optimization set this const ORTCHAR_T* optimized_model_path = ORT_TSTR("optimized_model_path"); g_ort->SetOptimizedModelFilePath(session_options, optimized_model_path); OrtSession* session; const ORTCHAR_T* model_path = ORT_TSTR("model_path"); g_ort->CreateSession(env, model_path, session_options, &session); ``` #### C# API Example ```c# SessionOptions so = new SessionOptions(); // Set graph optimization level so.GraphOptimizationLevel = GraphOptimizationLevel.ORT_ENABLE_EXTENDED; // To enable model serialization after graph optimization set this so.OptimizedModelFilePath = "model_output_path\optimized_model.onnx" var session = new InferenceSession(modelPath, so); ``` #### C++ API Example ```c++ Ort::SessionOptions session_options; // Set graph optimization level session_options.SetGraphOptimizationLevel(GraphOptimizationLevel::ORT_ENABLE_EXTENDED); // To enable model serialization after graph optimization set this session_options.SetOptimizedModelFilePath("optimized_file_path"); auto session_ = Ort::Session(env, "model_file_path", session_options); ``` ================================================ FILE: docs/performance/model-optimizations/index.md ================================================ --- title: Model optimizations parent: Performance nav_order: 2 has_children: true --- # Model Optimizations In addition to [tuning performance](./../tune-performance/index.md) using ONNX Runtime configurations, there are techniques that can be applied to reduce model size and/or complexity to improve performance. ================================================ FILE: docs/performance/model-optimizations/ort-format-model-runtime-optimization.md ================================================ --- title: ORT model format runtime optimization grand_parent: Performance parent: Model optimizations nav_order: 5 redirect_from: - /docs/performance/ort-format-model-runtime-optimization - /docs/reference/mobile/ort-format-model-runtime-optimization --- {::options toc_levels="2" /} # ORT Format Model Runtime Optimization ## Contents {: .no_toc } * TOC placeholder {:toc} ## Background The full ONNX Runtime build supports [graph optimizations](./graph-optimizations.md) at runtime for ONNX models. The ORT format model was designed to be used with ONNX Runtime [minimal builds](../../build/custom.md#minimal-build) for environments where smaller binary size is important. To reduce the binary size, some or all of the graph optimizer code is excluded from a minimal build. As such, ONNX models and ORT format models do not share the same graph optimization process. In ONNX Runtime **1.11 and later**, there is limited support for graph optimizations at runtime for ORT format models. This only applies to extended minimal builds or full builds. In ONNX Runtime **1.10 and earlier**, there is **no support** for graph optimizations at runtime for ORT format models. Any graph optimizations must be done at model conversion time. See [this page](./../mobile-performance-tuning.md) for guidance using older ORT versions. As a rule, [basic graph optimizations](./graph-optimizations.md#basic-graph-optimizations) are semantics-preserving and result in a valid ONNX graph. The basic optimizations can and generally should be baked in to the converted ORT format model at conversion time - this is the default behavior of the conversion script. In fact, any runtime optimization support for ORT format models will not include basic optimizations at all. ## Types of runtime optimization These only apply to extended minimal builds or full builds. ### Saved runtime optimizations Some graph optimizers support additional modes to save and load information about potential graph optimizations to and from the ORT format model. These potential optimizations are known as saved runtime optimizations. Saved runtime optimizations are only applied at runtime if they are still applicable. For example, a CPU Execution Provider (EP)-specific optimization for some nodes is only applicable if those nodes are assigned to the CPU EP at runtime. When converting from ONNX to ORT format, the potential optimizations are identified (1) and their effects are saved alongside the graph (without those optimizations applied) in the ORT format model. Later, when loading the ORT format model with saved runtime optimizations, the effects of potential optimizations are applied (2) if the potential optimizations are still applicable. In an extended minimal build, only enough implementation to support (2) is included, reducing the binary size. ### Graph optimizers Some graph optimizers are also fully enabled in an extended minimal build and can be directly applied to an ORT format model. One example is the NHWC transformer. ## Choosing whether to use runtime optimizations The use of runtime optimizations is optional. Generally, the choice is between using an ORT format model with saved runtime optimizations and using a fully optimized ORT format model. A fully optimized model will have the full set of ONNX Runtime optimizations (at the extended level or higher) available but will be fully optimized for the configuration at model conversion time. A model with saved runtime optimizations has fewer optimizations available but has more flexibility at runtime. For example, at runtime, a compiling EP like the NNAPI EP can claim the set of nodes it can handle and the remaining nodes can be further optimized with saved runtime optimizations. You can compare the performance of: - A fully optimized model run with only the CPU EP enabled - A model with saved runtime optimizations run with additional EPs enabled The [model usability checker](../../tutorials/mobile/helpers/model-usability-checker.md) will provide guidance for a particular model. ================================================ FILE: docs/performance/model-optimizations/ort-format-models.md ================================================ --- title: ORT model format description: Define the ORT format and show how to convert an ONNX model to ORT format to run on mobile or web grand_parent: Performance parent: Model optimizations nav_order: 4 redirect_from: - /docs/tutorials/mobile/model-conversion - /docs/tutorials/mobile/model-execution - /docs/reference/ort-format-models --- # ORT model format {: .no_toc} ## Contents {: .no_toc} * TOC {:toc} ## What is the ORT model format? The ORT format is the format supported by reduced size ONNX Runtime builds. Reduced size builds may be more appropriate for use in size-constrained environments such as mobile and web applications. Both ORT format models and ONNX models are supported by a full ONNX Runtime build. ## Backwards Compatibility Generally, the goal is that a particular version of ONNX Runtime can run models at the current (at time of the ONNX Runtime release) or older versions of the ORT format. Though we try to maintain backwards compatibility, there have been some breaking changes. ONNX Runtime version|ORT format version support|Notes -|-|- 1.14+|v5, v4 (limited support)| See [here](https://github.com/microsoft/onnxruntime/blob/rel-1.14.0/docs/ORT_Format_Update_in_1.13.md#onnx-runtime-114) for details about limited v4 support. 1.13|v5|v5 breaking change: removed kernel def hashes. 1.12-1.8|v4|v4 breaking change: updated kernel def hash computation. 1.7|v3, v2, v1| 1.6|v2, v1| 1.5|v1|ORT format introduced ## Convert ONNX models to ORT format ONNX models are converted to ORT format using the `convert_onnx_models_to_ort` script. The conversion script performs two functions: 1. Loads and optimizes ONNX format models, and saves them in ORT format 2. Determines the operators, and optionally data types, required by the optimized models, and saves these in a configuration file for use in a reduced operator build, if required The conversion script can run on a single ONNX model, or a directory. If run against a directory, the directory will be recursively searched for '.onnx' files to convert. Each '.onnx' file is loaded, optimized, and saved in ORT format as a file with the '.ort' extension in the same location as the original '.onnx' file. ### Outputs of the script 1. One ORT format model for each ONNX model 2. A [build configuration file](../../reference/operators/reduced-operator-config-file.md) ('required_operators.config') with the operators required by the optimized ONNX models. If [type reduction](#enable-type-reduction) is enabled (ONNX Runtime version 1.7 or later) the configuration file will also include the required types for each operator, and is called 'required_operators_and_types.config'. If you are using a pre-built ONNX Runtime [iOS](../../install/index.md#install-on-ios), [Android](../../install/index.md#install-on-android) or [web](../../install/index.md#javascript-installs) package, the build configuration file is not used and can be ignored. ### Script location The ORT model format is supported by version 1.5.2 of ONNX Runtime or later. Conversion of ONNX format models to ORT format utilizes the ONNX Runtime python package, as the model is loaded into ONNX Runtime and optimized as part of the conversion process. For ONNX Runtime version 1.8 and later the conversion script is run directly from the ONNX Runtime python package. For earlier versions, the conversion script is run from the local ONNX Runtime repository. ### Install ONNX Runtime Install the onnxruntime python package from [https://pypi.org/project/onnxruntime/](https://pypi.org/project/onnxruntime/) in order to convert models from ONNX format to the internal ORT format. Version 1.5.3 or higher is required. #### Install the latest release ```bash pip install onnxruntime ``` #### Install a previous release If you are building ONNX Runtime from source (custom, reduced or minimal builds), you must match the python package version to the branch of the ONNX Runtime repository you checked out. For example, to use the 1.7 release: ```bash git checkout rel-1.7.2 pip install onnxruntime==1.7.2 ``` If you are using the `main` branch in the git repository you should use the nightly ONNX Runtime python package: ```bash pip install -U -i https://test.pypi.org/simple/ ort-nightly ``` ### Convert ONNX models to ORT format script usage ONNX Runtime version 1.8 or later: ```bash python -m onnxruntime.tools.convert_onnx_models_to_ort ``` where: * onnx model file or dir is a path to .onnx file or directory containing one or more .onnx models The current optional arguments are available by running the script with the `--help` argument. Supported arguments and defaults differ slightly across ONNX Runtime versions. Help text from ONNX Runtime 1.11: ```bash python -m onnxruntime.tools.convert_onnx_models_to_ort --help ``` ```output usage: convert_onnx_models_to_ort.py [-h] [--optimization_style {Fixed,Runtime} [{Fixed,Runtime} ...]] [--enable_type_reduction] [--custom_op_library CUSTOM_OP_LIBRARY] [--save_optimized_onnx_model] [--allow_conversion_failures] [--nnapi_partitioning_stop_ops NNAPI_PARTITIONING_STOP_OPS] [--target_platform {arm,amd64}] model_path_or_dir Convert the ONNX format model/s in the provided directory to ORT format models. All files with a `.onnx` extension will be processed. For each one, an ORT format model will be created in the same directory. A configuration file will also be created containing the list of required operators for all converted models. This configuration file should be used as input to the minimal build via the `--include_ops_by_config` parameter. positional arguments: model_path_or_dir Provide path to ONNX model or directory containing ONNX model/s to convert. All files with a .onnx extension, including those in subdirectories, will be processed. optional arguments: -h, --help show this help message and exit --optimization_style {Fixed,Runtime} [{Fixed,Runtime} ...] Style of optimization to perform on the ORT format model. Multiple values may be provided. The conversion will run once for each value. The general guidance is to use models optimized with 'Runtime' style when using NNAPI or CoreML and 'Fixed' style otherwise. 'Fixed': Run optimizations directly before saving the ORT format model. This bakes in any platform-specific optimizations. 'Runtime': Run basic optimizations directly and save certain other optimizations to be applied at runtime if possible. This is useful when using a compiling EP like NNAPI or CoreML that may run an unknown (at model conversion time) number of nodes. The saved optimizations can further optimize nodes not assigned to the compiling EP at runtime. --enable_type_reduction Add operator specific type information to the configuration file to potentially reduce the types supported by individual operator implementations. --custom_op_library CUSTOM_OP_LIBRARY Provide path to shared library containing custom operator kernels to register. --save_optimized_onnx_model Save the optimized version of each ONNX model. This will have the same level of optimizations applied as the ORT format model. --allow_conversion_failures Whether to proceed after encountering model conversion failures. --nnapi_partitioning_stop_ops NNAPI_PARTITIONING_STOP_OPS Specify the list of NNAPI EP partitioning stop ops. In particular, specify the value of the "ep.nnapi.partitioning_stop_ops" session options config entry. --target_platform {arm,amd64} Specify the target platform where the exported model will be used. This parameter can be used to choose between platform-specific options, such as QDQIsInt8Allowed(arm), NCHWc (amd64) and NHWC (arm/amd64) format, different optimizer level options, etc. ``` #### Optional script arguments ##### Optimization style **Since ONNX Runtime 1.11** Specify whether the converted model will be fully optimized ("Fixed") or have saved runtime optimizations ("Runtime"). Both types of models are produced by default. See [here](./ort-format-model-runtime-optimization.md) for more information. This replaces the [optimization level](#optimization-level) option from earlier ONNX Runtime versions. ##### Optimization level **ONNX Runtime version 1.10 and earlier** Set the optimization level that ONNX Runtime will use to optimize the model prior to saving in ORT format. For ONNX Runtime version 1.8 and later, *all* is recommended if the model will be run with the CPU EP. For earlier versions, *extended* is recommended, as the *all* level previously included device specific optimizations that would limit the portability of the model. If the model is to be run with the NNAPI EP or CoreML EP, it is recommended to create an ORT format model using the *basic* optimization level. Performance testing should be done to compare running this model with the NNAPI or CoreML EP enabled vs. running the model optimized to a higher level using the CPU EP to determine the optimal setup. See the documentation on [performance tuning mobile scenarios](../../performance/mobile-performance-tuning.md) for more information. ##### Enable type reduction With ONNX Runtime version 1.7 and later it is possible to limit the data types the required operators support to further reduce the build size. This pruning is referred to as "operator type reduction" in this documentation. As the ONNX model/s are converted, the input and output data types required by each operator are accumulated and included in the configuration file. If you wish to enable operator type reduction, the [Flatbuffers](https://google.github.io/flatbuffers/) python package must be installed. ```bash pip install flatbuffers ``` For example, the ONNX Runtime kernel for Softmax supports both float and double. If your model/s uses Softmax but only with float data, we can exclude the implementation that supports double to reduce the kernel's binary size. ##### Custom Operator support If your ONNX model uses [custom operators](../../reference/operators/add-custom-op.md), the path to the library containing the custom operator kernels must be provided so that the ONNX model can be successfully loaded. The custom operators will be preserved in the ORT format model. ##### Save optimized ONNX model Add this flag to save the optimized ONNX model. The optimized ONNX model contains the same nodes and initializers as the ORT format model, and can be viewed in [Netron](https://netron.app/) for debugging and performance tuning. ### Previous versions of ONNX Runtime Prior to ONNX Runtime version 1.7, the model conversion script must be run from a cloned source repository: ```bash python /tools/python/convert_onnx_models_to_ort.py ``` ## Load and execute a model in ORT format The API for executing ORT format models is the same as for ONNX models. See the [ONNX Runtime API documentation](../../api) for details on individual API usage. ### APIs by platform | Platform | Available APIs | |----------|----------------| | Android | C, C++, Java, Kotlin | | iOS | C, C++, Objective-C (Swift via bridge) | | Web | JavaScript | ### ORT format model loading If you provide a filename for the ORT format model, a file extension of '.ort' will be inferred to be an ORT format model. If you provide in-memory bytes for the ORT format model, a marker in those bytes will be checked to determine if it's an ORT format model. If you wish to explicitly say that the InferenceSession input is an ORT format model you can do so via SessionOptions, although this generally should not be necessary. #### Load ORT format model from a file path C++ API ```c++ Ort::SessionOptions session_options; session_options.AddConfigEntry("session.load_model_format", "ORT"); Ort::Env env; Ort::Session session(env, , session_options); ``` Java API ```java SessionOptions session_options = new SessionOptions(); session_options.addConfigEntry("session.load_model_format", "ORT"); OrtEnvironment env = OrtEnvironment.getEnvironment(); OrtSession session = env.createSession(, session_options); ``` JavaScript API ```js import * as ort from "onnxruntime-web"; const session = await ort.InferenceSession.create(""); ``` #### Load ORT format model from an in-memory byte array If a session is created using an input byte array containing the ORT format model data, by default we will copy the model bytes at the time of session creation to ensure the model bytes buffer is valid. You may also enable the option to use the model bytes directly by setting the SessionOptions config entry `session.use_ort_model_bytes_directly` to `1`. This may reduce the peak memory usage of ONNX Runtime Mobile, but you will need to guarantee that the model bytes are valid throughout the lifespan of the ORT session. For ONNX Runtime Web, this option is set by default. If `session.use_ort_model_bytes_directly` is enabled there is also an option to directly use the model bytes for initializers to further reduce peak memory usage. Set the Session Options config entry `session.use_ort_model_bytes_for_initializers` to `1` to enable this. Note that if an initializer gets pre-packed it will undo the peak memory usage saving from using the model bytes directly for that initializer, as a new buffer for the pre-packed data needs to be allocated. Pre-packing is an optional performance optimization that involves changing the initializer layout to the optimal ordering for the current platform if it differs. If reducing peak memory usage is more important than potential performance optimizations pre-packing can be disabled by setting `session.disable_prepacking` to `1`. C++ API ```c++ Ort::SessionOptions session_options; session_options.AddConfigEntry("session.load_model_format", "ORT"); session_options.AddConfigEntry("session.use_ort_model_bytes_directly", "1"); std::ifstream stream(, std::ios::in | std::ios::binary); std::vector model_bytes((std::istreambuf_iterator(stream)), std::istreambuf_iterator()); Ort::Env env; Ort::Session session(env, model_bytes.data(), model_bytes.size(), session_options); ``` Java API ```java SessionOptions session_options = new SessionOptions(); session_options.addConfigEntry("session.load_model_format", "ORT"); session_options.addConfigEntry("session.use_ort_model_bytes_directly", "1"); byte[] model_bytes = Files.readAllBytes(Paths.get()); OrtEnvironment env = OrtEnvironment.getEnvironment(); OrtSession session = env.createSession(model_bytes, session_options); ``` JavaScript API ```js import * as ort from "onnxruntime-web"; const response = await fetch(modelUrl); const arrayBuffer = await response.arrayBuffer(); model_bytes = new Uint8Array(arrayBuffer); const session = await ort.InferenceSession.create(model_bytes); ``` ================================================ FILE: docs/performance/model-optimizations/quantization.md ================================================ --- title: Quantize ONNX models grand_parent: Performance parent: Model optimizations nav_order: 1 redirect_from: - /docs/how-to/quantization - /docs/performance/quantization --- # Quantize ONNX Models {: .no_toc } ## Contents {: .no_toc } * TOC placeholder {:toc} ## Quantization Overview Quantization in ONNX Runtime refers to 8 bit linear quantization of an ONNX model. During quantization, the floating point values are mapped to an 8 bit quantization space of the form: `val_fp32 = scale * (val_quantized - zero_point)` `scale` is a positive real number used to map the floating point numbers to a quantization space. It is calculated as follows: For asymmetric quantization: ``` scale = (data_range_max - data_range_min) / (quantization_range_max - quantization_range_min) ``` For symmetric quantization: ``` scale = max(abs(data_range_max), abs(data_range_min)) * 2 / (quantization_range_max - quantization_range_min) ``` `zero_point` represents zero in the quantization space. It is important that the floating point zero value be exactly representable in quantization space. This is because zero padding is used in many CNNs. If it is not possible to represent 0 uniquely after quantization, it will result in accuracy errors. ## ONNX quantization representation format There are two ways to represent quantized ONNX models: - Operator-oriented (QOperator) : All the quantized operators have their own ONNX definitions, like QLinearConv, MatMulInteger and etc. - Tensor-oriented (QDQ; Quantize and DeQuantize) : This format inserts DeQuantizeLinear(QuantizeLinear(tensor)) between the original operators to simulate the quantization and dequantization process. In Static Quantization, the QuantizeLinear and DeQuantizeLinear operators also carry the quantization parameters. In Dynamic Quantization, a ComputeQuantizationParameters function proto is inserted to calculate quantization parameters on the fly. - Models generated in the following ways are in the QDQ format: 1. Models quantized by quantize_static, explained below, with `quant_format=QuantFormat.QDQ`. 2. Quantization-Aware training (QAT) models converted from Tensorflow or exported from PyTorch. 3. Quantized models converted from TFLite and other frameworks. For the latter two cases, you don't need to quantize the model with the quantization tool. ONNX Runtime can run them directly as a quantized model. The picture below shows the equivalent representation with the QOperator and QDQ formats for quantized Conv. [This end-to-end example](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/quantization/image_classification/cpu/run.py) demonstrates the two formats. ![Changes to nodes from basic and extended optimizations](../../../images/QDQ_Format.png) ## Quantizing an ONNX model ONNX Runtime provides python APIs for converting 32-bit floating point model to an 8-bit integer model, a.k.a. quantization. These APIs include pre-processing, dynamic/static quantization, and debugging. ### Pre-processing {: .no_toc} Pre-processing is to transform a float32 model to prepare it for quantization. It consists of the following three optional steps: 1. Symbolic shape inference. This is best suited for transformer models. 2. Model optimization: This step uses ONNX Runtime native library to rewrite the computation graph, including merging computation nodes, eliminating redundancies to improve runtime efficiency. 3. ONNX shape inference. The goal of these steps is to improve quantization quality. Our quantization tool works best when the tensor's shape is known. Both symbolic shape inference and ONNX shape inference help figure out tensor shapes. Symbolic shape inference works best with transformer based models, and ONNX shape inference works with other models. Model optimization performs certain operator fusion that makes quantization tool's job easier. For instance, a Convolution operator followed by BatchNormalization can be fused into one during the optimization, which can be quantized very efficiently. Unfortunately, a known issue in ONNX Runtime is that model optimization can not output a model size greater than 2GB. So for large models, optimization must be skipped. Pre-processing API is in Python module `onnxruntime.quantization.shape_inference`, function `quant_pre_process()`. See [`shape_inference.py`](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/quantization/shape_inference.py). To read about additional options and finer controls available to pre-processing, run the following command: ```console python -m onnxruntime.quantization.preprocess --help ``` Model optimization may also be performed during quantization. However, this is *NOT* recommended, even though it's the default behavior due to historical reasons. Model optimization during quantization creates difficulties for debugging quantization caused accuracy losses, which will be discussed in [later sections](#qdqdebug). So, it is best to perform model optimization during pre-processing instead of during quantization. ### Dynamic Quantization {: .no_toc} There are two ways of quantizing a model: dynamic and static. Dynamic quantization calculates the quantization parameters (scale and zero point) for activations dynamically. These calculations increase the cost of inference, while usually achieve higher accuracy comparing to static ones. Python API for dynamic quantization is in module `onnxruntime.quantization.quantize`, function `quantize_dynamic()` ### Static Quantization {: .no_toc} Static quantization method first runs the model using a set of inputs called calibration data. During these runs, we compute the quantization parameters for each activations. These quantization parameters are written as constants to the quantized model and used for all inputs. Our quantization tool supports three calibration methods: MinMax, Entropy and Percentile. Please refer to [`calibrate.py`](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/quantization/calibrate.py) for details. Python API for static quantization is in module `onnxruntime.quantization.quantize`, function `quantize_static()`. Please refer to [quantize.py](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/quantization/quantize.py) for details. ### Quantization Debugging Quantization is not a loss-less transformation. It may negatively affect a model's accuracy. A solution to this problem is to compare the weights and activations tensors of the original computation graph vs those of the quantized one, identify where they differ most, and avoid quantizing these tensors, or choose another quantization/calibration method. This is called quantization debugging. To facilitate this process, we provide Python APIs for matching weights and activation tensors between a float32 model and its quantized counterpart. API for debugging is in module `onnxruntime.quantization.qdq_loss_debug`, which has the following functions: - Function `create_weight_matching()`. It takes a float32 model and its quantized model, and output a dictionary that matches the corresponding weights between these two models. - Function `modify_model_output_intermediate_tensors()`. It takes a float32 or quantized model, and augment it to save all its activations. - Function `collect_activations()`. It takes a model augmented by `modify_model_output_intermediate_tensors()`, and an input data reader, runs the augmented model to collect all the activations. - Function `create_activation_matching()`. You can imagine that you run `collect_activations(modify_model_output_intermediate_tensors())` on both the float32 and its quantized model, to collect two sets of activations. This function takes these two set of activations, and matches up corresponding ones, so that they can be easily compared by the user. In summary, ONNX Runtimes provides Python APIs for matching up corresponding weights and activation tensors between a float32 model and its quantized counterpart. This allows the user to easily compare them to locate where are the biggest differences. Model optimization during quantization creates difficulties for this debugging process though, since it may changes the computation graph in a significant way, resulting in a quantized model that is drastically different from the original. This makes it hard to match up corresponding tensors from the two models. As a result, we recommend performing model optimization during pre-processing instead of the quantization process. #### Example {: .no_toc } - Dynamic quantization: ```python import onnx from onnxruntime.quantization import quantize_dynamic, QuantType model_fp32 = 'path/to/the/model.onnx' model_quant = 'path/to/the/model.quant.onnx' quantized_model = quantize_dynamic(model_fp32, model_quant) ``` - Static quantization: please refer to the [end-to-end examples](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/quantization/image_classification/cpu). ### Method selection {: .no_toc} The main difference between dynamic and static quantization is how the scale and zero point of activations are calculated. For static quantization, they are calculated in advance (offline) using a calibration data set. The activations thus have the same scale and zero point during each forward pass. For dynamic quantization, they are calculated on-the-fly (online) and are specific for each forward pass. They are thus more accurate but introduce an extra computational overhead. In general, it is recommended to use dynamic quantization for RNNs and transformer-based models, and static quantization for CNN models. If neither post-training quantization method can meet your accuracy goal, you can try using quantization-aware training (QAT) to retrain the model. ONNX Runtime does not provide retraining at this time, but you can retrain your models with the original framework and convert them back to ONNX. ### Data type selection {: .no_toc} The quantized values are 8 bits wide and can be either signed (int8) or unsigned (uint8). We can choose the signedness of the activations and the weights separately, so the data format can be (activations: uint8, weights: uint8), (activations: uint8, weights: int8), etc. Let's use U8U8 as a shorthand for (activations: uint8, weights: uint8), U8S8 for (activations: uint8, weights: int8), and similarly S8U8 and S8S8 for the remaining two formats. ONNX Runtime quantization on CPU can run U8U8, U8S8 and S8S8. S8S8 with QDQ is the default setting and balances performance and accuracy. It should be the first choice. Only in cases that the accuracy drops a lot, you can try U8U8. Note that S8S8 with QOperator will be slow on x86-64 CPUs and should be avoided in general. ONNX Runtime quantization on GPU only supports S8S8. #### When and why do I need to try U8U8? {: .no_toc } On x86-64 machines with AVX2 and AVX512 extensions, ONNX Runtime uses the VPMADDUBSW instruction for U8S8 for performance. This instruction might suffer from saturation issues: it can happen that the output does not fit into a 16-bit integer and has to be clamped (saturated) to fit. Generally, this is not a big issue for the final result. However, if you do encounter a large accuracy drop, it may be caused by saturation. In this case, you can either try [reduce_range](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/quantization/quantize.py) or the U8U8 format which doesn't have saturation issues. There is no such issue on other CPU architectures (x64 with VNNI and Arm®). ### List of Supported Quantized Ops {: .no_toc} Please refer to the [registry](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/quantization/registry.py) for the list of supported Ops. ### Quantization and model opset versions {: .no_toc} Models must be opset10 or higher to be quantized. Models with opset < 10 must be reconverted to ONNX from their original framework using a later opset. ## Transformer-based models There are specific optimizations for transformer-based models, such as QAttention for quantization of attention layers. In order to leverage these optimizations, you need to optimize your models using the [Transformer Model Optimization Tool](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/python/tools/transformers) before quantizing the model. This [notebook](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/quantization/notebooks/bert) demonstrates the process. ## Quantization on GPU Hardware support is required to achieve better performance with quantization on GPUs. You need a device that supports Tensor Core int8 computation, like T4 or A100. Older hardware will not benefit from quantization. ONNX Runtime leverages the TensorRT Execution Provider for quantization on GPU now. Unlike the CPU Execution Provider, TensorRT takes in a full precision model and a calibration result for inputs. It decides how to quantize with their own logic. The overall procedure to leverage TensorRT EP quantization is: - Implement a [CalibrationDataReader](https://github.com/microsoft/onnxruntime/blob/07788e082ef2c78c3f4e72f49e7e7c3db6f09cb0/onnxruntime/python/tools/quantization/calibrate.py). - Compute quantization parameters using a calibration data set. Note: In order to include all tensors from the model for better calibration, please run `symbolic_shape_infer.py` first. Please refer to [here](../../execution-providers/TensorRT-ExecutionProvider.md#samples) for details. - Save quantization parameters into a flatbuffer file - Load model and quantization parameter file and run with the TensorRT EP. We provide two end-to end examples: [Yolo V3](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/quantization/object_detection/trt/yolov3) and [resnet50](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/quantization/image_classification/trt/resnet50). ## Quantize to Int4/UInt4 ONNX Runtime can quantize certain operators in a model to 4 bit integer types. Block-wise weight-only quantization is applied to the operators. The supported op types are: - [MatMul](https://github.com/onnx/onnx/blob/main/docs/Operators.md#matmul): - The node is quantized only if the input `B` is constant - support QOperator or QDQ format. - If QOperator is selected, the node is converted to a [MatMulNBits](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftmatmulnbits) node. Weight `B` is blockwise quantized and saved in the new node. [HQQ](https://arxiv.org/pdf/2309.15531.pdf), [GPTQ](https://huggingface.co/docs/transformers/main/en/quantization/gptq) and RTN (default) algorithms are supported. - If QDQ is selected, the MatMul node is replaced by a DequantizeLinear -> MatMul pair. Weight `B` is blockwise quantized and saved in the DequantizeLinear node as an initializer. - [Gather](https://github.com/onnx/onnx/blob/main/docs/Operators.md#Gather): - The node is quantized only if the input `data` is constant. - support QOperator - Gather is quantized to a [GatherBlockQuantized](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftgatherblockquantized) node. Input `data` is blockwise quantized and saved in the new node. Only support RTN algorithm. Since Int4/UInt4 types are introduced in [onnx opset 21](https://github.com/onnx/onnx/releases/tag/v1.16.0), if the model's onnx domain version is < 21, it is force upgraded to opset 21. Please make sure the operators in the model are compatible with onnx opset 21. To run a model that has GatherBlockQuantized nodes, ONNX Runtime 1.20 is needed. Code Examples: ```python from onnxruntime.quantization import ( matmul_4bits_quantizer, quant_utils, quantize ) from pathlib import Path model_fp32_path="path/to/orignal/model.onnx" model_int4_path="path/to/save/quantized/model.onnx" quant_config = matmul_4bits_quantizer.DefaultWeightOnlyQuantConfig( block_size=128, # 2's exponential and >= 16 is_symmetric=True, # if true, quantize to Int4. otherwise, quantize to uint4. accuracy_level=4, # used by MatMulNbits, see https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#attributes-35 quant_format=quant_utils.QuantFormat.QOperator, op_types_to_quantize=("MatMul","Gather"), # specify which op types to quantize quant_axes=(("MatMul", 0), ("Gather", 1),) # specify which axis to quantize for an op type. model = quant_utils.load_model_with_shape_infer(Path(model_fp32_path)) quant = matmul_4bits_quantizer.MatMul4BitsQuantizer( model, nodes_to_exclude=None, # specify a list of nodes to exclude from quantization nodes_to_include=None, # specify a list of nodes to force include from quantization algo_config=quant_config,) quant.process() quant.model.save_model_to_file( model_int4_path, True) # save data to external file ``` For AWQ and GTPQ quantization usage, please refer to [Gen-AI model builder](https://github.com/microsoft/onnxruntime-genai/tree/main/src/python/py/models#quantized-pytorch-model). ## FAQ ### Why am I not seeing performance improvements? {: .no_toc } The performance improvement depends on your model and hardware. The performance gain from quantization has two aspects: compute and memory. Old hardware has none or few of the instructions needed to perform efficient inference in int8. And quantization has overhead (from quantizing and dequantizing), so it is not rare to get worse performance on old devices. x86-64 with VNNI, GPU with Tensor Core int8 support and Arm®-based processors with dot-product instructions can get better performance in general. ### Which quantization method should I choose, dynamic or static? {: .no_toc} Please refer to the [Method selection](#method-selection) section. ### When to use reduce-range and per-channel quantization? {: .no_toc} Reduce-range will quantize the weights to 7 bits. It is designed for the U8S8 format on AVX2 and AVX512 (non-VNNI) machines to mitigate [saturation issues](#data-type-selection). This is not needed on machines supporting VNNI. Per-channel quantization can improve the accuracy for models whose weight ranges are large. Try it if the accuracy loss is large. On AVX2 and AVX512 machines, you will generally need to enable reduce-range as well if per-channel is enabled. ### Why are operators like MaxPool not quantized? {: .no_toc} 8-bit type support for certain operators such as MaxPool was added in ONNX opset 12. Please check your model version and upgrade it to opset 12 and above. ================================================ FILE: docs/performance/tune-performance/index.md ================================================ --- title: Tune performance parent: Performance has_children: true nav_order: 1 redirect_from: - /docs/how-to/tune-performance - /docs/performance/tune-performance --- # ONNX Runtime Performance Tuning {: .no_toc } ONNX Runtime provides high performance for running deep learning models on a range of hardwares. Based on usage scenario requirements, latency, throughput, memory utilization, and model/application size are common dimensions for how performance is measured. While ORT out-of-box aims to provide good performance for the most common usage patterns, there are model optimization techniques and runtime configurations that can be utilized to improve performance for specific use cases and models. * TOC placeholder {:toc} ================================================ FILE: docs/performance/tune-performance/iobinding.md ================================================ --- title: I/O Binding grand_parent: Performance parent: Tune performance nav_order: 5 --- # I/O Binding When working with non-CPU execution providers, it's most efficient to have inputs (and/or outputs) arranged on the target device (abstracted by the execution provider used) prior to executing the graph (calling `Run()`). When the input is not copied to the target device, ORT copies it from the CPU as part of the `Run()` call. Similarly, if the output is not pre-allocated on the device, ORT assumes that the output is requested on the CPU and copies it from the device as the last step of the `Run()` call. This eats into the execution time of the graph, misleading users into thinking ORT is slow when the majority of the time is spent in these copies. To address this, we've introduced the notion of IOBinding. The key idea is to arrange for inputs to be copied to the device and for outputs to be pre-allocated on the device prior to calling `Run()`. IOBinding is available in all our language bindings. Following are code snippets in various languages demonstrating the usage of this feature. * C++ ```c++ Ort::Env env; Ort::Session session(env, model_path, session_options); Ort::IoBinding io_binding{session}; auto input_tensor = Ort::Value::CreateTensor(memory_info, input_tensor_values.data(), input_tensor_size, input_node_dims.data(), 4); io_binding.BindInput("input1", input_tensor); Ort::MemoryInfo output_mem_info{"Cuda", OrtDeviceAllocator, 0, OrtMemTypeDefault}; // Use this to bind output to a device when the shape is not known in advance. If the shape is known you can use the other overload of this function that takes an Ort::Value as input (IoBinding::BindOutput(const char* name, const Value& value)). // This internally calls the BindOutputToDevice C API. io_binding.BindOutput("output1", output_mem_info); session.Run(run_options, io_binding); ``` Notice that in the above code sample the output tensor is not allocated before binding it, rather an `Ort::MemoryInfo` is bound as output. This is an effective way to let the session allocate the tensor depending on the needed shapes. Especially for data dependent shapes or dynamic shapes this can be a great solution to get the right allocation. However in case the output shape is known and the output tensor should be reused it is beneficial to bind an `Ort::Value` to the output as well. This can be allocated using the session allocator or external memory. Please refer to the [device tensor docs](../device-tensor.md) for more details: ```c++ Ort::Allocator gpu_allocator(session, output_mem_info); auto output_value = Ort::Value::CreateTensor( gpu_allocator, output_shape.data(), output_shape.size(), ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT16); io_binding.BindOutput("output1", output_mem_info); ``` * Python (see [Python API docs](https://onnxruntime.ai/docs/api/python)) * C# (see [OrtIoBindingAllocationTest.cs](https://github.com/microsoft/onnxruntime/blob/main/csharp/test/Microsoft.ML.OnnxRuntime.Tests.Common/OrtIoBindingAllocationTest.cs)) ================================================ FILE: docs/performance/tune-performance/logging_tracing.md ================================================ --- title: Logging & Tracing grand_parent: Performance parent: Tune performance nav_order: 2 --- # Logging & Tracing ## Contents {: .no_toc } * TOC placeholder {:toc} ## Developer Logging ONNX Runtime has built-in cross-platform internal [printf style logging LOGS()](https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/common/logging/macros.h). This logging is available to configure in *production builds* for a dev **using the API**. There will likely be a performance penalty for using the default sink output (stdout) with higher log severity levels. ### log_severity_level [Python](https://onnxruntime.ai/docs/api/python/api_summary.html#onnxruntime.SessionOptions.log_severity_level) (below) - [C/C++ CreateEnv](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a22085f699a2d1adb52f809383f475ed1) / [OrtLoggingLevel](https://onnxruntime.ai/docs/api/c/group___global.html#ga1c0fbcf614dbd0e2c272ae1cc04c629c) - [.NET/C#](https://onnxruntime.ai/docs/api/csharp/api/Microsoft.ML.OnnxRuntime.SessionOptions.html#Microsoft_ML_OnnxRuntime_SessionOptions_LogSeverityLevel) ```python sess_opt = SessionOptions() sess_opt.log_severity_level = 0 // Verbose sess = ort.InferenceSession('model.onnx', sess_opt) ``` ### Note Note that [log_verbosity_level](https://onnxruntime.ai/docs/api/python/api_summary.html#onnxruntime.SessionOptions.log_verbosity_level) is a separate setting and only available in DEBUG custom builds. ## Tracing About Tracing is a super-set of logging in that tracing - Includes the previously mentioned logging - Adds tracing events that are more structured than printf style logging - Can be integrated with a larger tracing eco-system of the OS, such that - Tracing from multiple systems with ONNX, OS system level, and user-mode software that uses ONNX can be combined - Timestamps are high resolution and consistent with other traced components - Can log at high performance with a high number of events / second. - Events are not logged via stdout, but instead usually via a high performance in memory sink - Can be enabled dynamically at run-time to investigate issues including in production systems Currently, only Tracelogging combined with Windows ETW is supported, although [TraceLogging](https://github.com/microsoft/tracelogging) is cross-platform and support for other OSes instrumentation systems could be added. ## Tracing - Windows There are 2 main ONNX Runtime TraceLogging providers that can be enabled at run-time that can be captured with Windows [ETW](https://learn.microsoft.com/en-us/windows-hardware/test/weg/instrumenting-your-code-with-etw) ### Quickstart Tracing with WPR On Windows, you can use Windows Performance Recorder ([WPR](https://learn.microsoft.com/en-us/windows-hardware/test/wpt/wpr-command-line-options)) to capture a trace. The 2 providers covered below are already configured in these WPR profiles. - Download [ort.wprp](https://github.com/microsoft/onnxruntime/blob/main/ort.wprp) and [etw_provider.wprp](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/test/platform/windows/logging/etw_provider.wprp) (these could also be combined later) ```dos wpr -start ort.wprp -start etw_provider.wprp echo Repro the issue allowing ONNX to run wpr -stop onnx.etl -compress ``` ### ONNXRuntimeTraceLoggingProvider Beginning in ONNX Runtime 1.17 the [ONNXRuntimeTraceLoggingProvider](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/test/platform/windows/logging/HowToValidateEtwSinkOutput.md) can also be enabled. This will dynamically trace with high-performance the previously mentioned LOGS() macro printf logs that were previously only controlled by log_severity_level. A user or developer tracing with this provider will have the log severity level set dynamically with what ETW level they provide at run-time. Provider Name: ONNXRuntimeTraceLoggingProvider Provider GUID: 929DD115-1ECB-4CB5-B060-EBD4983C421D Keyword: Logs (0x2) keyword per [logging.h](https://github.com/ivberg/onnxruntime/blob/9cb97ee507b9b45d4a896f663590083e7e7568ac/include/onnxruntime/core/common/logging/logging.h#L83) Level: 1 (CRITICAL ) through 5 (VERBOSE) per [TraceLoggingLevel](https://learn.microsoft.com/en-us/windows/win32/api/traceloggingprovider/nf-traceloggingprovider-tracelogginglevel#remarks) ### Microsoft.ML.ONNXRuntime The [Microsoft.ML.ONNXRuntime](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/platform/windows/telemetry.cc#L47) provider provides structured logging. Provider Name: Microsoft.ML.ONNXRuntime Provider GUID: 3a26b1ff-7484-7484-7484-15261f42614d Keywords: Multiple per [logging.h](https://github.com/ivberg/onnxruntime/blob/9cb97ee507b9b45d4a896f663590083e7e7568ac/include/onnxruntime/core/common/logging/logging.h#L81) Level: 1 (CRITICAL ) through 5 (VERBOSE) per [TraceLoggingLevel](https://learn.microsoft.com/en-us/windows/win32/api/traceloggingprovider/nf-traceloggingprovider-tracelogginglevel#remarks) Note: This provider supports ETW [CaptureState](https://learn.microsoft.com/en-us/windows-hardware/test/wpt/capturestateonsave) (Rundown) for logging state for example when a trace is saved ORT 1.17 includes new events logging session options and EP provider options #### Profiling Microsoft.ML.ONNXRuntime can also output profiling events. That is covered in [profiling](profiling-tools.md) ### WinML WindowsML has it's own tracing providers that be enabled in addition the providers above - Microsoft.Windows.WinML - d766d9ff-112c-4dac-9247-241cf99d123f - Microsoft.Windows.AI.MachineLearning - BCAD6AEE-C08D-4F66-828C-4C43461A033D ================================================ FILE: docs/performance/tune-performance/memory.md ================================================ --- title: Memory consumption grand_parent: Performance parent: Tune performance nav_order: 3 --- # Reduce memory consumption ## Contents {: .no_toc } * TOC placeholder {:toc} ## Shared arena based allocator Memory consumption can be reduced between multiple sessions by configuring the shared arena based allocation. See the `Share allocator(s) between sessions` section in the [C API documentation](../../get-started/with-c.md). ## mimalloc allocator usage ONNX Runtime supports overriding memory allocations using [mimalloc](https://github.com/microsoft/mimalloc), a fast, general-purpose allocator. Depending on your model and usage, it can deliver single- or double-digit improvements in performance. The GitHub README page describes various scenarios on how mimalloc can be leveraged for performance tuning. mimalloc is a submodule in the ONNX Runtime source tree. On Windows, one can employ the `--use_mimalloc` build flag which builds a static version of mimalloc and links it to ONNX Runtime. This redirects ONNX Runtime allocators and all new/delete calls to mimalloc. Currently, there are no special provisions to employ mimalloc on Linux. It is recommended to use the LD_PRELOAD mechanism using pre-built binaries of mimalloc that you can build/obtain separately. ================================================ FILE: docs/performance/tune-performance/profiling-tools.md ================================================ --- title: Profiling tools grand_parent: Performance parent: Tune performance nav_order: 1 --- # Profiling Tools ## Contents {: .no_toc } * TOC placeholder {:toc} ## In-code performance profiling The onnxruntime_perf_test.exe tool (available from the build drop) can be used to test various knobs. Please find the usage instructions using `onnxruntime_perf_test.exe -h`. The [perf_view tool](https://github.com/microsoft/onnxruntime/tree/main/tools/perf_view) can also be used to render the statistics as a summarized view in the browser. You can enable ONNX Runtime latency profiling in code: ```python import onnxruntime as rt sess_options = rt.SessionOptions() sess_options.enable_profiling = True ``` If you are using the onnxruntime_perf_test.exe tool, you can add `-p [profile_file]` to enable performance profiling. In both cases, you will get a JSON file which contains the detailed performance data (threading, latency of each operator, etc). This file is a standard performance tracing file, and to view it in a user-friendly way, you can open it by using multiple tools. * (Windows) Use the WPA GUI to open the trace using the Perfetto OSS plugin - [Microsoft-Performance-Tools-Linux-Android](https://github.com/microsoft/Microsoft-Performance-Tools-Linux-Android) * [Perfetto UI](https://www.ui.perfetto.dev/) - Successor to Chrome Tracing UI * chrome://tracing: * Open a Chromium based browser such as Edge or Chrome * Type chrome://tracing in the address bar * Load the generated JSON file ## Execution Provider (EP) Profiling Starting with ONNX 1.17 support has been added to profile EPs or Neural Processing Unit (NPU)s, if that EP supports profiling in it's SDK ## Qualcomm QNN EP As mentioned in the [QNN EP Doc](../../execution-providers/QNN-ExecutionProvider.md) profiling is supported ### Cross-Platform CSV Tracing The Qualcomm AI Engine Direct SDK (QNN SDK) supports profiling. QNN will output to CSV in a text format if a dev were to use the QNN SDK directly outside ONNX. To enable equivalent functionality, ONNX mimics this support and outputs the same CSV formatting. If profiling_level is provided then ONNX will append log to current working directory a qnn-profiling-data.csv [file](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc#L911) ### TraceLogging ETW (Windows) Profiling As covered in [logging](logging_tracing.md) ONNX supports dynamic enablement of tracing ETW providers. Specifically the following settings. If the Tracelogging provider is enabled and profiling_level was provided, then CSV support is automatically disabled - Provider Name: Microsoft.ML.ONNXRuntime - Provider GUID: 3a26b1ff-7484-7484-7484-15261f42614d - Keywords: Profiling = 0x100 per [logging.h](https://github.com/ivberg/onnxruntime/blob/9cb97ee507b9b45d4a896f663590083e7e7568ac/include/onnxruntime/core/common/logging/logging.h#L81) - Level: - 5 (VERBOSE) = profiling_level=basic (good details without perf loss) - greater than 5 = profiling_level=detailed (individual ops are logged with inference perf hit) - Event: [QNNProfilingEvent](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc#L1083) ## GPU Profiling To profile CUDA kernels, please add the cupti library to your PATH and use the onnxruntime binary built from source with `--enable_cuda_profiling`. To profile ROCm kernels, please add the roctracer library to your PATH and use the onnxruntime binary built from source with `--enable_rocm_profiling`. Performance numbers from the device will then be attached to those from the host. For example: ```json {"cat":"Node", "name":"Add_1234", "dur":17, ...} {"cat":"Kernel", "name":"ort_add_cuda_kernel", dur:33, ...} ``` Here, the "Add" operator from the host initiated a CUDA kernel on device named "ort_add_cuda_kernel" which lasted for 33 microseconds. If an operator called multiple kernels during execution, the performance numbers of those kernels will all be listed following the call sequence: ```json {"cat":"Node", "name":, ...} {"cat":"Kernel", "name":, ...} {"cat":"Kernel", "name":, ...} ``` ================================================ FILE: docs/performance/tune-performance/threading.md ================================================ --- title: Thread management grand_parent: Performance parent: Tune performance nav_order: 4 --- # Thread management ## Contents {: .no_toc } * TOC placeholder {:toc} For the default CPU execution provider, setting defaults are provided to get fast inference performance. You can customize the performance using the following knobs in the API to control the thread count and other settings: Python (Defaults): ```python import onnxruntime as rt sess_options = rt.SessionOptions() sess_options.intra_op_num_threads = 0 sess_options.execution_mode = rt.ExecutionMode.ORT_SEQUENTIAL sess_options.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_ALL sess_options.add_session_config_entry("session.intra_op.allow_spinning", "1") ``` * INTRA Thread Count * Controls the _total_ number of INTRA threads to use to run the model. * INTRA = parallelize computation _inside_ each operator * Default: (not specified or 0). `sess_options.intra_op_num_threads = 0` * INTRA Threads Total = Number of physical CPU Cores. Leaving at default also enables some affinitization (explained below) * E.g. 6-core machine (with 12 HT logical processors) = 6 total INTRA threads * Sequential vs Parallel Execution * Controls whether _multiple_ operators in the graph (_across_ nodes) run sequentially or in parallel. * Default: `sess_options.execution_mode = rt.ExecutionMode.ORT_SEQUENTIAL` * Usually when a model has many branches, setting this option to `ORT_PARALLEL` will provide better performance. This could also hurt performance on some models without many branches. * When `sess_options.execution_mode = rt.ExecutionMode.ORT_PARALLEL`, you can set `sess_options.inter_op_num_threads` to control the number of threads used to parallelize the execution of the graph (_across_ nodes). * Graph Optimization Level * Default: `sess_options.graph_optimization_level = rt.GraphOptimizationLevel.ORT_ENABLE_ALL` enables all optimizations. * Please see [onnxruntime_c_api.h](https://github.com/microsoft/onnxruntime/tree/main/include/onnxruntime/core/session/onnxruntime_c_api.h#L286) (enum `GraphOptimizationLevel`) for the full list of all optimization levels. For details regarding available optimizations and usage, please refer to the [Graph Optimizations](../model-optimizations/graph-optimizations.md) documentation. * Thread-Pool Spinning Behavior * Controls whether additional INTRA or INTER threads spin waiting for work. Provides faster inference but consumes more CPU cycles, resources, and power * Default: 1 (Enabled) * `spin_duration_us`: optional time-bounded spin window in microseconds (not set by default; uses legacy fixed iteration count) * `spin_backoff_max`: optional exponential-backoff cap for spin pause density (default `1`, no backoff). Set to a power of two (e.g. `8`) to reduce CPU/power usage during the spin window ## Set number of intra-op threads Onnxruntime sessions utilize multi-threading to parallelize computation _inside_ each operator. By default with intra_op_num_threads=0 or not set, each session will start with the main thread on the 1st core (not affinitized). Then extra threads per additional physical core are created, and affinitized to that core (1 or 2 logical processors). Customer could manually configure the total number of threads like: [Python](https://onnxruntime.ai/docs/api/python/api_summary.html#onnxruntime.SessionOptions.intra_op_num_threads) (below) - [C/C++](https://onnxruntime.ai/docs/api/c/struct_ort_api.html) - [.NET/C#](https://onnxruntime.ai/docs/api/csharp/api/Microsoft.ML.OnnxRuntime.SessionOptions.html#Microsoft_ML_OnnxRuntime_SessionOptions_IntraOpNumThreads) ```python sess_opt = SessionOptions() sess_opt.intra_op_num_threads = 3 sess = ort.InferenceSession('model.onnx', sess_opt) ``` With the above configuration of 3 total threads, two extra threads will be created in the addtional INTRA pool, so along with the main calling thread, there will be three threads in total to participate in intra-op computation. However, if customer explicitly set the number of threads like showcased above, there will be no affinity set to any of the created thread. In addition, Onnxruntime also allow customers to create a global intra-op thread pool to prevent overheated contentions among session thread pools, please find its usage [here](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/onnxruntime/test/global_thread_pools/test_main.cc#L98). ## Thread spinning behavior Controls whether additional INTRA or INTER threads spin waiting for work. Provides faster inference but consumes more CPU cycles, resources, and power. Example disabling spinning so WorkerLoop doesn't consume extra active cycles spinning waiting or attempting to steal work [Python](https://onnxruntime.ai/docs/api/python/api_summary.html#onnxruntime.SessionOptions.add_session_config_entry) (below) - [C++](https://onnxruntime.ai/docs/api/c/struct_ort_api.html) - [.NET/C#](https://onnxruntime.ai/docs/api/csharp/api/Microsoft.ML.OnnxRuntime.SessionOptions.html#Microsoft_ML_OnnxRuntime_SessionOptions_AddSessionConfigEntry_System_String_System_String_) - [Keys](https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h) ```python sess_opt = SessionOptions() sess_opt.AddConfigEntry("session.intra_op.allow_spinning", "0") sess_opt.AddConfigEntry("session.inter_op.allow_spinning", "0") ``` ### Spin duration By default, thread pool workers spin for a fixed number of iterations before going to sleep. The `session.intra_op.spin_duration_us` and `session.inter_op.spin_duration_us` config keys let you specify a time-bounded spin window in microseconds instead. At session creation the runtime calibrates how many spin-loop iterations fit into the requested duration, so the actual spin time adapts to the host CPU speed. * Default: not set (uses the legacy fixed iteration count). * Setting the value to `0` disables spinning entirely (equivalent to `allow_spinning = 0`). * A positive value (e.g. `1000` for 1 ms) caps the spin window to that duration. ```python sess_opt = SessionOptions() # Spin for at most 1 ms before sleeping sess_opt.add_session_config_entry("session.intra_op.spin_duration_us", "1000") sess_opt.add_session_config_entry("session.inter_op.spin_duration_us", "1000") sess = ort.InferenceSession('model.onnx', sess_opt) ``` ### Spin backoff (exponential) When spinning is enabled, each spin iteration normally executes a single `SpinPause()` instruction. The `session.intra_op.spin_backoff_max` and `session.inter_op.spin_backoff_max` config keys activate an **exponential-backoff** mode: each successive iteration doubles the number of `SpinPause()` calls (1, 2, 4, … capped at `spin_backoff_max`). This reduces pause-instruction density and lowers CPU/power usage during the spin window — particularly beneficial on hybrid (P-core / E-core) and mobile platforms. The iteration count is automatically scaled so the total wall-clock spin budget (set by `spin_duration_us` or the legacy default) is preserved. * Default: `1` (one `SpinPause()` per iteration — identical to existing behavior). * Must be a power of two (e.g. 1, 2, 4, 8, …). Values that are not a power of two are rounded down. * Subordinate to `allow_spinning` — when spinning is disabled, this setting is ignored. * Composable with `spin_duration_us` — the two knobs are orthogonal and can be combined. ```python sess_opt = SessionOptions() # Combine 1 ms time-bounded spinning with exponential backoff capped at 8 sess_opt.add_session_config_entry("session.intra_op.spin_duration_us", "1000") sess_opt.add_session_config_entry("session.intra_op.spin_backoff_max", "8") sess = ort.InferenceSession('model.onnx', sess_opt) ``` In [benchmarks](https://github.com/microsoft/onnxruntime/pull/28096), `spin_duration_us=1000` combined with `spin_backoff_max=8` was the most consistent best performer across models and thread counts. ## Set number of inter-op threads A inter-op thread pool is for parallelism _between_ operators, and will only be created when session execution mode set to parallel: By default, inter-op thread pool will also have one thread per physical core. [Python](https://onnxruntime.ai/docs/api/python/api_summary.html#onnxruntime.SessionOptions.inter_op_num_threads) (below) - [C/C++](https://onnxruntime.ai/docs/api/c/struct_ort_api.html) - [.NET/C#](https://onnxruntime.ai/docs/api/csharp/api/Microsoft.ML.OnnxRuntime.SessionOptions.html#Microsoft_ML_OnnxRuntime_SessionOptions_InterOpNumThreads) ```python sess_opt = SessionOptions() sess_opt.execution_mode = ExecutionMode.ORT_PARALLEL sess_opt.inter_op_num_threads = 3 sess = ort.InferenceSession('model.onnx', sess_opt) ``` ## Set intra-op thread affinity It is normally best to not set thread affinity and let the OS handle thread assignment for perf and power reasons. However, for certain scenarios, it may be beneficial to customize intra-op thread affinities, for example: * There are multiple sessions run in parallel, customer might prefer their intra-op thread pools run on separate cores to avoid contention. * Customer want to limit a intra-op thread pool to run on only one of the NUMA nodes to reduce overhead of expensive cache miss among nodes. For session intra-op thread pool, please read the [configuration](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h#L180) and consume it like: [Python](https://onnxruntime.ai/docs/api/python/api_summary.html#onnxruntime.SessionOptions.add_session_config_entry) (below) - [C++](https://onnxruntime.ai/docs/api/c/struct_ort_api.html) - [.NET/C#](https://onnxruntime.ai/docs/api/csharp/api/Microsoft.ML.OnnxRuntime.SessionOptions.html#Microsoft_ML_OnnxRuntime_SessionOptions_AddSessionConfigEntry_System_String_System_String_) - [Keys](https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h#L176) ```python sess_opt = SessionOptions() sess_opt.intra_op_num_threads = 3 sess_opt.add_session_config_entry('session.intra_op_thread_affinities', '1;2') sess = ort.InferenceSession('model.onnx', sess_opt, ...) ``` For global thread pool, please read the [API](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/include/onnxruntime/core/session/onnxruntime_c_api.h#L3636) and [usage](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/onnxruntime/test/global_thread_pools/test_main.cc#L98). ## Numa support and performance tuning Since release 1.14, Onnxruntime thread pool could utilize all physical cores that are available over NUMA nodes. The intra-op thread pool will create an extra thread on every physical core (except the 1st core). E.g. assume there is a system of 2 NUMA nodes, each has 24 cores. Hence intra-op thread pool will create 47 threads, and set thread affinity to each core. For NUMA systems, it is recommended to test a few thread settings to explore for best performance, in that threads allocated among NUMA nodes might has higher cache-miss overhead when cooperating with each other. For example, when number of intra-op threads has to be 8, there are different ways to set affinity: [Python]() (below) - [C++]() - [.NET/C#]() ```python sess_opt = SessionOptions() sess_opt.intra_op_num_threads = 8 sess_opt.add_session_config_entry('session.intra_op_thread_affinities', '3,4;5,6;7,8;9,10;11,12;13,14;15,16') # set affinities of all 7 threads to cores in the first NUMA node # sess_opt.add_session_config_entry('session.intra_op_thread_affinities', '3,4;5,6;7,8;9,10;49,50;51,52;53,54') # set affinities for first 4 threads to the first NUMA node, and others to the second sess = ort.InferenceSession('resnet50.onnx', sess_opt, ...) ``` Test showed that setting affinities to a single NUMA node has nearly 20 percent performance improvement aginst the other case. ## Custom threading callbacks Occasionally, users may prefer to use their own fine-tuned threads for multithreading. ORT offers thread creation and joining callbacks in the [C++ API](https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_cxx_api.h): ```c++ std::vector threads; void* custom_thread_creation_options = nullptr; // initialize custom_thread_creation_options // On thread pool creation, ORT calls CreateThreadCustomized to create a thread OrtCustomThreadHandle CreateThreadCustomized(void* custom_thread_creation_options, OrtThreadWorkerFn work_loop, void* param) { threads.push_back(std::thread(work_loop, param)); // configure the thread by custom_thread_creation_options return reinterpret_cast(threads.back().native_handle()); } // On thread pool destruction, ORT calls JoinThreadCustomized for each created thread void JoinThreadCustomized(OrtCustomThreadHandle handle) { for (auto& t : threads) { if (reinterpret_cast(t.native_handle()) == handle) { // recycling resources ... t.join(); } } } int main(...) { ... Ort::Env ort_env; Ort::SessionOptions session_options; session_options.SetCustomCreateThreadFn(CreateThreadCustomized); session_options.SetCustomThreadCreationOptions(&custom_thread_creation_options); session_options.SetCustomJoinThreadFn(JoinThreadCustomized); Ort::Session session(*ort_env, MODEL_URI, session_options); ... } ``` For global thread pool: ```c++ int main() { const OrtApi* g_ort = OrtGetApiBase()->GetApi(ORT_API_VERSION); OrtThreadingOptions* tp_options = nullptr; g_ort->CreateThreadingOptions(&tp_options); g_ort->SetGlobalCustomCreateThreadFn(tp_options, CreateThreadCustomized); g_ort->SetGlobalCustomThreadCreationOptions(tp_options, &custom_thread_creation_options); g_ort->SetGlobalCustomJoinThreadFn(tp_options, JoinThreadCustomized); // disable per-session thread pool, create a session for inferencing g_ort->ReleaseThreadingOptions(tp_options); } ``` Note that `CreateThreadCustomized` and `JoinThreadCustomized`, once set, will be applied to both ORT intra op and inter op thread pools uniformly. ## Usage in custom ops Since 1.17, custom op developers are entitled to parallelize their cpu code with ort intra-op thread pool. Please refer to the [API](https://github.com/microsoft/onnxruntime/blob/rel-1.17.0/include/onnxruntime/core/session/onnxruntime_c_api.h#L4543), and [example](https://github.com/microsoft/onnxruntime/blob/rel-1.17.0/onnxruntime/test/testdata/custom_op_library/cpu/cpu_ops.cc#L87) for usage. ================================================ FILE: docs/performance/tune-performance/troubleshooting.md ================================================ --- title: Troubleshooting grand_parent: Performance parent: Tune performance nav_order: 6 --- # Troubleshooting performance issues ## Contents {: .no_toc } * TOC placeholder {:toc} Here is a list of things to check when assessing performance issues: * Have you enabled all [graph optimizations](../model-optimizations/graph-optimizations.md)? The official published packages do enable all by default but when building from source, check that these are enabled in your build. * Have you searched through prior-filed [GitHub issues](https://github.com/microsoft/onnxruntime/issues) to see if your problem has been discussed previously? Please do this before filing new issues. * If using CUDA or TensorRT, do you have the right versions of the dependent libraries installed? [CUDA EP](../../execution-providers/CUDA-ExecutionProvider.md#requirements) / [TensorRT EP](../../execution-providers/TensorRT-ExecutionProvider.md#requirements) ## Why is the model graph not optimized even with graph_optimization_level set to ORT_ENABLE_ALL? The ONNX model from IR_VERSION 4 only treats initializers that appear in graph input as non-constant. This may prevent some of the graph optimizations like const folding, operator fusion etc. Move initializers out of graph inputs if there is no need to override them, by either re-generating the model with the latest exporter/converter or with the tool [remove_initializer_from_input.py](https://github.com/microsoft/onnxruntime/tree/main/tools/python/remove_initializer_from_input.py). ## Why is my model running slower on GPU than on CPU? Depending on which execution provider you're using, it may not have full support for all the operators in your model. Fallback to CPU ops can cause hits in performance speed. Moreover, even if an op is implemented by the CUDA execution provider, it may not necessarily assign/place the op to the CUDA EP due to performance reasons. To see the placement decided by ORT, turn on verbose logging and look at the console output. ## My converted TensorFlow model is slow - why? NCHW and NHWC are two different memory layout for 4-D tensors. Most TensorFlow operations used by a CNN support both NHWC and NCHW data format. The TensorFlow team suggests that on GPUs NCHW is faster but on CPUs NHWC is sometimes faster in TensorFlow. However, ONNX only supports NCHW. As a result, if the original model is in NHWC format, extra transposes may be added when the model is converted. The [tensorflow-onnx](https://github.com/onnx/tensorflow-onnx) converter does remove many of these transposes, but if this doesn't help sufficiently, consider retraining the model in the NCHW format. ## I am seeing high latency variance. On some platforms, onnxruntime may exhibit high latency variance during inferencing. This is caused by the constant cost model that onnxruntime uses to parallelize tasks in the thread pool. For each task, the constant cost model will calculate a granularity for parallelization among threads, which stays constant to the end of the task execution. This approach can bring imbalanced load sometimes, causing high latency variance. To mitigate this, onnxruntime provides a dynamic cost model which can be enabled as a session option: ```python sess_options.add_session_config_entry('session.dynamic_block_base', '4') ``` Whenever set with a positive value, the onnxruntime thread pool will parallelize internal tasks with a decreasing granularity. Specifically, assuming there is a function expected to run N number of times by the thread pool, with the dynamic cost model enabled, each thread in the pool will claim ```python residual_of_N / (dynamic_block_base * num_of_threads) ``` whenever it is ready to run. So over a period of time, threads in the pool are likely to be better load balanced, thereby lowering the latency variance. Due to the same reason, the dynamic cost model may also improve the performance for cases when threads are more likely be preempted. Per our tests, by far the best configuration for dynamic_block_base is 4, which lowers the variance while keeping good performance. ## I am seeing high CPU usage on windows It is observed that for machines have more than 64 logical cores, CPU usage could be notably lowered by letting the thread pool use a lock-free task queue, which utilizes spinlock instead of mutex for synchronization. The lock-free task queue could be enabled by building onnxruntime from source with following flag: ``` --use_lock_free_queue ```