FP16 -- [08/02/2023-10:56:39] [E] Saving engine to file failed. [08/02/2023-10:56:39] [E] Engine set up failed
Closed this issue · 1 comments
monajalal commented
I get this error for fp16:
(base) mona@ard-gpu-01:~/sdg_pallet_model$ ./build_trt_fp16.sh pallet_model_v1_all.onnx all_pallets_out
&&&& RUNNING TensorRT.trtexec [TensorRT v8503] # /usr/src/tensorrt/bin/trtexec --onnx=pallet_model_v1_all.onnx --minShapes=input:1x3x192x192 --maxShapes=input:1x3x1536x1536 --optShapes=input:1x3x256x256 --saveEngine=all_pallets_out --fp16
[08/02/2023-10:56:08] [I] === Model Options ===
[08/02/2023-10:56:08] [I] Format: ONNX
[08/02/2023-10:56:08] [I] Model: pallet_model_v1_all.onnx
[08/02/2023-10:56:08] [I] Output:
[08/02/2023-10:56:08] [I] === Build Options ===
[08/02/2023-10:56:08] [I] Max batch: explicit batch
[08/02/2023-10:56:08] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[08/02/2023-10:56:08] [I] minTiming: 1
[08/02/2023-10:56:08] [I] avgTiming: 8
[08/02/2023-10:56:08] [I] Precision: FP32+FP16
[08/02/2023-10:56:08] [I] LayerPrecisions:
[08/02/2023-10:56:08] [I] Calibration:
[08/02/2023-10:56:08] [I] Refit: Disabled
[08/02/2023-10:56:08] [I] Sparsity: Disabled
[08/02/2023-10:56:08] [I] Safe mode: Disabled
[08/02/2023-10:56:08] [I] DirectIO mode: Disabled
[08/02/2023-10:56:08] [I] Restricted mode: Disabled
[08/02/2023-10:56:08] [I] Build only: Disabled
[08/02/2023-10:56:08] [I] Save engine: all_pallets_out
[08/02/2023-10:56:08] [I] Load engine:
[08/02/2023-10:56:08] [I] Profiling verbosity: 0
[08/02/2023-10:56:08] [I] Tactic sources: Using default tactic sources
[08/02/2023-10:56:08] [I] timingCacheMode: local
[08/02/2023-10:56:08] [I] timingCacheFile:
[08/02/2023-10:56:08] [I] Heuristic: Disabled
[08/02/2023-10:56:08] [I] Preview Features: Use default preview flags.
[08/02/2023-10:56:08] [I] Input(s)s format: fp32:CHW
[08/02/2023-10:56:08] [I] Output(s)s format: fp32:CHW
[08/02/2023-10:56:08] [I] Input build shape: input=1x3x192x192+1x3x256x256+1x3x1536x1536
[08/02/2023-10:56:08] [I] Input calibration shapes: model
[08/02/2023-10:56:08] [I] === System Options ===
[08/02/2023-10:56:08] [I] Device: 0
[08/02/2023-10:56:08] [I] DLACore:
[08/02/2023-10:56:08] [I] Plugins:
[08/02/2023-10:56:08] [I] === Inference Options ===
[08/02/2023-10:56:08] [I] Batch: Explicit
[08/02/2023-10:56:08] [I] Input inference shape: input=1x3x256x256
[08/02/2023-10:56:08] [I] Iterations: 10
[08/02/2023-10:56:08] [I] Duration: 3s (+ 200ms warm up)
[08/02/2023-10:56:08] [I] Sleep time: 0ms
[08/02/2023-10:56:08] [I] Idle time: 0ms
[08/02/2023-10:56:08] [I] Streams: 1
[08/02/2023-10:56:08] [I] ExposeDMA: Disabled
[08/02/2023-10:56:08] [I] Data transfers: Enabled
[08/02/2023-10:56:08] [I] Spin-wait: Disabled
[08/02/2023-10:56:08] [I] Multithreading: Disabled
[08/02/2023-10:56:08] [I] CUDA Graph: Disabled
[08/02/2023-10:56:08] [I] Separate profiling: Disabled
[08/02/2023-10:56:08] [I] Time Deserialize: Disabled
[08/02/2023-10:56:08] [I] Time Refit: Disabled
[08/02/2023-10:56:08] [I] NVTX verbosity: 0
[08/02/2023-10:56:08] [I] Persistent Cache Ratio: 0
[08/02/2023-10:56:08] [I] Inputs:
[08/02/2023-10:56:08] [I] === Reporting Options ===
[08/02/2023-10:56:08] [I] Verbose: Disabled
[08/02/2023-10:56:08] [I] Averages: 10 inferences
[08/02/2023-10:56:08] [I] Percentiles: 90,95,99
[08/02/2023-10:56:08] [I] Dump refittable layers:Disabled
[08/02/2023-10:56:08] [I] Dump output: Disabled
[08/02/2023-10:56:08] [I] Profile: Disabled
[08/02/2023-10:56:08] [I] Export timing to JSON file:
[08/02/2023-10:56:08] [I] Export output to JSON file:
[08/02/2023-10:56:08] [I] Export profile to JSON file:
[08/02/2023-10:56:08] [I]
[08/02/2023-10:56:08] [I] === Device Information ===
[08/02/2023-10:56:08] [I] Selected Device: NVIDIA GeForce RTX 3080 Laptop GPU
[08/02/2023-10:56:08] [I] Compute Capability: 8.6
[08/02/2023-10:56:08] [I] SMs: 48
[08/02/2023-10:56:08] [I] Compute Clock Rate: 1.365 GHz
[08/02/2023-10:56:08] [I] Device Global Memory: 16116 MiB
[08/02/2023-10:56:08] [I] Shared Memory per SM: 100 KiB
[08/02/2023-10:56:08] [I] Memory Bus Width: 256 bits (ECC disabled)
[08/02/2023-10:56:08] [I] Memory Clock Rate: 6.001 GHz
[08/02/2023-10:56:08] [I]
[08/02/2023-10:56:08] [I] TensorRT version: 8.5.3
[08/02/2023-10:56:08] [I] [TRT] [MemUsageChange] Init CUDA: CPU +568, GPU +0, now: CPU 580, GPU 455 (MiB)
[08/02/2023-10:56:10] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +542, GPU +116, now: CPU 1177, GPU 571 (MiB)
[08/02/2023-10:56:10] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
[08/02/2023-10:56:10] [I] Start parsing network model
[08/02/2023-10:56:10] [I] [TRT] ----------------------------------------------------------------
[08/02/2023-10:56:10] [I] [TRT] Input filename: pallet_model_v1_all.onnx
[08/02/2023-10:56:10] [I] [TRT] ONNX IR version: 0.0.7
[08/02/2023-10:56:10] [I] [TRT] Opset version: 14
[08/02/2023-10:56:10] [I] [TRT] Producer name: pytorch
[08/02/2023-10:56:10] [I] [TRT] Producer version: 2.0.0
[08/02/2023-10:56:10] [I] [TRT] Domain:
[08/02/2023-10:56:10] [I] [TRT] Model version: 0
[08/02/2023-10:56:10] [I] [TRT] Doc string:
[08/02/2023-10:56:10] [I] [TRT] ----------------------------------------------------------------
[08/02/2023-10:56:10] [I] Finish parsing network model
[08/02/2023-10:56:11] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1301, GPU +362, now: CPU 2589, GPU 933 (MiB)
[08/02/2023-10:56:11] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +247, GPU +58, now: CPU 2836, GPU 991 (MiB)
[08/02/2023-10:56:11] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[08/02/2023-10:56:38] [I] [TRT] Total Activation Memory: 19244719104
[08/02/2023-10:56:38] [I] [TRT] Detected 1 inputs and 2 output network tensors.
[08/02/2023-10:56:39] [I] [TRT] Total Host Persistent Memory: 199040
[08/02/2023-10:56:39] [I] [TRT] Total Device Persistent Memory: 3539456
[08/02/2023-10:56:39] [I] [TRT] Total Scratch Memory: 0
[08/02/2023-10:56:39] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 55 MiB, GPU 8510 MiB
[08/02/2023-10:56:39] [I] [TRT] [BlockAssignment] Started assigning block shifts. This will take 98 steps to complete.
[08/02/2023-10:56:39] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 1.83672ms to assign 10 blocks to 98 nodes requiring 450626560 bytes.
[08/02/2023-10:56:39] [I] [TRT] Total Activation Memory: 450626560
[08/02/2023-10:56:39] [W] [TRT] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[08/02/2023-10:56:39] [W] [TRT] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[08/02/2023-10:56:39] [W] [TRT] Check verbose logs for the list of affected weights.
[08/02/2023-10:56:39] [W] [TRT] - 63 weights are affected by this issue: Detected subnormal FP16 values.
[08/02/2023-10:56:39] [W] [TRT] - 27 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
[08/02/2023-10:56:39] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +55, GPU +58, now: CPU 55, GPU 58 (MiB)
[08/02/2023-10:56:39] [E] Saving engine to file failed.
[08/02/2023-10:56:39] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8503] # /usr/src/tensorrt/bin/trtexec --onnx=pallet_model_v1_all.onnx --minShapes=input:1x3x192x192 --maxShapes=input:1x3x1536x1536 --optShapes=input:1x3x256x256 --saveEngine=all_pallets_out --fp16
monajalal commented
dummy fix but I think it wasn't quite clear we want a dir or file. we need a file
(base) mona@ard-gpu-01:~/sdg_pallet_model$ ./build_trt_fp16.sh pallet_model_v1_all.onnx all_pallets_out/out.onnx
&&&& RUNNING TensorRT.trtexec [TensorRT v8503] # /usr/src/tensorrt/bin/trtexec --onnx=pallet_model_v1_all.onnx --minShapes=input:1x3x192x192 --maxShapes=input:1x3x1536x1536 --optShapes=input:1x3x256x256 --saveEngine=all_pallets_out/out.onnx --fp16
[08/02/2023-11:07:13] [I] === Model Options ===
[08/02/2023-11:07:13] [I] Format: ONNX
[08/02/2023-11:07:13] [I] Model: pallet_model_v1_all.onnx
[08/02/2023-11:07:13] [I] Output:
[08/02/2023-11:07:13] [I] === Build Options ===
[08/02/2023-11:07:13] [I] Max batch: explicit batch
[08/02/2023-11:07:13] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[08/02/2023-11:07:13] [I] minTiming: 1
[08/02/2023-11:07:13] [I] avgTiming: 8
[08/02/2023-11:07:13] [I] Precision: FP32+FP16
[08/02/2023-11:07:13] [I] LayerPrecisions:
[08/02/2023-11:07:13] [I] Calibration:
[08/02/2023-11:07:13] [I] Refit: Disabled
[08/02/2023-11:07:13] [I] Sparsity: Disabled
[08/02/2023-11:07:13] [I] Safe mode: Disabled
[08/02/2023-11:07:13] [I] DirectIO mode: Disabled
[08/02/2023-11:07:13] [I] Restricted mode: Disabled
[08/02/2023-11:07:13] [I] Build only: Disabled
[08/02/2023-11:07:13] [I] Save engine: all_pallets_out/out.onnx
[08/02/2023-11:07:13] [I] Load engine:
[08/02/2023-11:07:13] [I] Profiling verbosity: 0
[08/02/2023-11:07:13] [I] Tactic sources: Using default tactic sources
[08/02/2023-11:07:13] [I] timingCacheMode: local
[08/02/2023-11:07:13] [I] timingCacheFile:
[08/02/2023-11:07:13] [I] Heuristic: Disabled
[08/02/2023-11:07:13] [I] Preview Features: Use default preview flags.
[08/02/2023-11:07:13] [I] Input(s)s format: fp32:CHW
[08/02/2023-11:07:13] [I] Output(s)s format: fp32:CHW
[08/02/2023-11:07:13] [I] Input build shape: input=1x3x192x192+1x3x256x256+1x3x1536x1536
[08/02/2023-11:07:13] [I] Input calibration shapes: model
[08/02/2023-11:07:13] [I] === System Options ===
[08/02/2023-11:07:13] [I] Device: 0
[08/02/2023-11:07:13] [I] DLACore:
[08/02/2023-11:07:13] [I] Plugins:
[08/02/2023-11:07:13] [I] === Inference Options ===
[08/02/2023-11:07:13] [I] Batch: Explicit
[08/02/2023-11:07:13] [I] Input inference shape: input=1x3x256x256
[08/02/2023-11:07:13] [I] Iterations: 10
[08/02/2023-11:07:13] [I] Duration: 3s (+ 200ms warm up)
[08/02/2023-11:07:13] [I] Sleep time: 0ms
[08/02/2023-11:07:13] [I] Idle time: 0ms
[08/02/2023-11:07:13] [I] Streams: 1
[08/02/2023-11:07:13] [I] ExposeDMA: Disabled
[08/02/2023-11:07:13] [I] Data transfers: Enabled
[08/02/2023-11:07:13] [I] Spin-wait: Disabled
[08/02/2023-11:07:13] [I] Multithreading: Disabled
[08/02/2023-11:07:13] [I] CUDA Graph: Disabled
[08/02/2023-11:07:13] [I] Separate profiling: Disabled
[08/02/2023-11:07:13] [I] Time Deserialize: Disabled
[08/02/2023-11:07:13] [I] Time Refit: Disabled
[08/02/2023-11:07:13] [I] NVTX verbosity: 0
[08/02/2023-11:07:13] [I] Persistent Cache Ratio: 0
[08/02/2023-11:07:13] [I] Inputs:
[08/02/2023-11:07:13] [I] === Reporting Options ===
[08/02/2023-11:07:13] [I] Verbose: Disabled
[08/02/2023-11:07:13] [I] Averages: 10 inferences
[08/02/2023-11:07:13] [I] Percentiles: 90,95,99
[08/02/2023-11:07:13] [I] Dump refittable layers:Disabled
[08/02/2023-11:07:13] [I] Dump output: Disabled
[08/02/2023-11:07:13] [I] Profile: Disabled
[08/02/2023-11:07:13] [I] Export timing to JSON file:
[08/02/2023-11:07:13] [I] Export output to JSON file:
[08/02/2023-11:07:13] [I] Export profile to JSON file:
[08/02/2023-11:07:13] [I]
[08/02/2023-11:07:13] [I] === Device Information ===
[08/02/2023-11:07:13] [I] Selected Device: NVIDIA GeForce RTX 3080 Laptop GPU
[08/02/2023-11:07:13] [I] Compute Capability: 8.6
[08/02/2023-11:07:13] [I] SMs: 48
[08/02/2023-11:07:13] [I] Compute Clock Rate: 1.365 GHz
[08/02/2023-11:07:13] [I] Device Global Memory: 16116 MiB
[08/02/2023-11:07:13] [I] Shared Memory per SM: 100 KiB
[08/02/2023-11:07:13] [I] Memory Bus Width: 256 bits (ECC disabled)
[08/02/2023-11:07:13] [I] Memory Clock Rate: 6.001 GHz
[08/02/2023-11:07:13] [I]
[08/02/2023-11:07:13] [I] TensorRT version: 8.5.3
[08/02/2023-11:07:14] [I] [TRT] [MemUsageChange] Init CUDA: CPU +568, GPU +0, now: CPU 580, GPU 455 (MiB)
[08/02/2023-11:07:15] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +542, GPU +116, now: CPU 1177, GPU 571 (MiB)
[08/02/2023-11:07:15] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
[08/02/2023-11:07:15] [I] Start parsing network model
[08/02/2023-11:07:15] [I] [TRT] ----------------------------------------------------------------
[08/02/2023-11:07:15] [I] [TRT] Input filename: pallet_model_v1_all.onnx
[08/02/2023-11:07:15] [I] [TRT] ONNX IR version: 0.0.7
[08/02/2023-11:07:15] [I] [TRT] Opset version: 14
[08/02/2023-11:07:15] [I] [TRT] Producer name: pytorch
[08/02/2023-11:07:15] [I] [TRT] Producer version: 2.0.0
[08/02/2023-11:07:15] [I] [TRT] Domain:
[08/02/2023-11:07:15] [I] [TRT] Model version: 0
[08/02/2023-11:07:15] [I] [TRT] Doc string:
[08/02/2023-11:07:15] [I] [TRT] ----------------------------------------------------------------
[08/02/2023-11:07:15] [I] Finish parsing network model
[08/02/2023-11:07:16] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1301, GPU +362, now: CPU 2589, GPU 933 (MiB)
[08/02/2023-11:07:16] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +247, GPU +58, now: CPU 2836, GPU 991 (MiB)
[08/02/2023-11:07:16] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
[08/02/2023-11:07:44] [I] [TRT] Total Activation Memory: 19244717056
[08/02/2023-11:07:44] [I] [TRT] Detected 1 inputs and 2 output network tensors.
[08/02/2023-11:07:44] [I] [TRT] Total Host Persistent Memory: 194560
[08/02/2023-11:07:44] [I] [TRT] Total Device Persistent Memory: 3539456
[08/02/2023-11:07:44] [I] [TRT] Total Scratch Memory: 0
[08/02/2023-11:07:44] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 55 MiB, GPU 8510 MiB
[08/02/2023-11:07:44] [I] [TRT] [BlockAssignment] Started assigning block shifts. This will take 94 steps to complete.
[08/02/2023-11:07:44] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 1.67377ms to assign 9 blocks to 94 nodes requiring 450626048 bytes.
[08/02/2023-11:07:44] [I] [TRT] Total Activation Memory: 450626048
[08/02/2023-11:07:44] [W] [TRT] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[08/02/2023-11:07:44] [W] [TRT] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[08/02/2023-11:07:44] [W] [TRT] Check verbose logs for the list of affected weights.
[08/02/2023-11:07:44] [W] [TRT] - 63 weights are affected by this issue: Detected subnormal FP16 values.
[08/02/2023-11:07:44] [W] [TRT] - 27 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
[08/02/2023-11:07:44] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +55, GPU +58, now: CPU 55, GPU 58 (MiB)
[08/02/2023-11:07:44] [I] Engine built in 31.1852 sec.
[08/02/2023-11:07:45] [I] [TRT] Loaded engine size: 55 MiB
[08/02/2023-11:07:45] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +58, now: CPU 0, GPU 58 (MiB)
[08/02/2023-11:07:45] [I] Engine deserialized in 0.0113893 sec.
[08/02/2023-11:07:45] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +433, now: CPU 0, GPU 491 (MiB)
[08/02/2023-11:07:45] [W] [TRT] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
[08/02/2023-11:07:45] [I] Setting persistentCacheLimit to 0 bytes.
[08/02/2023-11:07:45] [I] Using random values for input input
[08/02/2023-11:07:45] [I] Created input binding for input with dimensions 1x3x256x256
[08/02/2023-11:07:45] [I] Using random values for output heatmap
[08/02/2023-11:07:45] [I] Created output binding for heatmap with dimensions 1x1x256x256
[08/02/2023-11:07:45] [I] Using random values for output vectormap
[08/02/2023-11:07:45] [I] Created output binding for vectormap with dimensions 1x16x256x256
[08/02/2023-11:07:45] [I] Starting inference
[08/02/2023-11:07:48] [I] Warmup completed 141 queries over 200 ms
[08/02/2023-11:07:48] [I] Timing trace has 2066 queries over 3.00418 s
[08/02/2023-11:07:48] [I]
[08/02/2023-11:07:48] [I] === Trace details ===
[08/02/2023-11:07:48] [I] Trace averages of 10 runs:
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45971 ms - Host latency: 1.69382 ms (enqueue 0.229193 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46258 ms - Host latency: 1.69739 ms (enqueue 0.226613 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.40831 ms - Host latency: 1.64128 ms (enqueue 0.225211 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4125 ms - Host latency: 1.63504 ms (enqueue 0.223546 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48817 ms - Host latency: 1.7233 ms (enqueue 0.230911 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48542 ms - Host latency: 1.71089 ms (enqueue 0.244614 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44793 ms - Host latency: 1.68636 ms (enqueue 0.228146 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41926 ms - Host latency: 1.66215 ms (enqueue 0.23916 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44425 ms - Host latency: 1.67543 ms (enqueue 0.233484 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.39285 ms - Host latency: 1.62079 ms (enqueue 0.224023 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.40247 ms - Host latency: 1.66063 ms (enqueue 0.219098 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44189 ms - Host latency: 1.6725 ms (enqueue 0.199524 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41629 ms - Host latency: 1.68914 ms (enqueue 0.237024 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.39612 ms - Host latency: 1.63324 ms (enqueue 0.229416 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43617 ms - Host latency: 1.69265 ms (enqueue 0.236938 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.50385 ms - Host latency: 1.76374 ms (enqueue 0.229822 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.49165 ms - Host latency: 1.74723 ms (enqueue 0.240463 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46053 ms - Host latency: 1.70718 ms (enqueue 0.239542 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43442 ms - Host latency: 1.67149 ms (enqueue 0.241501 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43994 ms - Host latency: 1.66942 ms (enqueue 0.234094 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44732 ms - Host latency: 1.69343 ms (enqueue 0.229059 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45561 ms - Host latency: 1.68185 ms (enqueue 0.224817 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41722 ms - Host latency: 1.66804 ms (enqueue 0.205859 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45674 ms - Host latency: 1.67717 ms (enqueue 0.229462 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41097 ms - Host latency: 1.6327 ms (enqueue 0.23385 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43483 ms - Host latency: 1.66712 ms (enqueue 0.230841 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41292 ms - Host latency: 1.63651 ms (enqueue 0.226581 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4472 ms - Host latency: 1.69167 ms (enqueue 0.224103 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43236 ms - Host latency: 1.65679 ms (enqueue 0.257037 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46176 ms - Host latency: 1.71451 ms (enqueue 0.246692 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.40933 ms - Host latency: 1.66502 ms (enqueue 0.230798 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46044 ms - Host latency: 1.75634 ms (enqueue 0.252576 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4295 ms - Host latency: 1.68127 ms (enqueue 0.227863 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45131 ms - Host latency: 1.72092 ms (enqueue 0.238 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44056 ms - Host latency: 1.73661 ms (enqueue 0.25249 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.50886 ms - Host latency: 1.76627 ms (enqueue 0.231195 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42786 ms - Host latency: 1.67579 ms (enqueue 0.238873 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.50211 ms - Host latency: 1.76423 ms (enqueue 0.234259 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43421 ms - Host latency: 1.7305 ms (enqueue 0.240234 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47025 ms - Host latency: 1.7165 ms (enqueue 0.211322 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.438 ms - Host latency: 1.71165 ms (enqueue 0.215259 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43637 ms - Host latency: 1.68851 ms (enqueue 0.21178 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47538 ms - Host latency: 1.73408 ms (enqueue 0.2099 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46023 ms - Host latency: 1.72546 ms (enqueue 0.225696 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45441 ms - Host latency: 1.69765 ms (enqueue 0.211273 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.50516 ms - Host latency: 1.74845 ms (enqueue 0.204962 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.53539 ms - Host latency: 1.76566 ms (enqueue 0.208777 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.52974 ms - Host latency: 1.77023 ms (enqueue 0.212103 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.51553 ms - Host latency: 1.75131 ms (enqueue 0.204193 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44434 ms - Host latency: 1.6765 ms (enqueue 0.197614 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.50579 ms - Host latency: 1.73585 ms (enqueue 0.204553 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43493 ms - Host latency: 1.67892 ms (enqueue 0.20506 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.49514 ms - Host latency: 1.73344 ms (enqueue 0.204608 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43637 ms - Host latency: 1.67058 ms (enqueue 0.200793 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43054 ms - Host latency: 1.66993 ms (enqueue 0.211938 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46187 ms - Host latency: 1.6971 ms (enqueue 0.212518 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42233 ms - Host latency: 1.66158 ms (enqueue 0.202283 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44076 ms - Host latency: 1.66971 ms (enqueue 0.207654 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.49064 ms - Host latency: 1.71844 ms (enqueue 0.209082 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.49701 ms - Host latency: 1.72854 ms (enqueue 0.215869 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47413 ms - Host latency: 1.71637 ms (enqueue 0.205359 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42601 ms - Host latency: 1.66443 ms (enqueue 0.200574 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45818 ms - Host latency: 1.69441 ms (enqueue 0.217151 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.52576 ms - Host latency: 1.7698 ms (enqueue 0.204956 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42552 ms - Host latency: 1.66198 ms (enqueue 0.202905 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44761 ms - Host latency: 1.68076 ms (enqueue 0.275586 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.40176 ms - Host latency: 1.63334 ms (enqueue 0.353162 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41483 ms - Host latency: 1.64297 ms (enqueue 0.342078 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41875 ms - Host latency: 1.65609 ms (enqueue 0.335022 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46533 ms - Host latency: 1.72169 ms (enqueue 0.337695 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41569 ms - Host latency: 1.64723 ms (enqueue 0.340759 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45184 ms - Host latency: 1.6881 ms (enqueue 0.234351 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43842 ms - Host latency: 1.67629 ms (enqueue 0.227026 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45093 ms - Host latency: 1.6912 ms (enqueue 0.229004 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41627 ms - Host latency: 1.65801 ms (enqueue 0.223169 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46155 ms - Host latency: 1.69381 ms (enqueue 0.209436 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46584 ms - Host latency: 1.72457 ms (enqueue 0.202954 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41681 ms - Host latency: 1.67416 ms (enqueue 0.201819 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42009 ms - Host latency: 1.6928 ms (enqueue 0.224731 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46842 ms - Host latency: 1.70946 ms (enqueue 0.203821 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43156 ms - Host latency: 1.68474 ms (enqueue 0.228918 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43718 ms - Host latency: 1.67003 ms (enqueue 0.223755 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45797 ms - Host latency: 1.69781 ms (enqueue 0.225891 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.50662 ms - Host latency: 1.76124 ms (enqueue 0.230042 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41598 ms - Host latency: 1.68702 ms (enqueue 0.232898 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.51992 ms - Host latency: 1.77449 ms (enqueue 0.233484 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42866 ms - Host latency: 1.68547 ms (enqueue 0.22428 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45399 ms - Host latency: 1.70485 ms (enqueue 0.227466 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41323 ms - Host latency: 1.65271 ms (enqueue 0.232861 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44648 ms - Host latency: 1.67838 ms (enqueue 0.22981 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4769 ms - Host latency: 1.70883 ms (enqueue 0.231348 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44999 ms - Host latency: 1.70062 ms (enqueue 0.229163 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43801 ms - Host latency: 1.66862 ms (enqueue 0.24325 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41241 ms - Host latency: 1.66367 ms (enqueue 0.25918 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42817 ms - Host latency: 1.67419 ms (enqueue 0.229297 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.40167 ms - Host latency: 1.64337 ms (enqueue 0.226721 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41241 ms - Host latency: 1.67089 ms (enqueue 0.240527 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47435 ms - Host latency: 1.71504 ms (enqueue 0.238367 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48033 ms - Host latency: 1.74031 ms (enqueue 0.23623 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43236 ms - Host latency: 1.7174 ms (enqueue 0.226868 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43311 ms - Host latency: 1.67919 ms (enqueue 0.2328 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42468 ms - Host latency: 1.65569 ms (enqueue 0.224109 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42971 ms - Host latency: 1.67214 ms (enqueue 0.231226 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41587 ms - Host latency: 1.65725 ms (enqueue 0.228186 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42211 ms - Host latency: 1.64583 ms (enqueue 0.227991 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45336 ms - Host latency: 1.67588 ms (enqueue 0.229199 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44774 ms - Host latency: 1.6796 ms (enqueue 0.234473 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45162 ms - Host latency: 1.67959 ms (enqueue 0.232837 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42479 ms - Host latency: 1.66847 ms (enqueue 0.229773 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48049 ms - Host latency: 1.7226 ms (enqueue 0.206653 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48623 ms - Host latency: 1.73438 ms (enqueue 0.216028 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42334 ms - Host latency: 1.65114 ms (enqueue 0.212744 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48696 ms - Host latency: 1.72549 ms (enqueue 0.238379 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48953 ms - Host latency: 1.71591 ms (enqueue 0.223865 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47732 ms - Host latency: 1.7085 ms (enqueue 0.228333 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48337 ms - Host latency: 1.72267 ms (enqueue 0.235474 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45295 ms - Host latency: 1.70245 ms (enqueue 0.229297 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47374 ms - Host latency: 1.71088 ms (enqueue 0.22959 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43656 ms - Host latency: 1.67587 ms (enqueue 0.226929 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43442 ms - Host latency: 1.67317 ms (enqueue 0.226465 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47334 ms - Host latency: 1.70566 ms (enqueue 0.237476 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4889 ms - Host latency: 1.72922 ms (enqueue 0.237085 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46453 ms - Host latency: 1.70959 ms (enqueue 0.230737 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46482 ms - Host latency: 1.69927 ms (enqueue 0.226514 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46973 ms - Host latency: 1.69486 ms (enqueue 0.218127 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47822 ms - Host latency: 1.70719 ms (enqueue 0.200159 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46453 ms - Host latency: 1.71158 ms (enqueue 0.198132 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46313 ms - Host latency: 1.69333 ms (enqueue 0.198474 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41458 ms - Host latency: 1.64109 ms (enqueue 0.211938 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46045 ms - Host latency: 1.69807 ms (enqueue 0.204785 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47168 ms - Host latency: 1.70068 ms (enqueue 0.203735 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47246 ms - Host latency: 1.73953 ms (enqueue 0.248926 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44326 ms - Host latency: 1.70273 ms (enqueue 0.206396 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45996 ms - Host latency: 1.71294 ms (enqueue 0.205347 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46843 ms - Host latency: 1.73701 ms (enqueue 0.207764 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46899 ms - Host latency: 1.74619 ms (enqueue 0.221899 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4105 ms - Host latency: 1.66462 ms (enqueue 0.220825 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48306 ms - Host latency: 1.7552 ms (enqueue 0.219653 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41814 ms - Host latency: 1.65547 ms (enqueue 0.206299 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46848 ms - Host latency: 1.69424 ms (enqueue 0.223999 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44392 ms - Host latency: 1.68259 ms (enqueue 0.218164 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45164 ms - Host latency: 1.69082 ms (enqueue 0.210913 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45239 ms - Host latency: 1.71494 ms (enqueue 0.213037 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44792 ms - Host latency: 1.70051 ms (enqueue 0.213721 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4426 ms - Host latency: 1.67534 ms (enqueue 0.210522 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47783 ms - Host latency: 1.73533 ms (enqueue 0.212866 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45046 ms - Host latency: 1.70015 ms (enqueue 0.208545 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46777 ms - Host latency: 1.72053 ms (enqueue 0.227173 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41003 ms - Host latency: 1.64714 ms (enqueue 0.229468 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4262 ms - Host latency: 1.65361 ms (enqueue 0.201831 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41812 ms - Host latency: 1.66455 ms (enqueue 0.198584 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42161 ms - Host latency: 1.65698 ms (enqueue 0.233081 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45549 ms - Host latency: 1.71313 ms (enqueue 0.238892 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45669 ms - Host latency: 1.6865 ms (enqueue 0.217822 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45662 ms - Host latency: 1.7054 ms (enqueue 0.25354 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43057 ms - Host latency: 1.66958 ms (enqueue 0.237769 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44207 ms - Host latency: 1.6843 ms (enqueue 0.241382 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.451 ms - Host latency: 1.68499 ms (enqueue 0.237183 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.422 ms - Host latency: 1.64675 ms (enqueue 0.227368 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42688 ms - Host latency: 1.65491 ms (enqueue 0.237671 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4606 ms - Host latency: 1.71331 ms (enqueue 0.239893 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46665 ms - Host latency: 1.69641 ms (enqueue 0.227197 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4251 ms - Host latency: 1.6854 ms (enqueue 0.238696 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.40596 ms - Host latency: 1.63267 ms (enqueue 0.220557 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44805 ms - Host latency: 1.70369 ms (enqueue 0.219727 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41694 ms - Host latency: 1.64959 ms (enqueue 0.218091 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45923 ms - Host latency: 1.69365 ms (enqueue 0.22605 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43401 ms - Host latency: 1.67422 ms (enqueue 0.229541 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44307 ms - Host latency: 1.68574 ms (enqueue 0.220728 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45703 ms - Host latency: 1.69219 ms (enqueue 0.226001 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.41169 ms - Host latency: 1.65454 ms (enqueue 0.227271 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46055 ms - Host latency: 1.69744 ms (enqueue 0.235034 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.39551 ms - Host latency: 1.62434 ms (enqueue 0.232739 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47437 ms - Host latency: 1.71738 ms (enqueue 0.241333 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42039 ms - Host latency: 1.65962 ms (enqueue 0.229004 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46206 ms - Host latency: 1.7144 ms (enqueue 0.246973 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45935 ms - Host latency: 1.69812 ms (enqueue 0.240552 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43228 ms - Host latency: 1.6675 ms (enqueue 0.227075 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44258 ms - Host latency: 1.66575 ms (enqueue 0.226562 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47998 ms - Host latency: 1.71125 ms (enqueue 0.246021 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47466 ms - Host latency: 1.70505 ms (enqueue 0.224609 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44136 ms - Host latency: 1.69644 ms (enqueue 0.245361 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4603 ms - Host latency: 1.6876 ms (enqueue 0.242505 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46079 ms - Host latency: 1.70676 ms (enqueue 0.228857 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48257 ms - Host latency: 1.72959 ms (enqueue 0.233252 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46606 ms - Host latency: 1.70703 ms (enqueue 0.22688 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4198 ms - Host latency: 1.66291 ms (enqueue 0.222192 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.46694 ms - Host latency: 1.69883 ms (enqueue 0.221436 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42937 ms - Host latency: 1.66909 ms (enqueue 0.226367 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45613 ms - Host latency: 1.70076 ms (enqueue 0.228687 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48335 ms - Host latency: 1.71118 ms (enqueue 0.228149 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45774 ms - Host latency: 1.6967 ms (enqueue 0.248242 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.45735 ms - Host latency: 1.68381 ms (enqueue 0.229297 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42322 ms - Host latency: 1.65239 ms (enqueue 0.234546 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43442 ms - Host latency: 1.66689 ms (enqueue 0.228833 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47031 ms - Host latency: 1.70459 ms (enqueue 0.237036 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48142 ms - Host latency: 1.71743 ms (enqueue 0.230396 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.44363 ms - Host latency: 1.69136 ms (enqueue 0.227905 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47961 ms - Host latency: 1.7188 ms (enqueue 0.254395 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.47251 ms - Host latency: 1.74216 ms (enqueue 0.230957 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.4677 ms - Host latency: 1.74314 ms (enqueue 0.256104 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.48225 ms - Host latency: 1.72986 ms (enqueue 0.23269 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.43811 ms - Host latency: 1.69395 ms (enqueue 0.2354 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42661 ms - Host latency: 1.67151 ms (enqueue 0.249683 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.51074 ms - Host latency: 1.77026 ms (enqueue 0.248462 ms)
[08/02/2023-11:07:48] [I] Average on 10 runs - GPU latency: 1.42212 ms - Host latency: 1.65181 ms (enqueue 0.230078 ms)
[08/02/2023-11:07:48] [I]
[08/02/2023-11:07:48] [I] === Performance summary ===
[08/02/2023-11:07:48] [I] Throughput: 687.71 qps
[08/02/2023-11:07:48] [I] Latency: min = 1.56436 ms, max = 2.35498 ms, mean = 1.69344 ms, median = 1.62057 ms, percentile(90%) = 1.9364 ms, percentile(95%) = 1.99805 ms, percentile(99%) = 2.12686 ms
[08/02/2023-11:07:48] [I] Enqueue Time: min = 0.189941 ms, max = 0.44104 ms, mean = 0.22892 ms, median = 0.226074 ms, percentile(90%) = 0.262085 ms, percentile(95%) = 0.300537 ms, percentile(99%) = 0.359375 ms
[08/02/2023-11:07:48] [I] H2D Latency: min = 0.0350342 ms, max = 0.107788 ms, mean = 0.0401407 ms, median = 0.038208 ms, percentile(90%) = 0.0458984 ms, percentile(95%) = 0.0546875 ms, percentile(99%) = 0.0691833 ms
[08/02/2023-11:07:48] [I] GPU Compute Time: min = 1.34143 ms, max = 1.96094 ms, mean = 1.45063 ms, median = 1.38354 ms, percentile(90%) = 1.69373 ms, percentile(95%) = 1.7561 ms, percentile(99%) = 1.86157 ms
[08/02/2023-11:07:48] [I] D2H Latency: min = 0.175781 ms, max = 0.429138 ms, mean = 0.202669 ms, median = 0.19165 ms, percentile(90%) = 0.240723 ms, percentile(95%) = 0.303955 ms, percentile(99%) = 0.35791 ms
[08/02/2023-11:07:48] [I] Total Host Walltime: 3.00418 s
[08/02/2023-11:07:48] [I] Total GPU Compute Time: 2.997 s
[08/02/2023-11:07:48] [W] * GPU compute time is unstable, with coefficient of variance = 9.24371%.
[08/02/2023-11:07:48] [W] If not already in use, locking GPU clock frequency or adding --useSpinWait may improve the stability.
[08/02/2023-11:07:48] [I] Explanations of the performance metrics are printed in the verbose logs.
[08/02/2023-11:07:48] [I]
&&&& PASSED TensorRT.trtexec [TensorRT v8503] # /usr/src/tensorrt/bin/trtexec --onnx=pallet_model_v1_all.onnx --minShapes=input:1x3x192x192 --maxShapes=input:1x3x1536x1536 --optShapes=input:1x3x256x256 --saveEngine=all_pallets_out/out.onnx --fp16