The question is about the whole loop
CUDA Graphs let a program replay a previously captured sequence of GPU operations with less repeated CPU launch work. That sounds like a straightforward inference optimization. In an engine that forms batches dynamically, however, the CPU and GPU do not operate independently: changing how quickly one side consumes work can change the batch sizes the other side produces.
I ran into this while developing a TensorRT execution and tuning path for KataGo. KataGo evaluates neural networks inside a search loop. Search produces positions, inference workers assemble them into batches, and the network’s outputs determine subsequent search. The rate of neural evaluations is therefore an outcome of a feedback loop, not just the cost of a fixed matrix multiplication.
The question I want an experiment to answer is: does the implementation increase useful evaluations per second under the same workload and numerical requirements? A kernel timing is evidence about one component of that question.
What the public benchmark shows
The table below reproduces the rounded two-worker results in the public PR. Both columns use the updated execution implementation; the comparison switches CUDA Graphs off or on. It is not a comparison between the entire old and new codebases.
The setup was an RTX PRO 6000 Blackwell, CUDA 13.0, TensorRT 10.16.1, a fixed 19×19 Go Transformer, FP16 inference, and the official engine benchmark with 1,600 visits over 20 positions. The graph capture cap was 16. The graph-enabled column is the mean of two runs. Full configuration details and the accompanying one-worker results are in the benchmark record.
| Search threads | Graphs off, evaluations/s | Graphs on, evaluations/s | Change, approximately |
|---|---|---|---|
| 16 | 1,133 | 2,273 | +101% |
| 32 | 2,255 | 2,951 | +31% |
| 64 | 3,179 | 3,286 | +3% |
| 128 | 3,074 | 3,125 | +2% |
One percentage would be a poor summary. At low concurrency, reducing launch overhead has much more room to help. At higher concurrency, the graph-free execution path already amortizes more of that cost. This is a plausible interpretation of the pattern, rather than a complete causal decomposition of every difference.
Batch formation matters too. At 32 search threads, the reported average batch changes from 10.36 to 9.43 while throughput increases. A larger average batch is not automatically the better operating point. Waiting for more work can improve GPU efficiency while increasing delay elsewhere in the search loop.
What I changed
The implementation combines graph replay with explicit ownership of execution resources. Workers on the same GPU can share an immutable compiled TensorRT engine, while each worker retains its own execution context, activation storage, buffers, stream, and graph cache. Sharing the compiled model is different from concurrently mutating one execution context.
I also used pinned host staging, placed input and output transfers on the worker’s stream, and synchronized after the output transfers rather than serializing each output independently. These changes address the path around inference as well as the model’s arithmetic. The public implementation and validation notes describe the boundaries and compatibility checks.
The important design constraint is that performance should not come from silently changing the numerical computation or weakening validation thresholds. The PR reports numerical checks with the existing tolerances. That evidence supports the tested configurations; it does not certify every TensorRT version, GPU, or model.
Tuning changes the experiment
TensorRT can choose different execution tactics for different shape profiles. Increasing a maximum batch size is therefore not simply extending the same curve to the right. In a local calibration, exact batch profiles 18 through 22 had different behavior, with a noticeable drop at 22.
I built a bounded tuner around that observation. It first samples the thread range, tests a small neighborhood of exact batch profiles, ranks configurations using a small thread stencil, and confirms the strongest candidates in alternating A–B–B–A order. Rates are pooled as total evaluations divided by total time, rather than averaging percentages.
On the reported machine, the tuner selected the same batch-size-20 region as a 70-case calibration grid using 24 measured cases. The final two-candidate confirmation reported 3,462.86 evaluations/s for its selected configuration. This is a result about reducing the number of tested configurations on this workload. It is not a proof of global optimality, and 24 versus 70 is not automatically the same reduction in elapsed tuning time.
The tuner makes its search boundary visible. Unexamined regions and nearby nearly tied configurations are useful information, not something to hide behind a single “optimal” setting.
What would make the conclusion stronger?
The current evidence has three limits. There are few repeats, the headline table comes from one hardware/software combination, and a search benchmark can follow slightly different trajectories even when the intended settings are matched. GPU power, temperature, and competing CPU work can also affect the result.
My next experiment would freeze the binaries and model, randomize configuration order within blocks, and repeat the comparison across multiple operating conditions. I would report startup time and memory alongside steady-state throughput. Graph capture and extra contexts consume resources; a deployment that performs little work after startup may prefer a different configuration.
I would also keep a fixed-batch microbenchmark beside the engine benchmark. If one improves and the other does not, the disagreement identifies where to look: dispatch, transfers, queueing, batch formation, or search itself. The microbenchmark is a diagnostic, while the engine-level objective determines whether the change is useful.
For me, the useful result is the operating region and an explanation of its limits. That is more transferable than a claim that CUDA Graphs make inference a fixed percentage faster.
Sources and status
- KataGo PR #1219: implementation, benchmark tables, resource measurements, and validation. Still a draft as checked on September 19, 2026.
- KataGo v1.17.2 release notes: separate merged TensorRT reliability fixes. These should not be confused with the draft graph path.
- NVIDIA TensorRT best practices: background on profiling and inference performance.
The numbers above come from the existing public benchmark, not a new experiment run for this article.