Initial Vulkan Benchmark vs. DirectX 11 - AMD & NVidia in Talos Principle

NVidia and AMD had a bit of a back-and-forth with day-one Vulkan announcements, with nVidia taking a few shots at AMD's beta driver launch. “OpenGL Next” became Vulkan, which consumed parts of AMD's Mantle API in its move toward accommodating developers with lower-level access to hardware. The phrase “closer to the metal” applies to Mantle, Vulkan, and DirectX 12 in similar capacities; these APIs bypass overhead created by DirectX 11 and more directly tune for GPU hardware, off-loading parallelized tasks from the CPU and to the GPU. In a previous interview with Star Citizen's Chris Roberts, we talked about some of the developer side of Vulkan & DirectX 12 programming, learning that it's not as easy as just 'throwing an API call' switch.

For this benchmark, we ran Vulkan vs. DirectX 11 (D3D11) benchmarks in the Talos Principle to determine which API is presently 'best.' There's a giant disclaimer here, though, and we've dedicated an entire section of the article to that (see: “Read This First!”). Testing used an R9 390X, GTX 980 Ti, and i7-5930K; we hope to add low-end CPUs to determine the true advantage of low-level APIs, but are waiting until the driver set and software further iterate on Vulkan integration.

Read This First!

The Talos Principle has not fully optimized its Vulkan integration. The inclusion of the API is presently in early stages. As the Talos Principle team has indicated, Dx9 performance used to outrank Dx11 performance within the game, but later optimizations resolved and switched that performance. The team expects the same to happen with Vulkan, noting in its FAQ:

“Q: Then I can expect some great benchmark results?

A: Not so fast! Sorry. :( This is a beta phase, and there's still work to be done from our (Croteam) side and IHVs from driver side. But please be patient: in time, after optimizations are done, Vulkan will be faster, I'm sure.”

Our benchmarks then will serve as an interesting means to plot the historic progression of the Vulkan API's performance. As of now, performance will be lower than DirectX 11 – but updates to drivers and the game should eventually force a switch-over, something we're all too excited to plot as IHVs and ISVs roll-out those updates.

The results below are from limited scenario runs where we tested only two hardware configurations, both running the 5930K CPU. As Vulkan receives its updates from Croteam and the IHVs, we'll begin benchmarking lower-end CPUs (Core i3, for instance) to determine the impact from off-loaded draw calls. DirectX 12 and Vulkan should both aid low-end and mid-range CPUs more directly than the GPU, in most scenarios, by sending draw call processing to the parallel processors on the GPU.

For now, we've got an initial look at Vulkan benchmark / FPS performance using The Talos Principle's benchmark tool.

Test Methodology

We tested using our X99 test bench, detailed in the table below. Our thanks to supporting hardware vendors for supplying some of the test components.

The latest AMD drivers (16.15.1009 Beta 4) were used for testing. NVidia's 356.39 Vulkan beta drivers were used for testing. Game settings were manually controlled for the DUT. All games were run at presets defined in their respective charts. We disable brand-supported technologies in games, like The Witcher 3's HairWorks and HBAO. The Talos Principle is presently the only Vulkan-supported game; we used the built-in benchmark for automated test execution over multiple repeated 30-second test passes. This allows others to replicate our results by studying our bench courses.

Our usual test recording software does not work with Vulkan. For Vulkan testing, we used the built-in FPS logging utilities which report framerates somewhat differently, so rather than the usual “1% LOW” and “0.1% LOW” read-outs, we just have “lows,” which are somewhat ambiguously defined by the built-in benchmark. Lows will represent an average of the worst frame outputs over the course of benchmarking.

Vulkan vs. DirectX 11 Benchmark – 1080p, Ultra

Two charts are below – one for the reference GTX 980 Ti, one for the MSI R9 390X ($410).

At 1080p, Ultra, we observed a performance disparity of 57.89% for the R9 390X, favoring D3D11. That's a significant slow-down on Vulkan, but one which Croteam feels will eventually close and switch as optimizations occur. 75.5FPS for Vulkan and 113.3FPS (averaged over three passes) for DirectX 11 amount to our averages, with the low performance at 41.6FPS vs. 88.1FPS.

For the 980 Ti, performance was marginally closer with a 44.47% disparity, again favoring DirectX 11. The GTX 980 Ti at 1080, Ultra pushed 90.7FPS averages with a 57.7FPS low on Vulkan, or 127.2FPS average and 98.3FPS low on D3D11.

Between the two cards, there's absolutely no reason to run Vulkan on Windows right now – and we'd imagine that scales downward across all configurations, but need to perform further testing for validation. It's the low-end CPUs that should be most interesting once these drivers and software patches get pushed.

Vulkan vs. DirectX 11 FPS Benchmark – 4K Ultra

The next two charts use the same configurations, but run a 4K resolution with a max MP/s throughput of 8.7.

AMD's R9 390X shows a performance disparity of 47.8% between Vulkan and D3D11, favoring Microsoft's established API. The low performance is fairly tightly timed with the averages, at 22.1FPS (low) / 29.9FPS (AVG) vs. 39.2FPS (low) / 48.7FPS (AVG).

NVidia fronts a 31.67% performance disparity between Vulkan and D3D11, favoring DirectX 11. That's a significant lead over AMD in FPS deltas at 4K; we're curious to see whether nVidia can maintain its present delta advantage (between Vulkan and D3D11, not between the 980 Ti and 390X) as the API matures.

Conclusion: Vulkan's Getting There

Vulkan, or “OpenGL Next,” is still young. It's not ready yet. For Windows users, there's currently no reason to play The Talos Principle using the Vulkan API unless the user is a tester or developer of some kind. Vulkan theoretically opens some doors on other operating systems, like Linux, and we may begin investigating a Linux testing platform in the near future.

For now, it looks like it's a game of “wait and see.” Croteam seems to feel confident that Vulkan will overtake D3D11 performance as updates roll-out – and that'd make sense: We're not talking DirectX 12, here, we're talking DirectX 11, an aged, more restrictive API.

We'll follow-up with more benchmarks as the updates roll-out. It should be interesting to see these >40% gaps close, in the cases where they exist.

- Steve “Lelldorianx” Burke.