No Man's Sky Frametime Performance Review: Not Even on a Titan XP

It looks like that 4K screenshot we were provided was created in some magical developer environment. No Man's Sky is the least stable game that we've worked on in years. Most games with poor optimization are still playable, and are consistent in their pitfalls; consistency permits some level of comparative benchmarking. With No Man's Sky, we're seeing nearly constant stutters and spectacular frame latency spikes in excess of 4000ms. The game exhibits severe stuttering that makes it unplayable at times and, by extension, impossible to accurately benchmark.

We've generated a few sets of benchmark data on a Titan X Pascal, 980 Ti, and RX 480 specifically to demonstrate just how wildly unpredictable the performance is. This is not an instance where we can just test anyway and produce charts as normal, because the FPS range is so wide that you'd end up with performance results that make no sense – like a GTX 1080 performing equally to a 980 Ti in averages in one test, but the opposite in another. That's due to variance introduced from somewhat unpredictable frame latency fluctuations, something we explain in part in this video.

Game-specific GPU benchmarks serve a single purpose: Hierarchically ranking the “best” products for each graphics configuration, resolution, and budget. The very heart of game benchmarking is to produce an objective comparative analysis between components. We have decided to present our findings with No Man's Sky, but have opted out of an immediate graphics card benchmark. This is because, in our eyes, such a benchmark would not be fair to the GPUs. The variance in results is so great that listings end up chaotic, and so we end up constrained and benchmarking the poor performance of No Man's Sky, rather than the performance of the cards themselves. The game is inadequate as a test platform, and cannot be trusted to generate reliable, replicable data from one test iteration to the next.

This GPU performance analysis of No Man's Sky looks at stutters and frame drops (“hitching”), poor optimization, screen flickering, and low FPS.

The “Experimental Branch” for No Man's Sky – PC Patch

No Man's Sky devs Hello Games pushed an experimental branch for the game, available for opt-in via Steam's settings. The new branch resolves some stutters and black flickering during the game's intro/loading splash screen (the map view), fixes some issues with the FPS cap not permitting framerates that match the cap, and improves some issues with mouse “smoothing” and alt-tabbing.

This build does not resolve the primary issue of stutters during gameplay and produced no acceptable performance improvements, but we will evaluate again once the build is ratified into the game's code.

Game Test Methodology

We tested using our GPU test bench, detailed in the table below. Our thanks to supporting hardware vendors for supplying some of the test components.

The AMD 16.8.2 drivers were used for testing. NVidia's 369.05 drivers were used for game (FPS) testing on the Titan X (Pascal). The 368.81 drivers were used for other devices. Game settings were manually controlled for the DUT. All games were run at settings defined in their respective charts. These screenshots show the main settings:

Resolution was varied per charts. Other settings were tested for internal reasons (to learn if anything specifically triggers issues), but the above represents what went to publication. We also tested with borderless, but saw no measurable improvement.

Windows 10-64 build 10586 was used for testing.

Each game was tested for 30 seconds in an identical scenario, then repeated three times for parity.

Average FPS, 1% low, and 0.1% low times are measured. We do not measure maximum or minimum FPS results as we consider these numbers to be pure outliers. Instead, we take an average of the lowest 1% of results (1% low) to show real-world, noticeable dips; we then take an average of the lowest 0.1% of results for severe spikes.

No Man's Sky Benchmark – Titan XP

We'll start high-end, work to a previous-gen architecture, then work back toward a new generation AMD architecture.

This first chart shows one test run with a Titan XP, with the game configured to its lowest settings. This first test pass of 1080p/low shows a logged 30-second gameplay period. If you look at our multiple test passes, these spikes are seen in almost all of them; you should expect to encounter one of these game-interrupting stutters at least once every thirty seconds.

In this instance, we've actually got the Titan XP stalling for a full 100ms before the next frame is output. When we discuss frametimes, we normally talk about deviation from the mean latency as being the most jarring – but this is far beyond usual deviation from a mean frametime. The result is an output of 138FPS AVG, but a 36FPS 0.1% low and a minimum of effectively 0FPS for that 100ms period. The average framerate is nearly 4x higher than the 0.1% low FPS.

This next one shows the second test pass for 1080p/low on a Titan X, where we go from an average 6ms frametime to 20ms in the worst case scenario – that's observable, but not nearly as bad as the previous (or next) tests. Any deviation exceeding an 8ms to 16ms swing will become more noticeable, but the numbers are at least more consistent in this second pass. Still, that 20ms spike is more noticeable than it may sound – especially since it's repeated a few times per minute.

Let's look at what happens if we configure settings that actually make sense for a Titan XP.

Here's Ultra, first with 1080p. Notice again that we've got a 100ms spike, dragging our 0.1% lows down to 31FPS.

At 4K, we're looking at an output that spikes to one thousand milliseconds, or one full second of frozen gameplay. This dragged our 0.1% lows down to 18FPS, though the minimum value was 0FPS. It gets worse, though – even at 1080p.

No Man's Sky Benchmarks – 980 Ti

Let's look at the 980 Ti, just to make sure it's not an issue with the Titan XP drivers or the Pascal architecture.

With the GTX 980 Ti configured to Low settings, we're seeing spikes hit 40ms in the thirty-second test period, resulting in a 2x deviation between the 0.1% lows and the average. The second set of data perfectly illustrates why we've decided against benchmarking this game right now, showing a dip in frame performance from the previous 122FPS down to 93FPS on the second test pass. Those two test passes were conducted within one minute of each other and in the same exact course; all our games that are included in our GPU reviews have a known variance of less than 1FPS, on average. This game has a variance of >30FPS for this one test, but we've seen a range upwards of 100FPS for some configurations. We see the 93FPS AVG and 3FPS 0.1% low are produced by the outburst of a 4200ms frame latency – and this is exactly why we use those low metrics, because a 93FPS AVG looks pretty damn good aside from the fact that the game was unplayable for 4 seconds of our 30 second test pass.

No Man's Sky Benchmarks – RX 480

To further diversify architectures, we looked at an MSI RX 480 Gaming X in addition to the two nVidia cards. With low settings at 1080p, we hit one snag of 1500ms, or 1.5 seconds, and another of 1100ms – and that was in the same test pass with a 380ms snag. That's nearly three seconds of frozen game time across a fifteen second range of our thirty-second test, or ten percent of the entire test pass. In the next pass, we saw better performance – but not much. Snags were upwards of 120ms and several clustered at 60ms, resulting in a 0.1% low of 19FPS.

Things weren't much better at Ultra, either, with a 3FPS 0.1% low in this test pass, resultant of an 1100ms frame time followed in sequence by a 500ms frame time, then a 300ms frame time.

The game is simply unplayable right now; at least, it is on our three test platforms. We've validated this performance behavior on two identical X99/i7-5930K test benches, and on one i7-4790K test platform. Rather than invest our time in attempting to create a benchmark for a game which clearly will not produce reliable, replicable results, and for a game which is clearly hardly playable on several hardware configurations, we will focus our efforts on other content until patches arrive. To create benchmarks today would go against the entire purpose of game-specific GPU ranking: To compare graphics cards hierarchically and create tiered performance ratings. Right now, performance is so variable and unpredictable that any such test would be neither realistic nor representative of actual comparative performance between cards. We'd mostly be looking at No Man's Sky's performance, which is what this piece serves to highlight.

We're aware that some folks -- some of you, we're sure -- have been able to get the game functional enough to enjoy and play. At this time, however, we do not consider NMS to have acceptable performance on the systems tested.

We'll keep you all posted on this title. Maybe wait on it, for now.

- Steve “Lelldorianx” Burke.