As everyone begins running the Final Fantasy XV PC benchmark, we’d like to notify the userbase that, on our test platform, we have observed some run-to-run variance in frame-to-frame intervals from one pass to the next. This seems to stem entirely from the first pass of the benchmark, where the game is likely still loading all of the assets into memory. After the first pass, we’ve routinely observed improved performance on runs two, three, and onward. We attribute this to first-time launcher initialization of all the game assets.
This, of course, raises a philosophical question over how to count the data: If Final Fantasy’s own benchmark charts contain hundreds or thousands of records from users, none of whom control the test environment and many of whom run the test once, is that data valid? There could be risk of poisoning the data pool, although it is still objectively useful on the whole. Granted, as always with scores, you’re not really getting the whole story. What really needs to be analyzed is the frametime performance, particularly run-to-run. The philosophical question boils down to whether we “count” the first run, or we discard it for all devices tested. We typically discard an outlier result, but if it is consistent and reproducible, it’s not much of an outlier. This is something we will debate prior to publication of our final benchmarks.
Please note that we've only just started, and we still need to validate this behavior on more hardware. As it stands now, this is what we're seeing. Other configurations may behave differently. This test is not fully exhaustive (yet).
In the meantime, here are some frametime graphs. This is ad-hoc testing on an 8700K at 5GHz, 1.42V, with 3200MHz CL14 GSkill Trident Z Memory. We are using a GTX 1070 Gaming X card for this test, which is running on the latest drivers. Windows version 1703 is being used. The blue bar demonstrates performance variability during the first pass, something which we must reconcile as we continue to run benchmarks. The frametime spikes are felt in the form of long, noticeable stutters in gameplay.
Be sure to follow our testing as we iterate. We are working all day on FFXV PC benchmarks.
Editorial, Testing: Steve Burke