Testing Methodology Explanations: 1% & 0.1% Lows, Delta T over Ambient

Following several YouTube commenter questions on some of our testing methodology and presentation, we decided to put together a short guide to FPS and temperature measurements. Specifically speaking to framerate and frametime testing, we've spent a few years refining our collection and presentation of “1% low” and “0.1% low” framerates, which is a converted presentation of frame output over time. These help us look into instances where a product might produce high averages, but exhibit stuttering and jarring gameplay that negatively impacts the “fluidity” of the experience.

An example, as we note in the “What are 1% & 0.1% lows?” video, would be the G3258 in GTA V versus something like the X4 760K. In this particular case, we saw the G3258 sustaining a higher average FPS than the X4 760K, but it was actually a worse product for the setup – that's because of the low values. The G3258 was getting hammered by such big, sudden dips in performance (a result of its limited thread count) that the 0.1% low output would sometimes hit 4FPS. In the real world, this means that players see stuttering and jarring gameplay.

Here's the video explaining all of this in greater depth:

And here's our previous interview with Scott Wasson, one of the founders of frametime testing:

These videos should serve as a means to further the understanding of framerate testing, and hopefully underscore the concept that average FPS is by no means a definitive metric for locating the “best” video card or CPU. The first video, the one which specifically speaks to 1% and 0.1% low values, shows off some gameplay in GTA V where some of these low stutter events can be seen by external video camera capture. That should help illustrate things. We've also got some animations and charts, as made by GN video editor Andrew Coleman.

Now, as for the temperature measurements, that's explained in the next video:

Here, we speak to the value of using active ambient measurement with logging utilities to create a “delta” value. This is how we represent almost all of our temperature data, unless otherwise listed. Subtracting the local ambient temperature from the diode measurements allows us to ensure that our testing remains comparative in a linear fashion across all seasons, HVAC changes, test environment changes, movement of the bench within the room, and other variables which may cause spiking or reduction of ambient (e.g.: multiple people in the test room would increase ambient). Because cooler and case makers are often fighting over tenths of a degree to a few degrees max, these differences matter and inaccurate testing can result in products landing on the stack where they don't belong.

In a previous article, we very heavily detail our testing methodology and prove its concept in a real thermal chamber. You can find that here.

As we're in the middle of GTX 1060 testing right now, I'll leave this content here for today. This article will hopefully serve as a landing point for future readers and viewers who are curious about what these “1% low” and “delta T” values mean, exactly, and we'll look into centralizing some more of our methodology in the future.

Editorial: Steve “Lelldorianx” Burke
Video: Andrew “ColossalCake” Coleman