Follow-Up Interview: Principled Technologies on Intel Testing Concerns

After our exhaustive in-person interview with Principled Technologies, published on our YouTube channel, we followed-up via email to clarify some questions that were left unanswered during the initial video. As we noted in the initial interview, we give credit to Principled Technologies for endeavoring to sit down with us for these discussions. We recognize that it is not an easy decision to make – one of ignoring the problem (us showing up unannounced, in this instance) or confronting it – and we appreciate PT’s willingness to partake in a rational discussion about test methodology.

For full details of the interview, check the embedded video below. This written accompaniment aims to address follow-up questions where PT technicians closer to the testing had to be consulted. We are not going to transcribe the 40-minute interview and encourage that you watch the content to gain full perspective on both primary sides of the debate. We have timestamped key points in the video (timestamps are rendered within the video).

GPU Model Clarification

Regarding which specific GTX 1080 Ti was used, Principled Technologies emailed to let us know that it was the Gigabyte Aorus GTX 1080 Ti. The specific model number is GV-N108TAORUS-11GD.

Number of Test Benches

As for clarification on the number of test benches, we learned post-interview that Principled Technologies used a total of sixteen systems – eight core test platforms, then eight duplicates. We are still uncertain as to if PT then took the median of all six test passes or focused on only three passes from a single system.

PT’s Response to GN Claims of Unfair Cooler Arrangement

We pressed a question pertaining to thermals repeatedly in the interview, found at mark 11:43, and expressed concern over the choice to use aftermarket coolers only on the Intel DT parts, but not AMD. AMD’s Ryzen 7 2700X CPU was left with its stock cooler, whereas all competing Intel parts used the Noctua NH-U14S, which is one of the best 140mm coolers we’ve worked with. Our chief concern stemmed from multiple thermal constraints in stacking to potentially inhibit AMD’s CPU performance, including usage of the thermally obstructed front panel of the Thermaltake Suppressor case. Forcing air to follow multiple 90-degree turns (and thus lose pressure) before it meets a downdraft cooler is already inadvisable. To exaggerate this effect by then using objectively better coolers on the competing Intel parts is disingenuous at worst, or inexperienced at best. We asked why the AMD unit used a stock cooler (see 11:43 in the video) and received the start of an answer in the video, but got the rest via email:

“Only the 2700X came with a cooler. The rest required a cooler to be added. AMD certainly did not seem to think their cooler was inadequate, so it seemed a reasonable choice.”

Of course, our argument is two-fold: As an independent test outlet, AMD’s word shouldn’t be good enough for anything – and AMD might not know that the cooler is in a suffocated case with low static pressure. Further, all of this is irrelevant: If testing in a controlled environment, the single element which matters is equality between all test beds. If that’s an NH-U14S, so be it – AMD gets one, too. In this instance, for whatever reason, it seems the AMD platforms were relegated to objectively weaker stock coolers, whereas Intel’s lack of a stock cooler has anointed its CPUs with high-end air coolers. AMD is being irresponsibly punished by including a cooler in its packaging, whereas Intel is rewarded with its lack of cooler by receiving a better solution. We are still uncertain as to what was used for the Threadripper parts, as the NH-U14S is natively incompatible without using the significantly altered TR4 variant of the cooler.

Memory Selection and 64GB Questions

The next follow-up point pertained to memory, chiefly the question of why 64GB was used. As our audience is likely aware, opting for higher memory capacities can result in lowered sustainable frequency and loosened timings – particularly tertiary and secondary, which were (as far as we’re aware) left uncontrolled by PT. In the on-camera interview (mark 19:20) we asked why 64GB was chosen, receiving the answer that it seems “normal” for the configurations. At best, this is disjointed from the buyer of the 2700X or 8700K CPUs. To purchase the kit of 64GB used in this test would cost $620, or around 2x the cost of the processors. Very few are reasonably purchasing this much memory and, given that this benchmark focuses entirely on gaming tests (and not “production” tasks), we must look at it from a gaming scenario. No meaningful gaming build with a non-HEDT platform is opting for 64GB of memory. Anyway, following the interview, we received additional information pertaining to memory kit selection:

“We chose Corsair memory due to past experience with it being very reliable. We chose the specific model to be able to get matching two 2x16GB packages in the quantity necessary for this project […] We want to minimize disk access while providing a RAM amount that seemed appropriate for high-end processors. While that might be a bit much for some of the slower processors, we did need to keep the playing field level.”

A few things to note here: Corsair isn’t really the reason memory is reliable, although that is a flattering statement to the company. Realistically, it’s the IC provider (and the PCB to some extent, but less so), and that’d be SK Hynix, Micron, or Samsung. Second, disk access from games will rarely exceed the hundreds of megabytes, much less 32GB or beyond. We’re talking about texture files and meshes and assets, most of which inevitably get shoved into GPU memory, anyway. System memory rarely exceeds 8-12GB utilization in even the most abusive of games. If recommending a gaming PC build, even if it hosts an HEDT CPU (because gaming is the only thing tested in the PT document), we would push for 32GB of RAM for easier-to-manage timings and frequencies. You can still get four sticks, so quad-channel operation remains feasible, and 32GB is still more than enough for gaming benchmarks. It’s more than enough for most benchmarks.

At some point, leveling the playing field is not the right move, anyway. If testing HEDT CPUs for production tasks, we must consider the element of realism and what the users will do with those parts. HEDT CPUs make more sense for pairing with 64GB in video production and production tasks, for instance. It is sometimes better to populate a chart with fewer devices that are more closely related.

As for why memory was manually downclocked to 2666MHz (Intel) and 2933MHz (AMD), the post-interview answer changed to the following: “We wanted to make sure none of the processors were overclocking. We declocked the RAM from a maximum of 3000 to what the vendor recommended.” We are not aware of instances where the CPU will self-overclock as a result of higher frequency memory, but it is more likely that PT was referring to the memory clocks being officially over “stock” CPU support levels. GamersNexus does not consider XMP to be “overclocking” for a 3000MHz kit, so we will agree to disagree on this point. No one buys 2666MHz kits. They’re the same price as 3000MHz kits (these days), and it’s a simple toggle to jump into full rated speeds.

GTA Quality Settings

We separately asked about GTA V quality settings, as these were left undefined in the initial testing document. We also raised concern of GTA’s propensity to change settings between hardware changes, which PT acknowledged. This was their answer:

• GTA did have a strong tendency to try to reset its graphics settings, so we did have to carefully monitor that.
• Here is the output from the GTA benchmark text file:
• Display: 1920x1080 (FullScreen) @ 60Hz VSync OFF
• Tessellation: 0
• LodScale: 0.000000
• PedLodBias: 0.000000
• VehicleLodBias: 0.000000
• ShadowQuality: 1
• ReflectionQuality: 1
• ReflectionMSAA: 0
• SSAO: 0
• AnisotropicFiltering: 0
• MSAA: 0
• MSAAFragments: 0
• MSAAQuality: 0
• SamplingMode: 0
• TextureQuality: 2
• ParticleQuality: 0
• WaterQuality: 0
• GrassQuality: 0
• ShaderQuality: 0
• Shadow_SoftShadows: 0
• UltraShadows_Enabled: false
• Shadow_ParticleShadows: true
• Shadow_Distance: 1.000000
• Shadow_LongShadows: false
• Shadow_SplitZStart: 0.930000
• Shadow_SplitZEnd: 0.890000
• Shadow_aircraftExpWeight: 0.990000
• Shadow_DisableScreenSizeCheck: false
• Reflection_MipBlur: true
• FXAA_Enabled: false
• TXAA_Enabled: false
• Lighting_FogVolumes: true
• Shader_SSA: false
• DX_Version: 2
• CityDensity: 0.000000
• PedVarietyMultiplier: 1.000000
• VehicleVarietyMultiplier: 1.000000
• PostFX: 0
• DoF: false
• HdStreamingInFlight: false
• MaxLodScale: 0.000000
• MotionBlurStrength: 0.000000

Next Steps

Principled Technologies is actively working on a longer response to the most common concerns relating to its testing. We are uncertain as to when this will be published, but would expect its publication while the issue is still hot. PT Also alerted us that they are retesting most things, particularly highlighting that the company noticed greater “variability” with Game Mode on the 2700X. As Game Mode disables a CCX, it’d make sense that performance would be significantly handicapped on the AMD DT processor when running with half of its cores disabled.

For more discussion on this, view our embedded in-person interview above.

UPDATE: Principled Tech Publishes Comments

Principled Tech has published its own document commenting on current inquiries, found in full over here. We are rehosting it here for archival purposes. Key quotes are below:

"We have received a number of inquiries regarding the testing methodology we used and the potential for bias in favor of Intel. We are providing additional information to be as transparent as possible and to help allay these concerns.

"The following list summarizes many of the inquiries we have received and our responses. (We are continuing to work on addressing additional inquiries.)

• Use of "Game Mode" on the AMD Ryzen 7 2700X: Some inquiries we have received concern the use of the Ryzen utility and the number of active cores in the AMD-based systems. Based on AMD's recommendations and our initial testing on the Threadripper processors, we found installing the AMD Ryzen Master utility and enabling Game Mode increased most results. For consistency purposes, we did that for all AMD systems across Threadripper and Ryzen. We are now doing additional testing with the AMD systems in Creator Mode. We will update the report with new results.
• Cooler choice: We chose Noctua for the CPU coolers, due to having almost identical systems in the NH-U14S (Intel) and NH-U14S TR4-SP3 (AMD), which allowed us to maintain a comparable thermal profile. Because we were not performing any overclocking on any configuration, and because AMD has said it was a good cooler, we stuck with the stock AMD Ryzen 7 2700X Wraith Prism cooler.
• Memory speeds: To have complete parity across all systems, and to allow the Intel Core i9 X-series and AMD Ryzen Threadripper to fully utilize memory bandwidth, we used 4 16GB DDR4 DIMMs on all configurations.

[...]

"Because our goal is always to do the right thing and get the answers right, we are currently doing additional testing. We will share that data and will certainly call out if something is significantly different from what we've already published. 

"We are confident in our test methodology and results. We welcome questions and we are doing our best to respond to questions from our interim report, but doing so takes time. We will add responses if other issues come up. Thanks for listening."

Editorial: Steve Burke
Video: Andrew Coleman