Demand for GPUs is unquestionably ramping up again as prices eclipse $70,000 per GPU in some China locales (that's for a data center H800 GPU), and leaders in the U.S. computing industry are taking to social media to complain that cloud-based GPU resources are fully booked and GPU hardware supplies are all reserved for the year ahead. Drastic times call for drastic measures, and we are beginning to see evidence that GPUs that should be heading to home desktop PC rigs are instead being snapped up by the AI industry players.
Naturally, gamers and enthusiasts will be worried about a repeat of the cryptomining craze, which decimated consumer GPU supplies: Are the crypto-bros of old destined to be replaced by the AI-bros — snapping up our precious gaming GPUs?
The first solid evidence of AI-focused businesses buying up consumer GPUs comes from a boast Tweeted by iconic IT hacker and entrepreneur George Hotz (AKA geohot).
AMD (unlike Qualcomm) is a joy to buy chips from. 7.38 PFLOPS of compute was delivered today, with exaflops more to come.Nice to see them more active on the ROCm GitHub, we are working on the layer above @__tinygrad__ pic.twitter.com/5yE11TVA5FAugust 1, 2023
Above, if you expand the tweet, you can see Hotz, who is currently leading an automated AI-driven driving assistance business called Comma AI, talk about buying up cases of AMD gaming GPUs.
The image shows a partly unpacked purported "7.38 PFLOPS of compute." We can clearly spy an XFX MERC 310 GPU in the photo, and the box reads "AMD Radeon RX 7900 XTX." That's an XFX design based around AMD's flagship consumer GPU and the second-placed Best Graphics Cards for Gaming in 2023 in our frequently updated roundup.
Our calculations from the Hotz statement and image are that there are 60 GPUs in this "7.38 PFLOPS" batch (based on FP16 performance). Each of these RX 7900 XTX cards costs $979.99 on Amazon, at the time of writing. Thus, Hotz apparently just splashed out around $60,000 on this modest stack of GPU power.
Obviously this is a mere drop in the ocean, as the Comma AI founder and president says that there are "exaFLOPS more to come." For reference, one exaFLOPS is 1,000 petaFLOPS of performance. If we assume that Hotz aims to buy up to 7.38 exaFLOPS of GPU compute (1,000 x the first batch), he may have budgeted ~$60 million for this consumer graphics card buying spree. Or perhaps only ~$20 million, if he's only after around 2 exaFLOPS of AI compute.
One interesting aside with the above purchase of AMD GPUs is that certain AI workloads can use a lot of memory. While paying nearly $1,000 per GPU for an AMD 7900 XTX that packs 'only' 122.88 teraFLOPS of FP16 number crunching prowess might seem odd, given the RTX 4060 costs $300 and can provide around 121 teraFLOPS of FP16 compute (with sparsity), the AMD GPU provides three times as much VRAM. So, if the workloads need memory, you'd need three RTX 4060 cards or two RTX 4070 cards to get the same 24GB — plus the servers to hold all those GPUs.
The Tweeted statement from George Hotz isn't the first warning sign we have seen of AI-industry demand for GPUs heating up. A few days ago, we reported on the rapid inflation of GPUs in China. Fears of further US sanctions on China are thought to have helped stoke up prices of accelerators like the A800 and H800 GPUs. A few weeks earlier, the price of an A800 jumped 20% almost overnight due to impending sanction fears.
Given other recent indicators, we can clearly see that the rise of AI is having a profound impact on the supply of compute-focused GPUs. Now the only question is how far that insatiable demand will bleed over to gaming GPUs, too.
