AI Developers eyeing gaming GPUs

M

Marees

2[H]4U
Joined
Sep 28, 2018
Messages
2,115

Here We Go Again: AI Developers Are Buying Gaming GPUs in Bulk​


https://www.extremetech.com/computing/here-we-go-again-ai-developers-are-buying-gaming-gpus-in-bulk

a software engineer has revealed a bulk purchase of AMD Radeon cards for AI training.

The revelation of a sizable GPU purchase appeared on the social network formerly known as Twitter this week by George Hotz, whose profile says they are the president of an AI company. The post says AMD is a real pleasure to work with when purchasing GPUs in bulk.

AMD is negotiating and selling directly to developers instead of going through a channel partner like Amazon or Newegg. That's an interesting ripple, as in the mining craze it was never known (to our recollection, at least) if AMD and Nvidia were selling bulk orders to GPU miners. We'll never know if that happened, but it seems like it's happening now on the AI front.



One interesting aside with the above purchase of AMD GPUs is that certain AI workloads can use a lot of memory. While paying nearly $1,000 per GPU for an AMD 7900 XTX that packs 'only' 122.88 teraFLOPS of FP16 number crunching prowess might seem odd, given the RTX 4060 costs $300 and can provide around 121 teraFLOPS of FP16 compute (with sparsity), the AMD GPU provides three times as much VRAM. So, if the workloads need memory, you'd need three RTX 4060 cards or two RTX 4070 cards to get the same 24GB — plus the servers to hold all those GPUs.

(This favors the 16gb version of the 4060 ti)

https://www.tomshardware.com/news/evidence-shows-ai-driven-companies-are-buying-up-gaming-gpus
 
It is excessively hard to use Nvidia consumer GPU’s for AI.
If your model fits in the ram provided on a single card you can do it well, if it does not the interfaces are crippled to an astonishing degree and the bandwidth tanks.
Nvidia designed much of the 4000 series knowing people would try and they made it hard from the ground up.

3 4060’s would perform far worse than a single 7900xtx because of all the cross talk required to make it work.

I’m guessing these companies are leftover crypto miners with large hardware assets they are trying to find jobs for. Lots of PSU’s, mother boards, racks, networking equipment, and such.
 
Lakados said:
It is excessively hard to use Nvidia consumer GPU’s for AI.
If your model fits in the ram provided on a single card you can do it well, if it does not the interfaces are crippled to an astonishing degree and the bandwidth tanks.
Nvidia designed much of the 4000 series knowing people would try and they made it hard from the ground up.

3 4060’s would perform far worse than a single 7900xtx because of all the cross talk required to make it work.

I’m guessing these companies are leftover crypto miners with large hardware assets they are trying to find jobs for. Lots of PSU’s, mother boards, racks, networking equipment, and such.
Click to expand...

The company specified in the article is not.
It's Geohot buying them, the guy that hacked the iphone and PS3 when he was like 5 years old.
He runs comma.ai which is a self driving AI company, so the cards could be for that. But he's also doing other AI stuff it could be for.

One of the things he's working on is basically a personal chat gpt machine. It would be a PC with as many GPUs a standard power outlet can handle. So everyone can run their own personal AI assistant in their own home. No need for the cloud, sharing resources/waiting longer for responses, having to worry about censorship and privacy issues, etc.
He's been trying to get AMD gpus working with things.

If that sort of thing catches on there will be a lot of GPU demand.
 
Lakados said:
It is excessively hard to use Nvidia consumer GPU’s for AI.
If your model fits in the ram provided on a single card you can do it well, if it does not the interfaces are crippled to an astonishing degree and the bandwidth tanks.
Nvidia designed much of the 4000 series knowing people would try and they made it hard from the ground up.

3 4060’s would perform far worse than a single 7900xtx because of all the cross talk required to make it work.

I’m guessing these companies are leftover crypto miners with large hardware assets they are trying to find jobs for. Lots of PSU’s, mother boards, racks, networking equipment, and such.
Click to expand...
I don't think AI even works on multiple cards without nvlink. Maybe some custom program, but if your workload requires 24 GB of VRAM two 16GB cards are not gonna do it.
 
M76 said:
I don't think AI even works on multiple cards without nvlink. Maybe some custom program, but if your workload requires 24 GB of VRAM two 16GB cards are not gonna do it.
Click to expand...
You can transfer the data over the PCIE bus, the Linux drivers for both AMD and Nvidia allow for it, but it is slow as you are dividing the PCIE lanes on each card accordingly based on the number of GPUs in the stack in even multiples basically 2, 4, 8, and it does its best to just coordinate all the data between them but it's pretty obvious that it doesn't scale well and it is very ram intensive. But you can do it.
Given his past experiences though I would assume they plan on hacking firmware and pinouts to add the links back in, or use the Ram as the repository for the dataset and find a way to schedule it out from there. It will be interesting all the same to see if they can find a way to feasibly make it worthwhile.
sharknice said:
The company specified in the article is not.
It's Geohot buying them, the guy that hacked the iphone and PS3 when he was like 5 years old.
He runs comma.ai which is a self driving AI company, so the cards could be for that. But he's also doing other AI stuff it could be for.

One of the things he's working on is basically a personal chat gpt machine. It would be a PC with as many GPUs a standard power outlet can handle. So everyone can run their own personal AI assistant in their own home. No need for the cloud, sharing resources/waiting longer for responses, having to worry about censorship and privacy issues, etc.
He's been trying to get AMD gpus working with things.

If that sort of thing catches on there will be a lot of GPU demand.
Click to expand...
It’s not Geohot, it’s his company that was trying to make self driving cars a thing that tried to sell to Tesla and failed, then got shuttered when the regulators said his tech didn’t comply with any of the existing laws. Then they branched out to crypto and crypto exchanges which they have been doing for a few years and now that it’s tanking here comes AI.

Lots of former crypto miners are trying to make the switch all trying to find a way to make it feasible.
 
Last edited:
sharknice said:
The company specified in the article is not.
It's Geohot buying them, the guy that hacked the iphone and PS3 when he was like 5 years old.
He runs comma.ai which is a self driving AI company, so the cards could be for that. But he's also doing other AI stuff it could be for.

One of the things he's working on is basically a personal chat gpt machine. It would be a PC with as many GPUs a standard power outlet can handle. So everyone can run their own personal AI assistant in their own home. No need for the cloud, sharing resources/waiting longer for responses, having to worry about censorship and privacy issues, etc.
He's been trying to get AMD gpus working with things.

If that sort of thing catches on there will be a lot of GPU demand.
Click to expand...

He's also - admittedly - forcing himself/his company to use AMD GPUs vs Nvidia and trying to help AMD with feedback on all the ways he even admits AMD sucks at AI ATM - he's been doing the podcast rounds about this for a bit
 
staknhalo said:
He's also - admittedly - forcing himself/his company to use AMD GPUs vs Nvidia and trying to help AMD with feedback on all the ways he even admits AMD sucks at AI ATM - he's been doing the podcast rounds about this for a bit
Click to expand...
AMD consumer RDNA does suck for the existing AI models, if he can make a new model it could work, but the RDNA cards aren't gimped like the Nvidia ones so they are more feasible as AMD hasn't actively worked against trying to do this, unlike Nvidia who has designed the 4000 series from the ground up to be incredibly painful to use in workstation and enterprise roles.
Advancements to RocM could make it doable but I doubt the time and resource investment will balance out, but he needs to find something to do with his infrastructure.
 
Update:

AMD to open source Micro Engine Scheduler (MES) firmware for Radeon GPUs​


https://www.theregister.com/2024/04/05/amd_mes_open_source/

yesterday AMD divulged more details, specifying that one of the things it would be making open source was the MES firmware for Radeon GPUs. AMD says it will be publishing documentation for MES around the end of May, and will then release the source code some time afterward.

For one George Hotz and his startup, Tiny Corp, this is great news. Throughout March, Hotz had agitated for AMD to make MES open source in order to fix issues he was experiencing with his RX 7900 XTX-powered AI server box. He had talked several times to AMD representatives, and even the company's CEO, Lisa Su.

While Hotz was initially "70 percent confident" that MES would be made open source, after his conversation with Su in late March he figured it wasn't going to happen after all. This would have killed the AMD-powered TinyBox, which uses six RX 7900 XTXs, but the project was saved thanks to the discovery of a single "umr" repository that made the box work well enough.

Still, Tiny Corp had reluctantly decided to launch an RTX 4090-powered TinyBox anyway since it offered a solution that "just works." An Intel version might also be in the pipeline, though the performance of Intel's best GPU, the Arc A770, would be a significant concern since it comes nowhere near either the RX 7900 XTX and RTX 4090.


Hotz has not yet responded to AMD's announcement, and we've asked him and AMD to comment.
 
Marees said:
Update:

AMD to open source Micro Engine Scheduler (MES) firmware for Radeon GPUs​


https://www.theregister.com/2024/04/05/amd_mes_open_source/

yesterday AMD divulged more details, specifying that one of the things it would be making open source was the MES firmware for Radeon GPUs. AMD says it will be publishing documentation for MES around the end of May, and will then release the source code some time afterward.

For one George Hotz and his startup, Tiny Corp, this is great news. Throughout March, Hotz had agitated for AMD to make MES open source in order to fix issues he was experiencing with his RX 7900 XTX-powered AI server box. He had talked several times to AMD representatives, and even the company's CEO, Lisa Su.

While Hotz was initially "70 percent confident" that MES would be made open source, after his conversation with Su in late March he figured it wasn't going to happen after all. This would have killed the AMD-powered TinyBox, which uses six RX 7900 XTXs, but the project was saved thanks to the discovery of a single "umr" repository that made the box work well enough.

Still, Tiny Corp had reluctantly decided to launch an RTX 4090-powered TinyBox anyway since it offered a solution that "just works." An Intel version might also be in the pipeline, though the performance of Intel's best GPU, the Arc A770, would be a significant concern since it comes nowhere near either the RX 7900 XTX and RTX 4090.


Hotz has not yet responded to AMD's announcement, and we've asked him and AMD to comment.
Click to expand...
But this line here…
"If you like to tinker and feel pain, buy red. The driver still crashes the GPU and hangs sometimes, but we can work together to improve it," says Tiny Corp.
 
Does any of this really matter? You know in the end that us ordinary guys are going to get the shaft again, the same one we got during the cryptomining craze.

ETTS.
 
That a expensive but quite normal single new car type of purchase.... Did not know how it work, XFX make cards, AMD buy them and resales them to small buyer like that... ? or they just helped them with the contact of a reseller they know ?
 
Marees said:
Update:

AMD to open source Micro Engine Scheduler (MES) firmware for Radeon GPUs​


https://www.theregister.com/2024/04/05/amd_mes_open_source/

yesterday AMD divulged more details, specifying that one of the things it would be making open source was the MES firmware for Radeon GPUs. AMD says it will be publishing documentation for MES around the end of May, and will then release the source code some time afterward.

For one George Hotz and his startup, Tiny Corp, this is great news. Throughout March, Hotz had agitated for AMD to make MES open source in order to fix issues he was experiencing with his RX 7900 XTX-powered AI server box. He had talked several times to AMD representatives, and even the company's CEO, Lisa Su.

While Hotz was initially "70 percent confident" that MES would be made open source, after his conversation with Su in late March he figured it wasn't going to happen after all. This would have killed the AMD-powered TinyBox, which uses six RX 7900 XTXs, but the project was saved thanks to the discovery of a single "umr" repository that made the box work well enough.

Still, Tiny Corp had reluctantly decided to launch an RTX 4090-powered TinyBox anyway since it offered a solution that "just works." An Intel version might also be in the pipeline, though the performance of Intel's best GPU, the Arc A770, would be a significant concern since it comes nowhere near either the RX 7900 XTX and RTX 4090.


Hotz has not yet responded to AMD's announcement, and we've asked him and AMD to comment.
Click to expand...
The open source part doesn't seem to matter since AMD often can't deliver on the software side - whether it's video editing, GPUCompute (e.g. Blender) or AI/ML stuff - there's often complaints and reports of problems - and it's often stability (drivers, for e.g.) and after that - if something does work - then performance in comparison to Nvidia (usually).

The other thing I notice - is that it takes ages for AMD to get this stuff (their software stack and related technologies - for e.g., ROCm/HIP-RT etc.) working. This is just my impression and conclusions from various reports I've read. If you have an AMD gpu or read something else that doesn't align with this, feel free to explain. :) But, it's what I perceive.
 
pavel said:
The open source part doesn't seem to matter since AMD often can't deliver on the software side - whether it's video editing, GPUCompute (e.g. Blender) or AI/ML stuff - there's often complaints and reports of problems - and it's often stability (drivers, for e.g.) and after that - if something does work - then performance in comparison to Nvidia (usually).

The other thing I notice - is that it takes ages for AMD to get this stuff (their software stack and related technologies - for e.g., ROCm/HIP-RT etc.) working. This is just my impression and conclusions from various reports I've read. If you have an AMD gpu or read something else that doesn't align with this, feel free to explain. :) But, it's what I perceive.
Click to expand...
AMD probably can't staff up with as many software developers as nVidia. Classic situation with a #2.
 
philb2 said:
AMD probably can't staff up with as many software developers as nVidia. Classic situation with a #2.
Click to expand...
The problem with calling them #2 in this situation is the gap between them. It’s not a 60/40 split, it’s closer to 96/04. Nvidia spends more on drivers and development than AMD will make in revenue over the year. AMD wears a lot of hats and they are spread somewhat thin, they have to be very careful in choosing their battles.
 
This is good news. We’ve had some brainstorming sessions at work lately about how we could manufacture in the future using A.I. generated pictures of dogs wearing hats while playing basketball. We have some solid leads so far.
 
Maybe I should hold onto my 3090 even though I replaced it. It might be worth a hell of a lot more in a month or two.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
DukenukemX said:
The 7900 XTX is about $900. That's the lowest price I've seen them for. The RTX 4060 16GB is $450 which hasn't changed. If there's a sizable interest in purchasing retail GPU's for AI then I'm not seeing it. Usually the insane rush to purchase GPU's would come long before a Tom's Hardware article writes about it.
Click to expand...
And even then if their unit still clocks in at $15,000 and requires infrastructure and other things to be made useful and then requires a hell of a lot of tinkering to keep working in a usable manner what is the actual savings over the proper enterprise based solution.

This whole project just sounds like an attempt to find a use for the surplus hardware the former crypto miners haven’t been able to unload.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
Lakados said:
This whole project just sounds like an attempt to find a use for the surplus hardware the former crypto miners haven’t been able to unload.
Click to expand...
The articles above does not talk about the project at all, is it not to use themselve for training their car driving-gps technology ?

If it is about the TinyBox, it is aiming to be around $15000 yes (25k for 4090s), the goal of the project seem to make a petaflop of compute available somewhat easily and for people that need compute to find way to use it,
 
Lakados said:
And even then if their unit still clocks in at $15,000 and requires infrastructure and other things to be made useful and then requires a hell of a lot of tinkering to keep working in a usable manner what is the actual savings over the proper enterprise based solution.

This whole project just sounds like an attempt to find a use for the surplus hardware the former crypto miners haven’t been able to unload.
Click to expand...
If you're talking about the tiny box the whole point of it is it doesn't require any special infrastructure. It uses the max power a 15 amp outlet can provide, basically like a space heater. You just plug it in and you're good.

The point of it is you have AI services in your home, no reliance on the cloud, private, no censorship, etc.
 
sharknice said:
If you're talking about the tiny box the whole point of it is it doesn't require any special infrastructure. It uses the max power a 15 amp outlet can provide, basically like a space heater. You just plug it in and you're good.

The point of it is you have AI services in your home, no reliance on the cloud, private, no censorship, etc.
Click to expand...
At that price and with the current AI software offering, I imagine it is more for startup that need a lot of petaflop and calculate that they workload would be cheaper with this than azure or other credits (or need some low latency)
 
Comixbooks said:
I wish they would just hire Tom Cruise
Click to expand...
1712610077590.png



Maybe I'll get into botting and start selling GPUS to everyone in here at MSRP.
 
erek said:
isn't this similar to the Crypto Craze on GPUs
Click to expand...
Much harder to make money with a gaming GPU for the average person than just launching NiceHash, thus why it is still easy to buy them, many of the AI workloand have an hard time using completely decentralized gpu with a very small data versus compute that can be run completely independently like cryptos, that why we cannot easily rent our GPU to a cloud service provider doing AI training, something like that would be needed for the cryptoGPU rush like affair to start again.
 
Last edited:
You must log in or register to reply here.
Back
Top