$100 Nvidia CUDA card for A.I. ?

A

AirCool2

n00b
Joined
Oct 24, 2020
Messages
34
I am not a gamer. I have a decent Core-i7 desktop with 16GB RAM and PCIe-x16 slot. I want to play with A.I. but met the hard truth that my Intel UHD IGP desktop is just not up to the task. I mean calling it slow is an vastly understatement.

Are there CUDA compatible video card that I can buy for $100 (used is okay) so I can play around with some A.I. without spending days waiting?
 
NVidia Tesla P40 24 GB just went through ebay for $150. That's a good card.
 
It seems to me that if you wanted to play with AI, that you would be looking for a card with Tensor Cores. Any card going all the way back to the 8800 GTX can do CUDA... but only RTX cards have Tensor Cores. The P40 mentioned above would do CUDA just fine, but being Pascal based (same as GTX 1080, etc), it would not have any Tensor Cores. Prices on first-generation RTX cards (Turing, aka 2080, etc) have come way down. So your best bet might be to get something like an RTX 2070 or 2060, which are both often selling for below $200 now.
 
Zotac type brand used 1660 super seem to get that low from time to time, 2060 does not seem to go that low so it is probably around 1660 super/1070.
 
LukeTbk said:
Tensor would be nice but seem out of the price range by a lot, and what nice with CUDA is that like you said it will run on everything.

Really depends on what the OP is trying to do, a 1660 super/1070 still has 10 tflop of FP16...

this can give some idea of relative performance:
https://i0.wp.com/timdettmers.com/wp-content/uploads/2023/01/GPUS_Ada_raw_performance3.png?resize=1024,1006&ssl=1

Are you doing training or inference only ?
Click to expand...
1660S/2060 have way too little VRAM to do anything meaningful. @OP the price of entry is $250, save up for a 3060 12GB or better yet, a 4060 TI 16 GB.
 
Meaningful, what dataset/model people are trying to create ?

If we talk running them, 3b weight you have Lama, if you are an amateur trying to learn, could take a while for 6GB of vram to be an issue.

You can even run Lama 13 on a 2060 with a bit of tricks:

View: https://gist.github.com/rain-1/8cc12b4b334052a21af8029aa9c4fafc


Many text to voice would fit no problem, depend what one mean by want to play
 
LukeTbk said:
Tensor would be nice but seem out of the price range by a lot, and what nice with CUDA is that like you said it will run on everything.

Really depends on what the OP is trying to do, a 1660 super/1070 still has 10 tflop of FP16...

this can give some idea of relative performance:
https://i0.wp.com/timdettmers.com/wp-content/uploads/2023/01/GPUS_Ada_raw_performance3.png?resize=1024,1006&ssl=1

Are you doing training or inference only ?
Click to expand...
I don't know much about AI at all. I would say a little bit of training and more on inference.

Prior to posting this thread, I did some research and think that RTX 2070 is an entry level choice. For some reasons, used ones seems to be cheaper than used 2060 or around the same price. My plan is to keep an eye used or discounted 2070. But if there is something for $100, I can buy one now and start playing. How much of an impact if there is no Tensor Core?
 
AirCool2 said:
How much of an impact if there is no Tensor Core?
Click to expand...
Depends on how much the model is made to take advantage (like FP-16), a 2060 can beat a 1080ti in some ML stuff because of them and ML tend to do so:
https://towardsdatascience.com/rtx-...t-rtx-vs-most-expensive-gtx-card-cd47cd9931d2

Has you see it is not 2 different world, with or without:
1*zI4W0BXFZ7OHnO7ch0FC_Q.png


Nvidia also made a 12GB vram 2060 super, if you go turing and up you avoid a list of possible issue in general too
 
LukeTbk said:
Meaningful, what dataset/model people are trying to create ?

If we talk running them, 3b weight you have Lama, if you are an amateur trying to learn, could take a while for 6GB of vram to be an issue.

You can even run Lama 13 on a 2060 with a bit of tricks:

View: https://gist.github.com/rain-1/8cc12b4b334052a21af8029aa9c4fafc


Many text to voice would fit no problem, depend what one mean by want to play
Click to expand...

yes, but at what cost? at 3 tokens per second you might as well use a CPU.
also, cifar-10 in resnet50 is not representative of AI workloads in 2024. the weights and activations are so small that it becomes difficult to achieve high arithmetic intensity, which is why you only see a 2x speedup with tensor cores. for comparison, V100 is something like 5x faster than P100 in Stable Diffusion 1.5, which is a huge and mostly convolutional network at its core.
my real point is the OP's question is similar to "I want to play cyberpunk 2077, but I only have a $100 budget". yes, you can make it run, but the experience will be a pale imitation of actually playing the game on its target hardware and really the answer is to save up and buy a more expensive GPU.
 
A.I. experience is vastly different from gaming though. I would imagine an fps drop of 30 to 10 would be very noticeable. But all I want is cutting a wait time of 20+ hours to something manageable. AFAIK, there is virtual no hardware acceleration with my Intel IGP.
 
Would help if you would mention what you are trying to do (for some stuff AMD work well), training vs inference, etc...
 
AirCool2 said:
I am not a gamer. I have a decent Core-i7 desktop with 16GB RAM and PCIe-x16 slot. I want to play with A.I. but met the hard truth that my Intel UHD IGP desktop is just not up to the task. I mean calling it slow is an vastly understatement.

Are there CUDA compatible video card that I can buy for $100 (used is okay) so I can play around with some A.I. without spending days waiting?
Click to expand...
https://pytorch.org/get-started/previous-versions/

I would recommend pytorch and you can see they have older binaries, but those lack support.

# CUDA 11.8
conda install pytorch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 pytorch-cuda=11.8 -c pytorch -c nvidia
# CUDA 12.1
conda install pytorch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 pytorch-cuda=12.1 -c pytorch -c nvidia
# CPU Only
conda install pytorch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 cpuonly -c pytorch

The lowest I would go with is CUDA 11.8. I think this support goes all the way back to CUDA Compute of 3.5, which is pretty old.
https://en.wikipedia.org/wiki/CUDA

So, as you can see, Late Kepler and newer cards should be good to go. So I would say anything later than 2012 would work with pytorch. The next highest priority for me would be total vram. More is obviously better.
I would strongly consider the k80.
https://www.techpowerup.com/gpu-specs/tesla-k80.c2616
https://www.newegg.com/p/1DW-000D-00268, and it is less than 50 bucks.
 
bwang said:
yes, but at what cost? at 3 tokens per second you might as well use a CPU.
also, cifar-10 in resnet50 is not representative of AI workloads in 2024. the weights and activations are so small that it becomes difficult to achieve high arithmetic intensity, which is why you only see a 2x speedup with tensor cores. for comparison, V100 is something like 5x faster than P100 in Stable Diffusion 1.5, which is a huge and mostly convolutional network at its core.
my real point is the OP's question is similar to "I want to play cyberpunk 2077, but I only have a $100 budget". yes, you can make it run, but the experience will be a pale imitation of actually playing the game on its target hardware and really the answer is to save up and buy a more expensive GPU.
Click to expand...
Training a vision classification model on COCO data or running opencv on a gpu are excellent places to gain some experience. Why on earth would someone new to the field want to start on the bleeding edge?
 
You must log in or register to reply here.
Back
Top