NVIDIA Tesla M40 for AI work in Ubuntu VM - is it a good idea?
-
I built an AI vm a little while ago running ollama on Ubuntu 24.04. I have no GPU in that host so I threw 32GB RAM and 32 cores at it and it works, slowly. I knew that a GPU is really the way to go, especially if I want to be able to do image generation. Then today I saw this article. https://blog.briancmoses.com/2024/09/self-hosting-ai-with-spare-parts.html
A GPU for around $90 could be interesting. Any of you guys have a guess at the difficulty of getting an NVIDIA Tesla M40 (https://www.ebay.com/itm/274978990880) to work in an Ubuntu VM in a PowerEdge 730xd host? I see that the card has power requirements but don't know if the supplies in the server take care of that since this card is 10 years old and the server likely is too, and the card is designed for server use - maybe it wouldn't need tweaking.
Am I insane to be considering this?
-
@CodeMercenary Probably not insane if you want to learn using ollama or other LLM frameworks for inference. But the M40 is an ageing GPU with a low compute capability (v5.2), so with time, it might not be supported any more by platforms like ollama, vLLM, llama.cpp or aprhodite (did not check if they actually support that GPU, but Ollama has support for the M40). I doubt that you get an acceptable performance for stable diffusion (image generation) or training/fine-tuning. But what could you expect for $90?
The card has a power consumption (TDP) of 250W which is compatible with the 16x PCIe slot of riser #2. You have to be extra careful with the cable as it is not a standard cable. While most would suggest power supplies of 1100W for the Dell R730 to be on the save side, I run two P40s with 750W power supplies in a Dell R720. But I also power limit the card to 140W with little effect on the performance and have light workloads and no batch processing.
-
@gskger Thanks for the input. I also have dual 750W power supplies and I'd be totally fine limiting the power of the card. I'm not looking for crazy performance, just better than what I get with CPUs and enabling things that are a lot harder to even get working with CPU only, like stable diffusion.
I will absolutely trade performance for less noise and heat in my small server closet. Ideally I'd want the GPU card to consume nothing and need very minimal cooling if I'm not actively running a task in ollama. I can currently hear the fans spool up in that server every time I give ollama a task to run. I don't mind the extra noise when doing something as long as it doesn't permanently make the server more noisy.
I do have concerns about possibly needing external cooling for it as suggested in the article, I'd love to not need active cooling. If limiting the power consumption means that the existing case airflow is sufficient then I'd be happy with that.
-
@CodeMercenary The M40 is a server card and (physically) compatible with the R730, so no extra cooling is required (and possible). The downside is that the R730 most likely will still go full blast on all fans regardless of the actual power consumption since the server can not read the GPUs temperature. But there are scripts to manage the fan speeds based on server or GPU temperature. And once you have ollama installed, you can ask it how to write that code
-
@gskger Great to know, thank you.
I actually already have Ollama installed and running with Open WebUI so I can ask it ahead of time to see what it would be like. I've installed some more code specific models that are better suited to that kind of question.
Running the case fans full blast all the time would be a non-starter so I'm glad you let me know that as I count the costs of attempting this.
