Relative Content

Tag Archive for pytorchcudallama

PyTorch: CUDA out of memory with enough VRAM

When I run the PyTorch script for Llama 3 8B-Instruct (first window), I get the error that PyTorch couldn’t allocate 112MB GPU RAM (see red line). In the window below however, you can see that Python manages to allocate all available 2,4 GB RAM. So this error is confusing me and it prevents me from using Llama 3 locally.