PyTorch is using an ungodly amount of VRAM
I’m trying to finetune GPT-2 on my own dataset and when I train it, it all goes smoothly until it reaches the evaluation step (every 500) and throws this error
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 7.72 GiB. GPU 0 has a total capacity of 24.00 GiB of which 13.38 GiB is free. Process 13650 has 17179869184.00 GiB memory in use. Process 48439 has 17179869184.00 GiB memory in use. Process 21062 has 17179869184.00 GiB memory in use. Process 40988 has 17179869184.00 GiB memory in use. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 9.14 GiB is allocated by PyTorch, and 187.59 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Freeze layers of existing model in Pytorch
I want to apply additional training to a pre-trained model in Pytorch. Here is the result of print(model):