What happens when we define a BitsAndBytesConfig without enabling load in 4 or 8?
I have been experimenting with doing PEFT with LoRA on some HuggingFace models. After performing many experiments and documenting them, I noticed that I did not specify the type of quantization when defining a BitsAndBytesConfig as follows:
How do I load this Hugging Face model properly?
My goal is to load a pre-trained Hugging Face model, train it, save it, and then load it. Here are the steps that I took: