I’m generally working on image processing and computer vision but nowadays i’m curious about natural language processing and diving deep into transformers and large language models.I know that text data must be transformed into a number representation when tokenizing and vectorizing.As i understand the following sentence is: