Im using gpt 3.5 LLM model along with necessary prompts to get different levels of product description. My dataset contains 35K rows. I’m able to send 90 rows in one api call else I get rate limit error. But if im sending 35k in batches like this. Its consuming lot of time and I’m again getting same error.
This is my code and error attached. Please let me know how to handle for large csv dataset if anyone how I can use Batch API ( https://platform.openai.com/docs/api-reference/batch ) from Open AI here ?