I have one year worth of transaction data inside a mssql database and table. I’ve extracted the table day by day incrementally using pyodbc, then proceed to preprocess the extracted data, and then feature engineer them and then save the data to a csv file. As an example, the ranges of data inside the transaction data is 2022-01-01 until 2022-12-31. As the date goes up one by one, the process of table extraction takes longer and longer. During the extraction at date 2022-01-01, it will only take 10 seconds of extraction, but at 2022-01-29, it will take more than 3 hours of extraction. I don’t know the exact reason as to why this happened, and how to solve it. my current solution is to stop the python program at 9pm, and then a task scheduler will start the python program again the next day from a checkpoint.

I am tying to install ‘pyodbc’ in windows. I have installed the driver from https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-2017#download-for-windows and then did a [pip install pyodbc’. But getting the following: