Python cx_oracle concurrent fetch
When using Python cx_Oracle – we set a parameter cursor.arraysize = 10000. I am assuming that this means the Python client running on the server receives data “sequentially” in batches of 10K from the Oracle database. Let’s say that we are pulling 500Million records and the database can handle the load and the server on which Python is running has enough resources to handle and no issues with network bandwidth. How can we still keep it as 10k but parallelize the fetch to concurrently pull the data, concurrency of 15 let’s say. After pulling 10k records, we create files and ftp the files into an object storage. All I am trying to figure out is how to implement concurrency here while fetching the data from the database.