I created an ETL pipeline using python ETL PETL.
It works as expected, but my current transformation part is a bit slow.
I’m utilizing ProcessPoolExecutor to transform each petl data table in separate process. I realized that perhaps working on 1 data table at a time across all CPUs might be more performant than trying to work all data tables across CPUs since we might not be able to use all CPUs when there is a less number of work to be done than the number of cores.