Polars read_database does not respect iter_batches = True when using sqlalchemy/oracledb

  Kiến thức lập trình

This query using polars and sqlalchemy does not fetch chunked.

d = pl.read_database(sql_text(sql), e, iter_batches=True, batch_size=1000000)

The connection is a sqlalchemy create_engine(“oracle+oracledb://@xxx”) on top of oracledb.

It fetches every 2 billion rows, or tries to, ending in a memoryerror. Any way of getting the message through to the engine to do batched fetches in another way?

LEAVE A COMMENT