Skip to content

mecfs_bio.build_system.task.pipes.duckdb_mem_limit_pipe

Classes:

DuckdbMemLimitPipe

Bases: DataProcessingPipe

Side-effecting pipe that sets the duckdb memory limit. Useful to apply prior to operations that consume a lot of memory. In these cases, counterintuitive, it often optimal to actually reduce the memory limit, as this will cause duckdb to more aggressively conserve memory.

Methods:

Attributes:

limit_gb instance-attribute 

limit_gb: int

process 

process(x: LazyFrame) -> narwhals.LazyFrame
Source code in mecfs_bio/build_system/task/pipes/duckdb_mem_limit_pipe.py 
19
20
21
def process(self, x: narwhals.LazyFrame) -> narwhals.LazyFrame:
    duckdb.sql(f"SET memory_limit='{self.limit_gb}GB'")
    return x