Skip to content

mecfs_bio.build_system.runner.simple_runner

Classes:

Attributes:

logger module-attribute 

logger = get_logger()

SimpleRunner

Simple wrapper class that orchestrates an execution of the workflow build system with a topological scheduler and verifying trace rebuilder

info_store: path at which to store persistent cash for build-system internal information asset_root: root under which to create the asset store tracer: algorithm uses to calculate verifying traces of assets. An example would be a hashing algorithm. changing this forces all assets to be rebuilt

Methods:

  • run –

    Targets: the ultimate targets we aim to produce. All transitive dependencies of these targets will either be rebuilt, or fetched (determined according to that status of their trace)

Attributes:

asset_root instance-attribute 

asset_root: Path

info_store instance-attribute 

info_store: Path

meta_to_path property 

meta_to_path: SimpleMetaToPath

tracer class-attribute instance-attribute 

tracer: Tracer = md5_hasher()

run 

run(
    targets: list[Task],
    must_rebuild_transitive: Sequence[Task] = tuple(),
    incremental_save: bool = True,
    settings: TopologicalSchedulerSettings = TopologicalSchedulerSettings(),
) -> Mapping[AssetId, Asset]

Targets: the ultimate targets we aim to produce. All transitive dependencies of these targets will either be rebuilt, or fetched (determined according to that status of their trace) must_rebuild_transitive: list of tasks that the rebuilder will be forced to rebuild, regardless of the status of their trace. - This is particularly useful when you have changed the code that generates and asset, and so want it and its dependees to be regenerated. returns: mapping from asset id to file system information for all assets that were built or retrieved as part of the execution of the scheduler

Source code in mecfs_bio/build_system/runner/simple_runner.py 
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
def run(
    self,
    targets: list[Task],
    must_rebuild_transitive: Sequence[Task] = tuple(),
    incremental_save: bool = True,
    settings: TopologicalSchedulerSettings = TopologicalSchedulerSettings(),
) -> Mapping[AssetId, Asset]:
    """
    Targets: the ultimate targets we aim to produce.  All transitive dependencies of these targets will either be rebuilt, or fetched (determined according to that status of their trace)
    must_rebuild_transitive: list of tasks that the rebuilder will be forced to rebuild, regardless of the status of their trace.
       - This is particularly useful when you have changed the code that generates and asset, and so want it and its dependees to be regenerated.
    returns:
    mapping from asset id to file system information for all assets that were built or retrieved as part of the execution of the scheduler
    """
    if self.info_store.is_file():
        info = VerifyingTraceInfo.deserialize(self.info_store)
    else:
        info = VerifyingTraceInfo.empty()
    if incremental_save:
        logger.debug("incremental save is enabled")
    rebuilder = VerifyingTraceRebuilder(self.tracer)
    wf = RobustDownloadWF()
    # wf= SimpleWF()
    meta_to_path = self.meta_to_path
    tasks = find_tasks(targets)
    must_rebuild_graph = dependees_of_targets_from_tasks(
        tasks=tasks,
        targets=[task.asset_id for task in must_rebuild_transitive],
    )
    info = attrs.evolve(info, must_rebuild=set(must_rebuild_graph.nodes))
    info = info_purge(info, tasks=tasks)
    incremental_save_path = self.info_store if incremental_save else None
    store, info = topological(
        rebuilder=rebuilder,
        tasks=tasks,
        info=info,
        wf=wf,
        meta_to_path=meta_to_path,
        targets=[target.asset_id for target in targets],
        incremental_save_path=incremental_save_path,
        settings=settings,
    )
    info.serialize(self.info_store)
    _print_target_locs(store, targets=targets)
    return store