utils

lanfactory.utils

Modules:

• mlflow_utils –

  Utilities for MLflow tracking and lineage in LANfactory.

• util_funs –

  Some utility functions for the lanfactory package.

Functions:

• get_files_from_data_generation_experiment –

  Get all files generated across all runs in a data generation experiment.

• log_training_data_lineage –

  Log training data lineage information to MLflow.

• save_configs –

  Function to save the network and training configurations to a folder.

lanfactory.utils.get_files_from_data_generation_experiment

get_files_from_data_generation_experiment(experiment_id: str, tracking_uri: str = 'sqlite:///mlflow.db') -> dict

Get all files generated across all runs in a data generation experiment.

This function queries all runs in a data generation experiment and collects the complete inventory of generated files across all distributed runs.

Arguments

experiment_id : str
    The MLflow experiment ID for data generation
tracking_uri : str
    MLflow tracking URI (default: "sqlite:///mlflow.db")

Returns:

• dict –

  Dictionary with keys: - num_runs: number of runs in the experiment - total_files: total number of files generated - all_files: list of all filenames - runs_info: list of dicts with run details

Raises:

• ImportError –

  If mlflow is not installed
  

  Exception If experiment cannot be accessed or has no runs

lanfactory.utils.log_training_data_lineage

log_training_data_lineage(data_generation_experiment_id: str, training_data_folder: Path, valid_file_list: list, n_training_files: int, tracking_uri: str = None) -> dict

Log training data lineage information to MLflow.

This function retrieves the expected files from the data generation experiment, compares them with actual files found, and logs the lineage information.

Arguments

data_generation_experiment_id : str
    MLflow experiment ID of the data generation experiment
training_data_folder : Path
    Path to the folder containing training data
valid_file_list : list
    List of Path objects for available training files
n_training_files : int
    Number of files actually used for training
tracking_uri : str, optional
    MLflow tracking URI (uses current if None)

Returns:

• dict –

  Dictionary with lineage information including: - data_generation_experiment_id - expected_files - actual_files_used - missing_files - extra_files

lanfactory.utils.save_configs

save_configs(model_id: str | None = None, save_folder: str | Path | None = None, network_config: dict | None = None, train_config: dict | None = None) -> None

Function to save the network and training configurations to a folder.

Arguments

model_id (str):
    The id of the model.
save_folder (str):
    The folder to save the configurations to.
network_config (dict):
    The network configuration dictionary.
train_config (dict):
    The training configuration dictionary.