Option to track CPU/GPU metrics

[blumenstiel]

MLFlow can track the compute metrics similar to WandB. It would be good to give the user the possibility to activate it.

See https://mlflow.org/docs/latest/system-metrics/index.html

The metrics can be logged with:

    logger = MLFlowLogger(
        experiment_name=LOG_EXPERIMENT_NAME,
        run_name=run_name,
        tracking_uri=f"file:{LOG_DIR}{LOG_PROJECT}",
    )
    logger.log_hyperparams(config)

    # Start logging system metrics
    mlflow.start_run(
        run_id=logger.run_id,
        experiment_id=logger.experiment_id,
        log_system_metrics=True)

[leotizzei]

@blumenstiel , given that is it possible to set the environment variable MLFLOW_ENABLE_SYSTEM_METRICS_LOGGING to track system metrics, what is your opinion if we implement this way:

    if os.getenv("MLFLOW_ENABLE_SYSTEM_METRICS_LOGGING") is None:
        os.environ["MLFLOW_ENABLE_SYSTEM_METRICS_LOGGING"] = "true" 

By default, system metrics would be logged. If users set MLFLOW_ENABLE_SYSTEM_METRICS_LOGGING to false, then it will not be logged

[paolofraccaro]

Hey @leotizzei ! Me and @blumenstiel just checked that your option above worked and we can see in mlflow the system metrics.