How to Set Up MLflow for Experiment Tracking

How to Set Up MLflow for Experiment Tracking

MLflow is an open-source platform for managing the full machine learning lifecycle, including experiment tracking, model versioning, and deployment. Hosting MLflow on your Breeze gives your team a centralized place to log experiments, compare results, and store models.

Prerequisites

• A Breeze instance with at least 2 GB of RAM
• Python 3.9 or later
• A database backend (PostgreSQL or MySQL recommended for production)

Installing MLflow

python3 -m venv ~/mlflow-env
source ~/mlflow-env/bin/activate
pip install mlflow psycopg2-binary boto3

Setting Up the Backend Store

MLflow stores experiment metadata in a database. Create a PostgreSQL database for it:

sudo -u postgres psql -c "CREATE DATABASE mlflow;"
sudo -u postgres psql -c "CREATE USER mlflow_user WITH PASSWORD 'secure_password';"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE mlflow TO mlflow_user;"

Configuring Artifact Storage

Create a local directory for storing model artifacts, or configure an S3-compatible store:

sudo mkdir -p /data/mlflow-artifacts
sudo chown $USER:$USER /data/mlflow-artifacts

Starting the MLflow Tracking Server

mlflow server \
  --backend-store-uri postgresql://mlflow_user:secure_password@localhost:5432/mlflow \
  --default-artifact-root /data/mlflow-artifacts \
  --host 0.0.0.0 \
  --port 5000

Access the MLflow dashboard at http://your-breeze-ip:5000.

Logging Experiments from Your Training Code

Integrate MLflow tracking into your Python training scripts:

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

mlflow.set_tracking_uri("http://your-breeze-ip:5000")
mlflow.set_experiment("iris-classification")

with mlflow.start_run():
    n_estimators = 100
    max_depth = 5

    mlflow.log_param("n_estimators", n_estimators)
    mlflow.log_param("max_depth", max_depth)

    model = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth)
    model.fit(X_train, y_train)

    accuracy = accuracy_score(y_test, model.predict(X_test))
    mlflow.log_metric("accuracy", accuracy)

    mlflow.sklearn.log_model(model, "model")
    print(f"Accuracy: {accuracy:.4f}")

Comparing Experiments

The MLflow UI lets you compare runs side by side. Select multiple runs in the experiment view and click “Compare” to see parameter and metric differences in parallel charts and tables.

Model Registry

Register your best models for versioned deployment:

result = mlflow.register_model(
    "runs:/<run_id>/model",
    "iris-classifier"
)

# Transition to production
from mlflow.tracking import MlflowClient
client = MlflowClient()
client.transition_model_version_stage(
    name="iris-classifier",
    version=result.version,
    stage="Production"
)

Running as a Systemd Service

Create a systemd unit so MLflow starts on boot:

[Unit]
Description=MLflow Tracking Server
After=network.target postgresql.service

[Service]
User=deploy
ExecStart=/home/deploy/mlflow-env/bin/mlflow server \
  --backend-store-uri postgresql://mlflow_user:secure_password@localhost:5432/mlflow \
  --default-artifact-root /data/mlflow-artifacts \
  --host 0.0.0.0 --port 5000
Restart=always

[Install]
WantedBy=multi-user.target

Enable with sudo systemctl enable --now mlflow to keep your experiment tracking server running on your Breeze.