#Installation

MLflow is delivered as an OLM Operator (the MLflow Operator) and installed from the platform OperatorHub. Installing the tracking server is a two-step flow: install the operator, then create one MLflow custom resource that reconciles the tracking-server stack.

#Prerequisites

• Alauda AI installed on the target cluster. The operator and its images ship both linux/amd64 and linux/arm64.

• Operator Lifecycle Manager (OLM) available on the target cluster (provided by the platform).

• A PostgreSQL 12 or later database reachable from the cluster, holding a database named mlflow for MLflow metadata. Have the host, port, username, and password ready, and create the database before you install:

  CREATE DATABASE mlflow;

  Use a highly available PostgreSQL service for production (see High availability and storage).

• For production artifact storage, an existing S3-compatible bucket in SeaweedFS, MinIO, Ceph RGW, or AWS S3. The operator can proxy artifact access so clients do not need object-storage credentials. The in-pod filesystem default is suitable only for smoke tests.

• The platform OAuth / OIDC provider (provided by Alauda AI) — the OAuth proxy in front of the tracking server uses it for single sign-on.

• For multi-tenancy, the default workspace namespace must already exist and carry the label mlflow-enabled=true before the tracking server first starts. Otherwise the server crash-loops with Workspace '<name>' not found in the Kubernetes cluster.

  kubectl create namespace finetune
  kubectl label namespace finetune mlflow-enabled=true

#Upload Operator

Download the MLflow Operator bundle from the Customer Portal / Marketplace (e.g. mlflow.ALL.xxxx.tgz), then publish it to the platform repository with the violet command-line tool:

violet push \
  --platform-address=<platform-access-address> \
  --platform-username=<platform-admin> \
  --platform-password=<platform-admin-password> \
  mlflow.ALL.xxxx.tgz

#Install Operator

In the Administrator view:

1. Click Marketplace / OperatorHub.
2. At the top of the console, from the Cluster dropdown, select the destination cluster.
3. Search for and select MLflow Operator, then click Install.
4. Leave Channel unchanged (stable).
5. Check that the Version is v3.13.0.
6. Leave Installation Location unchanged — it defaults to the mlflow-operator namespace.
7. Choose an Upgrade Strategy (Manual is recommended for production).
8. Click Install.

Confirm the operator is ready before continuing:

# the CSV reports Succeeded
kubectl get csv -n mlflow-operator | grep mlflow-operator

# the operator controller-manager pod is Running
kubectl get pods -n mlflow-operator

#Create the MLflow tracking server

The operator does nothing until you create an MLflow custom resource (mlflow.alauda.io/v1alpha1). You can create it two ways:

• kubectl — apply an MLflow manifest.
• Alauda Console UI — from Administrator → Marketplace → OperatorHub → MLflow Operator → All instances → Create.

Both create the same resource; use whichever fits your workflow.

#Method A — using kubectl

Set the database connection and multi-tenancy fields, then apply:

apiVersion: mlflow.alauda.io/v1alpha1
kind: MLflow
metadata:
  name: mlflow
spec:
  # Namespace the tracking server is deployed into (chart-mandated: kubeflow).
  namespace: kubeflow

  # External PostgreSQL for MLflow metadata — the `mlflow` database must already exist.
  pgHost: postgresql.database.svc
  pgPort: 5432
  pgUsername: mlflow
  pgPassword: <db-password>

  # Multi-tenancy: map Kubernetes namespaces to MLflow workspaces.
  multitenancy:
    enabled: true
    defaultWorkspace: finetune                 # must exist and be labelled before first boot
    workspaceLabelSelector: mlflow-enabled=true

  # Run-artifact location (see High availability and storage below).
  artifacts:
    defaultArtifactRoot: /mlflow/artifacts
    # Recommended for production. The bucket and Secret must already exist.
    s3:
      enabled: true
      endpointUrl: https://object-storage.example.com
      bucket: mlflow
      prefix: production
      existingSecret: mlflow-s3-credentials
      accessKeyIdKey: AWS_ACCESS_KEY_ID
      secretAccessKeyKey: AWS_SECRET_ACCESS_KEY
      proxied: true

Save as mlflow.yaml and apply:

kubectl apply -f mlflow.yaml

#Method B — using the Alauda Console UI

1. In the Administrator view, go to Marketplace → OperatorHub.
2. From the Cluster dropdown at the top, select the target cluster.
3. Open the installed MLflow Operator.
4. Switch to the All instances tab and click Create (choose MLflow if prompted).
5. Fill in the form, or switch to YAML view and paste the manifest from Method A. Set the PostgreSQL connection, the multi-tenancy workspace fields, and the S3 artifact-storage fields for production.
6. Click Create.

#Verification

Verify the tracking server reconciled in the configured namespace (kubeflow):

# the operand exists
kubectl get mlflow

# the tracking-server Deployment is Available (two containers: the server + oauth2-proxy)
kubectl -n kubeflow get deploy mlflow-tracking-server

# its pod is Running
kubectl -n kubeflow get pods -l app.kubernetes.io/name=mlflow-tracking-server

Then open Alauda AI → Tools → MLFlow. The run owner and visible workspaces reflect the identity you sign in as.

#High availability and storage

• Database. MLflow stores all experiment/run/registry metadata in the external PostgreSQL you configured. Use a highly available PostgreSQL service for production; the tracking server itself is stateless with respect to metadata.
• Artifacts. The default artifacts.defaultArtifactRoot: /mlflow/artifacts is an emptyDir local to the tracking-server pod. It is not durable, and external clients cannot use the server-local path reliably. Starting with v3.13.0, enable artifacts.s3 for production before users log artifacts or register models. With the default proxied: true, the tracking server stores the S3 credentials and clients upload through the authenticated MLflow endpoint.
• Replicas. The default deployment is single-replica. It is not a multi-replica high-availability deployment unless your release notes state otherwise.

Create the referenced credentials Secret in the tracking-server namespace before applying the MLflow resource:

kubectl -n kubeflow create secret generic mlflow-s3-credentials \
  --from-literal=AWS_ACCESS_KEY_ID='<access-key>' \
  --from-literal=AWS_SECRET_ACCESS_KEY='<secret-key>'

The bucket must already exist. Leave endpointUrl empty for AWS S3. For S3-compatible services, set it to the service endpoint. You can also set region, ignoreTls, and prefix under artifacts.s3 when required.

#Bearer-token clients

Starting with v3.13.0, the operator enables spec.auth.oauth.skipJwtBearerTokens: true by default, allowing the MLflow Python SDK to authenticate with a Dex id token through MLFLOW_TRACKING_TOKEN. If an existing installation overrides the OAuth settings, verify that this field remains enabled. See Verify the token method in the SDK guide. The browser session-cookie method also remains available.

#Upgrade from v3.10.x to v3.13.0

The stable OLM channel supports this upgrade. Before approving it, back up the PostgreSQL backend database and the S3 artifact bucket. The backend-store schema migration runs automatically when the tracking server starts.

After the upgrade, keep spec.auth.kubernetes.authorizationMode: user_identity_token and spec.auth.oauth.skipJwtBearerTokens: true. These defaults preserve the documented Dex bearer-token flow.

#Troubleshooting

• If the MLFlow Tools-menu entry is missing, verify that the aml-mlflow-menu-config ConfigMap exists in the tracking-server namespace and has the label aml.cpaas.io/centralMenuItem: "true".
• If the server does not start and the pod logs Workspace '<name>' not found, the defaultWorkspace namespace does not exist or is not labelled mlflow-enabled=true. Create and label it, then restart the pod.
• If the server does not start for another reason, verify PostgreSQL connectivity, that the mlflow database exists, and that the credentials in spec.pg* are correct.
• If artifact uploads or model registration fail, verify that artifacts.s3.enabled is true, the bucket exists, and the referenced credentials Secret and key names are correct.
• If a workspace is not visible, verify that its namespace matches the configured workspaceLabelSelector.
• If requests are denied (403 PERMISSION_DENIED), check the user's Kubernetes RoleBinding in the workspace namespace — see Workspaces and access control.