#Use Reusable Kubeflow Pipeline Components

Alauda AI packages reusable components from the mirrored pipelines-components project with Kubeflow Pipelines 2.16.1. A Kubeflow Pipelines installation preloads two beta AutoGluon pipelines:

The managed pipelines are the shortest path to a complete training run. You can also clone the mirror and import individual components when you need a custom pipeline DAG.

#Prerequisites

The input bucket can differ from the object store configured as the KFP artifact store. The Secret below authenticates the data loader to the input bucket; KFP supplies artifact-store credentials to its launcher separately.

#Create the input-storage Secret

Create the Secret in the pipeline run namespace:

NS=my-pipeline-namespace

kubectl -n "$NS" create secret generic pipeline-input-s3 \
  --from-literal=AWS_ACCESS_KEY_ID='<access-key>' \
  --from-literal=AWS_SECRET_ACCESS_KEY='<secret-key>' \
  --from-literal=AWS_S3_ENDPOINT='https://s3.example.com' \
  --from-literal=AWS_DEFAULT_REGION='us-east-1'

Keep the URI scheme in AWS_S3_ENDPOINT. The component passes this value to the S3 client as its endpoint URL.

Upload the training file to the bucket before starting the run. For example, the following tabular input has two features and a regression label:

feature_a,feature_b,price
1.0,2.0,10.5
2.0,3.0,20.5
3.0,4.0,30.5

Tabular data must contain at least 100 valid rows after duplicate rows, invalid labels, and other unusable values are removed.

#Run a managed tabular pipeline

1. Open Kubeflow Pipelines and select Pipelines.
2. Search for autogluon-tabular-training-pipeline and select its latest version.
3. Select Create run, choose the namespace containing the Secret, and set the inputs:

4. Start the run and follow the data-loader and model-training tasks in the run graph.

The pipeline samples and splits the source data, trains and ranks AutoGluon models, and refits the best top_n models. Training splits are shared through the run's workspace PVC. The test split, leaderboard, model predictors, metrics, and generated inference notebooks are KFP artifacts persisted in the configured artifact store.

#Run a managed time-series pipeline

Select autogluon-timeseries-training-pipeline and provide the common storage inputs described above. Its time-series-specific inputs are:

The loader deduplicates ID/timestamp pairs and performs a temporal split for each series. It writes the working train splits to the workspace PVC and the test split and trained models to the artifact store.

#Compose a custom pipeline from the mirror

The kfp-components package is not published to PyPI. Clone the Alauda mirror and install it in a Python 3.11 or later virtual environment. Pin the SDK to the version shipped by Alauda AI:

git clone https://gitlab-ce.alauda.cn/aml/pipelines-components.git
cd pipelines-components

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install 'kfp==2.16.1' 'kfp-kubernetes==2.16.1'
python -m pip install --no-deps -e .

This example imports the tabular data loader and uses it in a smaller custom pipeline. Replace the image with the AutoML runtime mirrored into your cluster:

from kfp import compiler, dsl, kubernetes
from kfp_components.components.data_processing.automl.tabular_data_loader import (
    automl_data_loader,
)

AUTOML_IMAGE = "<registry>/mlops/kubeflow/odh-automl:odh-stable-20260720"


@dsl.pipeline(
    name="custom-tabular-loader",
    pipeline_config=dsl.PipelineConfig(
        workspace=dsl.WorkspaceConfig(
            size="12Gi",
            kubernetes=dsl.KubernetesWorkspaceConfig(
                pvcSpecPatch={"accessModes": ["ReadWriteOnce"]}
            ),
        )
    ),
)
def custom_tabular_loader(
    secret_name: str,
    bucket: str,
    object_key: str,
    label: str,
    task_type: str = "regression",
):
    loader = automl_data_loader(
        bucket_name=bucket,
        file_key=object_key,
        workspace_path=dsl.WORKSPACE_PATH_PLACEHOLDER,
        label_column=label,
        task_type=task_type,
    )
    loader.set_container_image(AUTOML_IMAGE)
    kubernetes.use_secret_as_env(
        loader,
        secret_name=secret_name,
        secret_key_to_env={
            "AWS_ACCESS_KEY_ID": "AWS_ACCESS_KEY_ID",
            "AWS_SECRET_ACCESS_KEY": "AWS_SECRET_ACCESS_KEY",
            "AWS_S3_ENDPOINT": "AWS_S3_ENDPOINT",
            "AWS_DEFAULT_REGION": "AWS_DEFAULT_REGION",
        },
    )


compiler.Compiler().compile(
    pipeline_func=custom_tabular_loader,
    package_path="custom-tabular-loader.yaml",
)

Upload custom-tabular-loader.yaml from the Kubeflow Pipelines UI, or submit it with the KFP SDK. Review the component's README.md, metadata.yaml, and tests in the mirror before using another asset; the repository contains alpha and beta assets with different external-service requirements.

#Troubleshooting

• Secret not found: Secrets are namespace-scoped. Create the input Secret in the same namespace as the run.
• Workspace PVC remains Pending: Configure a default storage class that supports ReadWriteOnce, or ask the platform administrator to configure the KFP workspace storage class.
• Run remains Pending: Compare available cluster resources with the CPU and memory requests in Prerequisites. Start with preset=speed.
• Input object cannot be read: Check the endpoint scheme, bucket and object key, credentials, region, network policy, and the S3 server certificate.
• Artifacts are missing after pods finish: The pipeline input Secret does not configure KFP persistence. Check the KFP installation's artifact-store endpoint and credentials with the platform administrator.