#Kubeflow Pipelines Execution and Storage Behavior

Kubeflow Pipelines 2.16.1 installed by the Data Science Pipelines Operator (DSPO) supports task caching, dsl.ParallelFor, typed KFP artifacts, and S3-compatible artifact persistence on x86 clusters. These mechanisms are KFP features; DSPO configures the API server, workflow controller, cache server, and artifact store that implement them.

#Feature summary

The verification pipeline linked below produced one Dataset, read it from three ParallelFor tasks, and then submitted the identical pipeline again. All three consumers succeeded, and four executor tasks in the second run were reported as SKIPPED because their cached results were reused. The dataset object remained present in S3 after the workflow pods completed.

#Cache a task safely

Caching is enabled for a run by default. It can also be set when submitting a compiled package:

client.create_run_from_pipeline_package(
    pipeline_file="pipeline.yaml",
    arguments={"dataset_version": "2026-07-23"},
    enable_caching=True,
)

Disable caching for a task with external side effects or mutable inputs:

task = publish_report(...)
task.set_caching_options(False)

KFP computes a cache key from the component and its declared inputs. It cannot detect that the contents behind an unchanged S3 object key, URL, database query, or image tag changed. Pass an immutable object version, digest, or explicit version parameter when fresh external content must invalidate the cache.

#Run tasks with ParallelFor

Use dsl.ParallelFor when the same component can process independent items:

from kfp import dsl


@dsl.pipeline(name="parallel-evaluation")
def evaluation_pipeline():
    dataset = produce_dataset()
    with dsl.ParallelFor(
        items=["accuracy", "precision", "recall"],
        parallelism=3,
    ) as metric:
        evaluate(dataset=dataset.outputs["dataset"], metric=metric)

The controller creates one evaluate task for each item. Set parallelism below the item count when namespace quota, node capacity, or an external API cannot sustain every iteration at once.

#Persist and pass a typed artifact

Write files to the path supplied by the output artifact. Do not invent a local path and expect it to survive the producer pod:

from kfp import dsl


@dsl.component(base_image="python:3.12-slim")
def produce_dataset(dataset: dsl.Output[dsl.Dataset]):
    from pathlib import Path

    output = Path(dataset.path)
    output.parent.mkdir(parents=True, exist_ok=True)
    output.write_text("value\n42\n", encoding="utf-8")
    dataset.metadata["rows"] = 1


@dsl.component(base_image="python:3.12-slim")
def consume_dataset(dataset: dsl.Input[dsl.Dataset]):
    from pathlib import Path

    print(Path(dataset.path).read_text(encoding="utf-8"))

Passing producer.outputs["dataset"] to a consumer establishes the dependency. The KFP launcher uploads the producer's file to the artifact store and materializes it at dataset.path in the consumer pod.

This is different from a KFP workspace PVC:

#Run the verification pipeline

Download verify-kfp-features.py and run it from a Workbench or another environment that can reach the KFP API:

python -m pip install 'kfp==2.16.1'

export KFP_ENDPOINT='http://<kfp-api-service>:8888'
export KFP_NAMESPACE='my-pipeline-namespace'
python verify-kfp-features.py

The script compiles one producer and three parallel consumers, submits it twice with the same parameter, and fails unless all of the following are true:

• the first run succeeds;
• exactly three verify-dataset tasks succeed;
• every consumer reads the producer's Dataset content; and
• the second run reports cached tasks as SKIPPED.

The final output contains one PASS line for each mechanism. An administrator can independently confirm persistence by listing the run prefix in the KFP artifact bucket after the pods finish.

#Platform compatibility and database backends

The following results apply to the Alauda builds aligned with KFP 2.16.1:

On arm64, the current kfp-metadata-envoy, persistence agent, scheduled workflow controller, API server, and pipelines-components images do not contain an arm64 manifest. The DSPO manager being Ready therefore does not establish that a pipeline service is usable.

Both operators currently expose only MySQL connection settings. DSPO performs its external database health check with the MySQL driver and renders DB_DRIVER_NAME=mysql. The kfp-operator chart renders dbType=mysql, DBDriverName=mysql, and DBCONFIG_MYSQLCONFIG_*. Pointing either operator at a PostgreSQL endpoint does not switch its driver and does not produce a usable KFP installation. Use MySQL until a PostgreSQL driver selection and the upstream PostgreSQL deployment patches are wired into the operator interface.