#Training Guides
End-to-end recipes for fine-tuning and pretraining LLMs on Alauda AI.
#TOC
Pick a path#Pick a path
| When you want… | Use | Guide |
|---|---|---|
| Reusable templates, repeatable runs, optional Kueue quotas | Kubeflow Trainer v2 + LlamaFactory | Fine-Tuning with Kubeflow Trainer v2 |
| Survive crashes, preemption, or a pause without losing hours of training | Checkpoint PVC + resume_from_checkpoint + maxRestarts | Checkpointing and Resuming TrainJobs |
| Mix training with online inference, yield GPU back on demand | Kueue cohort + preemption + checkpoint resume | Preemptible TrainJobs with Kueue |
| Run a fine-tune on a slice of a GPU instead of a whole card | Dynamic Resource Allocation (DRA) MIG slice + Trainer v2 | GPU Slicing with Dynamic Resource Allocation (DRA) |
A curated set of TrainingRuntime images (CUDA / CANN) | Trainer v2 runtime catalog | Training Runtime Images |
| One-shot quick start of distributed PyTorch on Trainer v2 | ClusterTrainingRuntime + MNIST | Kubeflow Trainer Quick Start |
| Production SFT / OSFT with automatic memory management | training_hub | Fine-tuning LLMs with Training Hub |
| Interactive exploration, custom scripts, VolcanoJob submission | Workbench Notebook | Fine-tuning LLMs using Workbench |
| Full-parameter SFT / pretraining on Ascend NPU | Workbench PyTorch CANN / MindSpore CANN | Fine-tune and Pretrain on Ascend NPU |
| Track a KFP run's parameters and metrics in MLflow | KFP component + MLflow SDK | Kubeflow Pipeline + MLflow Integration |
| Train tabular or time-series models with reusable AutoGluon assets | Managed KFP pipelines or composable components | Use Reusable Kubeflow Pipeline Components |
| Use KFP caching, parallel loops, and persistent typed artifacts | KFP 2.16.1 execution mechanisms | Kubeflow Pipelines Execution and Storage Behavior |
| Daily fine-tune → evaluate → compare loop with MLflow + TrustyAI | KFP Recurring Run + MLflow Model Registry + LMEvalJob | Daily Fine-Tuning Pipeline with MLflow and TrustyAI |