Train on GPU and Run Inference on NPU
Training and inference can use different accelerator types. This page is an index for workflows that fine-tune or train a model on NVIDIA GPUs and serve a compatible exported model on Huawei Ascend NPUs. It does not replace the linked guides or provide deployment steps.
TOC
Supported model artifacts for NPU inferenceFine-tune LLMs on GPU and run LLM inference on NPUTrain machine-learning models on GPU and run inference on NPUSupported model artifacts for NPU inference
The selected NPU runtime determines which artifacts it can load. The following are the model representations covered by the referenced NPU-serving guides; they are not a universal compatibility matrix for every Ascend device, model architecture, or runtime version.
transformers in an InferenceService or ClusterServingRuntime identifies the model-directory format expected by the runtime. It does not by itself state that every precision or quantization method is supported. Validate the exact model, weight representation, serving image, CANN version, driver, and NPU SKU before deployment.
Fine-tune LLMs on GPU and run LLM inference on NPU
Use one of these guides to fine-tune an LLM on GPU:
- Fine-Tuning with Kubeflow Trainer v2 for a reusable
TrainingRuntimeandTrainJobworkflow. - Fine-tuning LLMs with Training Hub for SFT, OSFT, LoRA, QLoRA, and continued pre-training workflows.
- Training Runtime Images to select a CUDA training runtime.
After the fine-tuned model is published to the model storage location used by your platform, use the NPU-serving guidance in Extend Inference Runtimes. Its vLLM-ascend section covers Ascend NPU runtime and InferenceService configuration. For platform-wide service lifecycle and operations, see Managing Inference Services.
Verify that the exported model, selected NPU serving engine, CANN stack, tokenizer, and model-storage format are compatible before moving a GPU-trained model to an NPU serving environment.
Train machine-learning models on GPU and run inference on NPU
For an end-to-end object-detection example, use Train, Fine-Tune, and Deploy YOLOv5. It covers GPU training with Kubeflow Trainer v2 and two Ascend NPU inference choices:
- A custom KServe runtime that serves the exported TorchScript model with
torch_npu. - A native CANN path that converts a YOLOv5 ONNX model to an
.ommodel with ATC and executes it through AscendCL.
The YOLOv5 guide also documents the validated CANN compiler image and NPU environment. Use it as the reference for artifact conversion, model storage, and NPU runtime compatibility for similar machine-learning models.