Create a Multi-cluster Pipeline

As cloud providers offer different hosted Kubernetes services, DevOps pipelines have to deal with use cases where multiple Kubernetes clusters are involved.

This tutorial demonstrates how to create a multi-cluster pipeline on KubeSphere.


Workflow Overview

This tutorial uses three clusters to serve as three isolated environments in the workflow. See the diagram as below.


The three clusters are used for development, testing, and production respectively. Once codes get submitted to a Git repository, a pipeline will be triggered to run through the following stages—Unit Test, SonarQube Analysis, Build & Push, and Deploy to Development Cluster. Developers use the development cluster for self-testing and validation. When developers give approval, the pipeline will proceed to the stage of Deploy to Testing Cluster for stricter validation. Finally, the pipeline, with necessary approval ready, will reach the stage of Deploy to Production Cluster to provide services externally.

Hands-on Lab

Step 1: Prepare clusters

See the table below for the role of each cluster.

Cluster NameCluster RoleUsage
hostHost ClusterTesting
shireMember ClusterProduction
rohanMember ClusterDevelopment


These Kubernetes clusters can be hosted across different cloud providers and their Kubernetes versions can also vary. Recommended Kubernetes versions for KubeSphere v3.1.0: v1.17.9, v1.18.8, v1.19.8 and v1.20.4.

Step 2: Create a workspace

  1. Log in to the web console of the Host Cluster as ws-manager. On the Workspaces page, click Create.

  2. On the Basic Information page, name the workspace devops-multicluster, select ws-admin for Administrator, and click Next.


  3. On the Select Clusters page, select all three clusters and click Create.


  4. The workspace created will display in the list. You need to log out of the console and log back in as ws-admin to invite both project-admin and project-regular to the workspace and grant them the role workspace-self-provisioner and workspace-viewer respectively. For more information, refer to Create Workspaces, Projects, Accounts and Roles.


Step 3: Create a DevOps project

  1. Log out of the console and log back in as project-admin. Go to the DevOps Projects page and click Create.

  2. In the dialog that appears, enter multicluster-demo for Name, select host for Cluster Settings, and then click OK.



    Only clusters with the DevOps component enabled will be available in the drop-down list.
  3. The DevOps project created will display in the list. Make sure you invite the account project-regular to this project with the role operator. For more information, refer to Create Workspaces, Projects, Accounts and Roles.


Step 4: Create projects on clusters

You must create the projects as shown in the table below in advance. Make sure you invite the account project-regular to these projects with the role operator. For more information about how to create a project, refer to Create Workspaces, Projects, Accounts and Roles.

Cluster NameUsageProject Name

Step 5: Create credentials

  1. Log out of the console and log back in as project-regular. On the DevOps Projects page, click the DevOps project multicluster-demo.

  2. On the DevOps Credentials page, you need to create the credentials as shown in the table below. For more information about how to create credentials, refer to Credential Management and Create a Pipeline Using a Jenkinsfile.

    Credential IDTypeWhere to Use
    hostkubeconfigThe Host Cluster for testing
    shirekubeconfigThe Member Cluster for production
    rohankubeconfigThe Member Cluster for development
    dockerhub-idAccount CredentialsDocker Hub
    sonar-tokenSecret TextSonarQube


    You have to manually enter the kubeconfig of your Member Clusters when creating the kubeconfig credentials shire and rohan. Make sure your Host Cluster can access the APIServer addresses of your Member Clusters.
  3. You will have five credentials in total.


Step 6: Create a pipeline

  1. Go to the Pipelines page and click Create. In the dialog that appears, enter build-and-deploy-application for Name and click Next.


  2. In the Advanced Settings tab, click Create to use the default settings.

  3. The pipeline created will display in the list. Click it to go to its detail page.


  4. Click Edit Jenkinsfile and copy and paste the following contents. Make sure you replace the value of DOCKERHUB_NAMESPACE with your own value, and then click OK.

    pipeline {
      agent {
        node {
          label 'maven'
      parameters {
            string(name:'BRANCH_NAME',defaultValue: 'master',description:'')
      environment {
            DOCKER_CREDENTIAL_ID = 'dockerhub-id'
            DEV_KUBECONFIG_CREDENTIAL_ID = 'rohan'
            REGISTRY = ''
            DOCKERHUB_NAMESPACE = 'your Docker Hub account ID'
            APP_NAME = 'devops-java-sample'
            SONAR_CREDENTIAL_ID = 'sonar-token'
      stages {
        stage('checkout') {
          steps {
            container('maven') {
              git branch: 'master', url: ''
        stage('unit test') {
          steps {
            container('maven') {
              sh 'mvn clean -o -gs `pwd`/configuration/settings.xml test'
        stage('sonarqube analysis') {
          steps {
            container('maven') {
              withCredentials([string(credentialsId: "$SONAR_CREDENTIAL_ID", variable: 'SONAR_TOKEN')]) {
                withSonarQubeEnv('sonar') {
                  sh "mvn sonar:sonar -o -gs `pwd`/configuration/settings.xml -Dsonar.login=$SONAR_TOKEN"
        stage('build & push') {
          steps {
            container('maven') {
              sh 'mvn -o -Dmaven.test.skip=true -gs `pwd`/configuration/settings.xml clean package'
              sh 'docker build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .'
              withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) {
                sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
        stage('push latest') {
          steps {
            container('maven') {
              sh 'docker push  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
        stage('deploy to dev') {
          steps {
            kubernetesDeploy(configs: 'deploy/dev-ol/**', enableConfigSubstitution: true, kubeconfigId: "$DEV_KUBECONFIG_CREDENTIAL_ID")
        stage('deploy to staging') {
          steps {
            input(id: 'deploy-to-staging', message: 'deploy to staging?')
            kubernetesDeploy(configs: 'deploy/prod-ol/**', enableConfigSubstitution: true, kubeconfigId: "$TEST_KUBECONFIG_CREDENTIAL_ID")
        stage('deploy to production') {
          steps {
            input(id: 'deploy-to-production', message: 'deploy to production?')
            kubernetesDeploy(configs: 'deploy/prod-ol/**', enableConfigSubstitution: true, kubeconfigId: "$PROD_KUBECONFIG_CREDENTIAL_ID")


    The flag -o in the mvn commands indicates that the offline mode is enabled. If you have relevant maven dependencies and caches ready locally, you can keep the offline mode on to save time.
  5. After the pipeline is created, you can view its stages and steps on the graphical editing panel as well.


Step 7: Run the pipeline and check the results

  1. Click Run to run the pipeline. The pipeline will pause when it reaches the stage deploy to staging as resources have been deployed to the cluster for development. You need to manually click Proceed twice to deploy resources to the testing cluster host and the production cluster shire.


  2. After a while, you can see the pipeline status shown as Success.


  3. Check the pipeline running logs by clicking Show Logs in the upper-right corner. For each stage, you click it to inspect logs, which can be downloaded to your local machine for further analysis.


  4. Once the pipeline runs successfully, click Code Quality to check the results through SonarQube.


  5. Go to the Projects page and you can view the resources deployed in different projects across the clusters by selecting a specific cluster from the drop-down list.




感谢您的反馈。如果您有关于如何使用 KubeSphere 的具体问题,请在 Slack 上提问。如果您想报告问题或提出改进建议,请在 GitHub 存储库中打开问题。