Cauchy 你好,我现在用的是3.0,具体机器和配置信息如下:
服务器(centos7.6):
master:192.168.3.180(本地测试虚拟机)
node1:192.168.3.181(本地测试虚拟机)
node2:47.116.131.0(阿里云ECS,eth0:172.27.3.152)
配置信息:
spec:
hosts:
- {name: master, address: 192.168.3.180, internalAddress: 192.168.3.180, password: 123456}
- {name: node1, address: 192.168.3.181, internalAddress: 192.168.3.181, password: 123456}
- {name: node2, address: 47.116.131.0, internalAddress: 172.27.3.152, password: 密码}
roleGroups:
etcd:
- master
master:
- master
worker:
- node1
- node2
下面是安装时的输出信息:
`[root@localhost soft]# ./kk create cluster -f config-sample.yaml
+——–+——+——+———+———-+——-+——-+———–+——–+————+————-+——————+————–+
| name | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time |
+——–+——+——+———+———-+——-+——-+———–+——–+————+————-+——————+————–+
| master | y | y | y | y | y | y | y | y | | | | CST 13:29:19 |
| node1 | y | y | y | y | y | y | y | y | | | | CST 13:29:20 |
| node2 | y | y | y | y | y | y | y | y | | | | CST 13:29:20 |
+——–+——+——+———+———-+——-+——-+———–+——–+————+————-+——————+————–+
This is a simple check of your environment.
Before installation, you should ensure that your machines meet all requirements specified at
https://github.com/kubesphere/kubekey#requirements-and-recommendations
Continue this installation? [yes/no]: yes
INFO[13:29:21 CST] Downloading Installation Files
INFO[13:29:21 CST] Downloading kubeadm …
INFO[13:29:59 CST] Downloading kubelet …
INFO[13:31:46 CST] Downloading kubectl …
INFO[13:32:26 CST] Downloading kubecni …
INFO[13:32:59 CST] Downloading helm …
INFO[13:33:36 CST] Configurating operating system …
[node2 47.116.131.0] MSG:
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
kernel.sysrq = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
[node1 192.168.3.181] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
[master 192.168.3.180] MSG:
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_local_reserved_ports = 30000-32767
INFO[13:33:40 CST] Installing docker …
INFO[13:33:42 CST] Start to download images on all nodes
[master] Downloading image: kubesphere/etcd:v3.3.12
[node2] Downloading image: kubesphere/pause:3.2
[node1] Downloading image: kubesphere/pause:3.2
[node1] Downloading image: kubesphere/kube-proxy:v1.18.6
[node2] Downloading image: kubesphere/kube-proxy:v1.18.6
[master] Downloading image: kubesphere/pause:3.2
[node1] Downloading image: coredns/coredns:1.6.9
[node2] Downloading image: coredns/coredns:1.6.9
[master] Downloading image: kubesphere/kube-apiserver:v1.18.6
[node2] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[node1] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[node2] Downloading image: calico/kube-controllers:v3.15.1
[master] Downloading image: kubesphere/kube-controller-manager:v1.18.6
[node1] Downloading image: calico/kube-controllers:v3.15.1
[node2] Downloading image: calico/cni:v3.15.1
[node1] Downloading image: calico/cni:v3.15.1
[node2] Downloading image: calico/node:v3.15.1
[node1] Downloading image: calico/node:v3.15.1
[master] Downloading image: kubesphere/kube-scheduler:v1.18.6
[node2] Downloading image: calico/pod2daemon-flexvol:v3.15.1
[master] Downloading image: kubesphere/kube-proxy:v1.18.6
[node1] Downloading image: calico/pod2daemon-flexvol:v3.15.1
[master] Downloading image: coredns/coredns:1.6.9
[master] Downloading image: kubesphere/k8s-dns-node-cache:1.15.12
[master] Downloading image: calico/kube-controllers:v3.15.1
[master] Downloading image: calico/cni:v3.15.1
[master] Downloading image: calico/node:v3.15.1
[master] Downloading image: calico/pod2daemon-flexvol:v3.15.1
INFO[13:38:20 CST] Generating etcd certs
INFO[13:38:23 CST] Synchronizing etcd certs
INFO[13:38:23 CST] Creating etcd service
[master 192.168.3.180] MSG:
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /etc/systemd/system/etcd.service.
INFO[13:38:27 CST] Starting etcd cluster
[master 192.168.3.180] MSG:
Configuration file will be created
INFO[13:38:28 CST] Refreshing etcd configuration
Waiting for etcd to start
INFO[13:38:35 CST] Get cluster status
[master 192.168.3.180] MSG:
Cluster will be created.
INFO[13:38:36 CST] Installing kube binaries
Push /soft/kubekey/v1.18.6/amd64/kubeadm to 192.168.3.180:/tmp/kubekey/kubeadm Done
Push /soft/kubekey/v1.18.6/amd64/kubeadm to 192.168.3.181:/tmp/kubekey/kubeadm Done
Push /soft/kubekey/v1.18.6/amd64/kubeadm to 47.116.131.0:/tmp/kubekey/kubeadm Done
Push /soft/kubekey/v1.18.6/amd64/kubelet to 192.168.3.180:/tmp/kubekey/kubelet Done
Push /soft/kubekey/v1.18.6/amd64/kubelet to 192.168.3.181:/tmp/kubekey/kubelet Done
Push /soft/kubekey/v1.18.6/amd64/kubectl to 192.168.3.181:/tmp/kubekey/kubectl Done
Push /soft/kubekey/v1.18.6/amd64/kubectl to 192.168.3.180:/tmp/kubekey/kubectl Done
Push /soft/kubekey/v1.18.6/amd64/helm to 192.168.3.180:/tmp/kubekey/helm Done
Push /soft/kubekey/v1.18.6/amd64/helm to 192.168.3.181:/tmp/kubekey/helm Done
Push /soft/kubekey/v1.18.6/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.3.180:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
Push /soft/kubekey/v1.18.6/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 192.168.3.181:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
Push /soft/kubekey/v1.18.6/amd64/kubelet to 47.116.131.0:/tmp/kubekey/kubelet Done
Push /soft/kubekey/v1.18.6/amd64/kubectl to 47.116.131.0:/tmp/kubekey/kubectl Done
Push /soft/kubekey/v1.18.6/amd64/helm to 47.116.131.0:/tmp/kubekey/helm Done
Push /soft/kubekey/v1.18.6/amd64/cni-plugins-linux-amd64-v0.8.6.tgz to 47.116.131.0:/tmp/kubekey/cni-plugins-linux-amd64-v0.8.6.tgz Done
INFO[13:39:17 CST] Initializing kubernetes cluster
[master 192.168.3.180] MSG:
W1119 13:39:18.730215 13449 utils.go:26] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
W1119 13:39:18.730661 13449 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.6
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using ‘kubeadm config images pull’
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder “/etc/kubernetes/pki”
[certs] Generating “ca” certificate and key
[certs] Generating “apiserver” certificate and key
[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local lb.kubesphere.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost lb.kubesphere.local master master.cluster.local node1 node1.cluster.local node2 node2.cluster.local] and IPs [10.233.0.1 192.168.3.180 127.0.0.1 192.168.3.180 192.168.3.181 47.116.131.0 172.27.3.152 10.233.0.1]
[certs] Generating “apiserver-kubelet-client” certificate and key
[certs] Generating “front-proxy-ca” certificate and key
[certs] Generating “front-proxy-client” certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating “sa” key and public key
[kubeconfig] Using kubeconfig folder “/etc/kubernetes”
[kubeconfig] Writing “admin.conf” kubeconfig file
[kubeconfig] Writing “kubelet.conf” kubeconfig file
[kubeconfig] Writing “controller-manager.conf” kubeconfig file
[kubeconfig] Writing “scheduler.conf” kubeconfig file
[control-plane] Using manifest folder “/etc/kubernetes/manifests”
[control-plane] Creating static Pod manifest for “kube-apiserver”
W1119 13:39:27.757376 13449 manifests.go:225] the default kube-apiserver authorization-mode is “Node,RBAC”; using “Node,RBAC”
[control-plane] Creating static Pod manifest for “kube-controller-manager”
W1119 13:39:27.769999 13449 manifests.go:225] the default kube-apiserver authorization-mode is “Node,RBAC”; using “Node,RBAC”
[control-plane] Creating static Pod manifest for “kube-scheduler”
W1119 13:39:27.773560 13449 manifests.go:225] the default kube-apiserver authorization-mode is “Node,RBAC”; using “Node,RBAC”
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory “/etc/kubernetes/manifests”. This can take up to 4m0s
[apiclient] All control plane components are healthy after 25.006952 seconds
[upload-config] Storing the configuration used in ConfigMap “kubeadm-config” in the “kube-system” Namespace
[kubelet] Creating a ConfigMap “kubelet-config-1.18” in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see –upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the label “node-role.kubernetes.io/master=''”
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: mx1pj3.1jsv6dzcvdctqw5g
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the “cluster-info” ConfigMap in the “kube-public” namespace
[kubelet-finalize] Updating “/etc/kubernetes/kubelet.conf” to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join lb.kubesphere.local:6443 –token mx1pj3.1jsv6dzcvdctqw5g \
–discovery-token-ca-cert-hash sha256:4e81db566e8ba4beb238a9b5286e8ad55fa098d3252475e5e877dca52cfa620d \
–control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join lb.kubesphere.local:6443 –token mx1pj3.1jsv6dzcvdctqw5g \
–discovery-token-ca-cert-hash sha256:4e81db566e8ba4beb238a9b5286e8ad55fa098d3252475e5e877dca52cfa620d
[master 192.168.3.180] MSG:
service “kube-dns” deleted
[master 192.168.3.180] MSG:
service/coredns created
[master 192.168.3.180] MSG:
serviceaccount/nodelocaldns created
daemonset.apps/nodelocaldns created
[master 192.168.3.180] MSG:
configmap/nodelocaldns created
[master 192.168.3.180] MSG:
I1119 13:40:21.227231 15324 version.go:252] remote version is much newer: v1.19.4; falling back to: stable-1.18
W1119 13:40:22.519691 15324 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[upload-certs] Storing the certificates in Secret “kubeadm-certs” in the “kube-system” Namespace
[upload-certs] Using certificate key:
baca0cb33d8c260614bf0461c736ef290570b36d2a63343f1363984ed383b408
[master 192.168.3.180] MSG:
secret/kubeadm-certs patched
[master 192.168.3.180] MSG:
secret/kubeadm-certs patched
[master 192.168.3.180] MSG:
secret/kubeadm-certs patched
[master 192.168.3.180] MSG:
W1119 13:40:24.262050 15439 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join lb.kubesphere.local:6443 –token udw90w.jjl7xalokee8nipc –discovery-token-ca-cert-hash sha256:4e81db566e8ba4beb238a9b5286e8ad55fa098d3252475e5e877dca52cfa620d
[master 192.168.3.180] MSG:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master NotReady master 35s v1.18.6 192.168.3.180 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.13
INFO[13:40:25 CST] Deploying network plugin …
[master 192.168.3.180] MSG:
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
INFO[13:40:28 CST] Joining nodes to cluster
[node1 192.168.3.181] MSG:
W1119 13:40:29.980483 12430 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster…
[preflight] FYI: You can look at this config file with ‘kubectl -n kube-system get cm kubeadm-config -oyaml’
W1119 13:40:31.781602 12430 utils.go:26] The recommended value for “clusterDNS” in “KubeletConfiguration” is: [10.233.0.10]; the provided value is: [169.254.25.10]
[kubelet-start] Downloading configuration for the kubelet from the “kubelet-config-1.18” ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file “/var/lib/kubelet/config.yaml”
[kubelet-start] Writing kubelet environment file with flags to file “/var/lib/kubelet/kubeadm-flags.env”
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap…
This node has joined the cluster:
- Certificate signing request was sent to apiserver and a response was received.
- The Kubelet was informed of the new secure connection details.
Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
[node1 192.168.3.181] MSG:
node/node1 labeled
[node2 47.116.131.0] MSG:
[preflight] Running pre-flight checks
W1119 13:45:39.238509 3739 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in “/var/lib/kubelet”
W1119 13:45:39.240998 3739 cleanupnode.go:99] [reset] Failed to evaluate the “/var/lib/kubelet” directory. Skipping its unmount and cleanup: lstat /var/lib/kubelet: no such file or directory
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/dockershim /var/run/kubernetes /var/lib/cni]
The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the “iptables” command.
If your cluster was setup to utilize IPVS, run ipvsadm –clear (or similar)
to reset your system’s IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
[node2 47.116.131.0] MSG:
[preflight] Running pre-flight checks
W1119 13:50:47.274262 3915 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in “/var/lib/kubelet”
W1119 13:50:47.276617 3915 cleanupnode.go:99] [reset] Failed to evaluate the “/var/lib/kubelet” directory. Skipping its unmount and cleanup: lstat /var/lib/kubelet: no such file or directory
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/dockershim /var/run/kubernetes /var/lib/cni]
The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d
The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the “iptables” command.
If your cluster was setup to utilize IPVS, run ipvsadm –clear (or similar)
to reset your system’s IPVS tables.
The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
ERRO[13:55:53 CST] Failed to add worker to cluster: Failed to exec command: sudo -E /bin/sh -c “/usr/local/bin/kubeadm join lb.kubesphere.local:6443 –token udw90w.jjl7xalokee8nipc –discovery-token-ca-cert-hash sha256:4e81db566e8ba4beb238a9b5286e8ad55fa098d3252475e5e877dca52cfa620d”
W1119 13:50:47.423182 3947 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
error execution phase preflight: couldn’t validate the identity of the API Server: Get https://lb.kubesphere.local:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
To see the stack trace of this error execute with –v=5 or higher: Process exited with status 1 node=47.116.131.0
WARN[13:55:53 CST] Task failed …
WARN[13:55:53 CST] error: interrupted by error
Error: Failed to join node: interrupted by error
Usage:
kk create cluster [flags]
Flags:
-f, –filename string Path to a configuration file
-h, –help help for cluster
–skip-pull-images Skip pre pull images
–with-kubernetes string Specify a supported version of kubernetes
–with-kubesphere Deploy a specific version of kubesphere (default v3.0.0)
-y, –yes Skip pre-check of the installation
Global Flags:
–debug Print detailed information (default true)
Failed to join node: interrupted by error
[root@localhost soft]# `