前言
kubeadm是Kubernetes官方推出的快速部署Kubernetes集群工具，其思路是将Kubernetes相关服务容器化(Kubernetes静态Pod)以简化部署。kubeadm当前处于beta阶段，不建议生产环境使用（比如etcd单点）。使用kubeadm部署Kubernetes集群非常简单方便，本文记录了在Red Hat 7上用kubeadm部署一个安全的Kubernetes集群的全过程，CentOS 7部署过程与之类似。
如需kubeadm部署一个高可用的集群，可采用如下方式
kubeadm init --api-advertise-addresses=vip --external-etcd-endpoints=http://x.x.x.x:2379,http://x.x.x.x:2379,http://x.x.x.x:2379 --pod-network-cidr 10.244.0.0/16
注：–api-advertise-addresses 本身支持多个api service ip，但操作kubeadm join加入集群节点会失败, 所以对外服务只配置为一个vip。
部署实例
etcd Version: 3.0.17 kubeadm: v1.7.1 kubernetes: v1.7.1 Flannel: v0.8.0 Docker: 17.03.1-ce
准备工作
1. Red Hat Enterprise Linux Server release 7.1 (Maipo)（ 1GB＋ RAM）
2. 集群机器之间网络互通
目标
1. 部署一个安全的Kubernetes v1.7.1集群
2. 部署pod网络以便pod之间可以互通
部署步骤
安装docker
Kubeadm目前并未在Docker 1.13，17.03+等高级版本进行验证，Kubernetes官方推荐的Docker1.10，1.11，1.12版本。对于Centos，推荐Docker Storage Driver 采用Devicemapper的direct-lvm模式。Docker以及direct-vm参见Docker官网。
Docker安装请参见官网：https://docs.docker.com/engine … ntos/
通过docker version查看docker版本：
sudo docker version Client: Version: 17.03.1-ce API version: 1.27 Go version: go1.7.5 Git commit: c6d412e Built: Mon Mar 27 17:05:44 2017 OS/Arch: linux/amd64 Server: Version: 17.03.1-ce API version: 1.27 (minimum version 1.12) Go version: go1.7.5 Git commit: c6d412e Built: Mon Mar 27 17:05:44 2017 OS/Arch: linux/amd64 Experimental: false
direct-lvm安装详见官网，在此略过： https://docs.docker.com/engine … iver/
... ... Server Version: 17.03.1-ce Storage Driver: devicemapper Pool Name: docker-thinpool Pool Blocksize: 524.3 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: Metadata file: Data Space Used: 1.429 GB Data Space Total: 306 GB Data Space Available: 304.6 GB Metadata Space Used: 782.3 kB Metadata Space Total: 3.217 GB Metadata Space Available: 3.216 GB Thin Pool Minimum Free Space: 30.6 GB Udev Sync Supported: true Deferred Removal Enabled: true Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Library Version: 1.02.135-RHEL7 (2016-11-16) Logging Driver: journald Cgroup Driver: cgroupfs Plugins: Volume: local Network: host null bridge overlay Swarm: inactive Runtimes: runc Default Runtime: runc Security Options: seccomp ... ...
安装kubelet，kubeadm
1. 添加yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF
2. 关闭SELinux
sudo setenforce 0
3. yum安装 kubelet kubeadm
sudo yum install -y kubelet kubeadm
或按需安装指定版本，查看kubeadm, kubelet, kubernets-cni版本：
yum list [kubeadm|kubelet|kubernets-cni] --showduplicates |sort -r
4. 编辑/etc/systemd/system/kubelet.service.d/10-kubeadm.conf，修改 “cgroup-driver”值 由systemd变为cgroupfs
KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs
5. Enable kubelet后启动kubelet
sudo systemctl enable kubelet && sudo systemctl start kubelet
6. 检查kubelet是否启动成功： sudo systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Sat 2017-07-15 01:22:15 UTC; 20min ago Docs: http://kubernetes.io/docs/ Main PID: 16755 (kubelet) Memory: 41.6M CGroup: /system.slice/kubelet.service ├─16755 /usr/bin/kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --ne... └─16891 journalctl -k -f
注：systemctl start kubelet kubelet命令一定要执行，可能会因为如下原因无法启动，不用担心，执行kubeadm init或kubeadm join命令后kubelet会自动被systemd重启成功。
error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory
初始化集群
1. 使用sudo kubeadm init命令初始化集群，可以指定Kubernetes master IP：–apiserver-advertise-address=<ip-address>， 如果选择Flannel做为Pod网络，需指定specify –pod-network-cidr=10.244.0.0/16. 示例如下：
sudo kubeadm init --apiserver-advertise-address 192.168.17.139 --pod-network-cidr 10.244.0.0/16
kubeadm init会做一系列的预检查已确保满足集群部署条件，检查过程大概会执行几分钟。
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters. [init] Using Kubernetes version: v1.7.1 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks [preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.03.1-ce. Max validated version: 1.12 [preflight] WARNING: docker service is not enabled, please run 'systemctl enable docker.service' [preflight] Starting the kubelet service [certificates] Generated CA certificate and key. [certificates] Generated API server certificate and key. [certificates] API Server serving cert is signed for DNS names [bjo-ep-dep-039.dev.fwmrm.net kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.17.139] [certificates] Generated API server kubelet client certificate and key. [certificates] Generated service account token signing key and public key. [certificates] Generated front-proxy CA certificate and key. [certificates] Generated front-proxy client certificate and key. [certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf" [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf" [apiclient] Created API client, waiting for the control plane to become ready [apiclient] All control plane components are healthy after 31.001311 seconds [token] Using token: 472def.6bbb304791b76492 [apiconfig] Created RBAC rules [addons] Applied essential addon: kube-proxy [addons] Applied essential addon: kube-dns Your Kubernetes master has initialized successfully! To start using your cluster, you need to run (as a regular user): mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: http://kubernetes.io/docs/admin/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join --token 472def.6bbb304791b76492 192.168.17.139:6443
Kubernetes根据在/etc/kubernetes/manifests目录下的manifests生成API server, controller manager and scheduler等静态pod。
sudo ls /etc/kubernetes/manifests etcd.yaml kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml
kube-apiserver.yaml文件内容为例，可以查看到启动kube-apiserver需要的参数（可定制，修改该文件即可，kubelet会监控该文件变化，一旦变化会立即重新生成pod）、image、health check探针、QoS等配置，也可以把image提前下载下来加快部署速度。
spec: containers: - command: - kube-apiserver - --experimental-bootstrap-token-auth=true - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --requestheader-allowed-names=front-proxy-client - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key - --secure-port=6443 - --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota - --requestheader-group-headers=X-Remote-Group - --allow-privileged=true - --requestheader-username-headers=X-Remote-User - --requestheader-extra-headers-prefix=X-Remote-Extra- - --service-account-key-file=/etc/kubernetes/pki/sa.pub - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key - --insecure-port=0 - --client-ca-file=/etc/kubernetes/pki/ca.crt - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt - --service-cluster-ip-range=10.96.0.0/12 - --authorization-mode=Node,RBAC - --advertise-address=192.168.17.139 - --etcd-servers=http://127.0.0.1:2379 - --service-node-port-range=20000-65535 image: gcr.io/google_containers/kube-apiserver-amd64:v1.7.1 livenessProbe: failureThreshold: 8 httpGet: host: 127.0.0.1 path: /healthz port: 6443 scheme: HTTPS initialDelaySeconds: 15 timeoutSeconds: 15 name: kube-apiserver resources: requests: cpu: 250m
kube-apiserver yaml文件kube-apiserver.yaml中的选项–insecure-port设置为0，说明kube-apiserver并未监听默认http 8080端口，而是监听了https 6443端口
sudo netstat -nltp | grep 6443 tcp 0 0 0.0.0.0:6443 0.0.0.0:* LISTEN 20936/kube-apiserve
注：从上面信息可知kube-dns已经以pod形式运行但处于pending状态，主要因为pod网络flannel还未部署，另外因下文中的Master Isolation特性导致kube-dns无节点可部署。加入节点以及解除Master Isolation均可以使kube-dns成功运行、处于running状态
2. kubeadm init命令最后生成的“kubeadm join –token 472def.6bbb304791b76492 192.168.17.139:6443”需要记录，用于节点加入集群，token也可通过 sudo kubeadm token list获取。
!769 $ sudo kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION 472def.6bbb304791b76492 <forever> <never> authentication,signing The default bootstrap token generated by 'kubeadm init'.
注：如果之前用’sudo kubeadm init’或’sudo kubeadm join’.已经初始化过集群或加过节点，预检查会失败，需用sudo kubeadm reset命令来revert：
[preflight] Running pre-flight checks [reset] Stopping the kubelet service [reset] Unmounting mounted directories in "/var/lib/kubelet" [reset] Removing kubernetes-managed containers [reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/lib/etcd] [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
3. kubeadm init命令执行完毕后，需执行如下命令（注：mkdir不要加sudo，$HOME不需要做替换变量替换）
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
查看全部namespace下的pod：
kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system etcd-bjo-ep-dep-039.dev.fwmrm.net 1/1 Running 0 2h kube-system kube-apiserver-bjo-ep-dep-039.dev.fwmrm.net 1/1 Running 0 2h kube-system kube-controller-manager-bjo-ep-dep-039.dev.fwmrm.net 1/1 Running 0 2h kube-system kube-dns-2425271678-8k4dn 0/3 Pending 0 2h kube-system kube-proxy-vd39t 1/1 Running 0 2h kube-system kube-scheduler-bjo-ep-dep-039.dev.fwmrm.net 1/1 Running 0 2h
Master Isolation
由于安全原因，默认情况下pod不会被schedule到master节点上，可以通过下面命令解除这种限制：kubectl taint nodes –all node-role.kubernetes.io/master
kubectl taint nodes --all node-role.kubernetes.io/master- node "bjo-ep-dep-039.dev.fwmrm.net" untainted
安装pod网络Flannel
1.通过如下2条命令执安装Flannel：
kubectl apply -f https://raw.githubusercontent. … c.yml
kubectl apply -f https://raw.githubusercontent. … l.yml
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml clusterrole "flannel" created clusterrolebinding "flannel" created
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml serviceaccount "flannel" created configmap "kube-flannel-cfg" created daemonset "kube-flannel-ds" created
2. 验证dns是否工作
创建busybox.yml,内容如下：
apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers: - image: busybox command: - sleep - "3600" imagePullPolicy: IfNotPresent name: busybox restartPolicy: Always
通过kubectl create -f busybox.yaml创建pod，并验证通过kubectl exec -ti busybox — nslookup kubernetes.default验证dns是否工作。
kubectl exec -ti busybox -- nslookup kubernetes.default Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
加入节点到集群
1. 使用前面kubeadm init产生的token加入节点sudo kubeadm join 192.168.17.139:6443 –token 472def.6bbb304791b76492
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters. [preflight] Running pre-flight checks [preflight] WARNING: hostname "" could not be reached [preflight] WARNING: hostname "" lookup : no such host [preflight] Some fatal errors occurred: hostname "" a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*') [preflight] If you know what you are doing, you can skip pre-flight checks with `--skip-preflight-checks`
预检查报了一个Fatal错误，这应该是kubeadm1.7的一个bug，可用–skip-preflight-checks取消预检查
sudo kubeadm join --skip-preflight-checks --token 472def.6bbb304791b76492 192.168.17.139:6443 [kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters. [preflight] Skipping pre-flight checks [discovery] Trying to connect to API Server "192.168.17.139:6443" [discovery] Created cluster-info discovery client, requesting info from "https://192.168.17.139:6443" [discovery] Cluster info signature and contents are valid, will use API Server "https://192.168.17.139:6443" [discovery] Successfully established connection with API Server "192.168.17.139:6443" [bootstrap] Detected server version: v1.7.1 [bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1) [csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request [csr] Received signed certificate from the API server, generating KubeConfig... [kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf" Node join complete: * Certificate signing request sent to master and response received. * Kubelet informed of new secure connection details. Run 'kubectl get nodes' on the master to see this machine join.
注：node上需要安装kubeadm，kubelet安装并启动，否则join提示成功但实际加入失败，前文已有说明。
2. 通过kubectl get nodes查看节点是否成功加入集群
kubectl get nodes NAME STATUS AGE VERSION bjo-ep-dep-039.dev.fwmrm.net Ready 1d v1.7.1 bjo-ep-dep-040.dev.fwmrm.net Ready 19m v1.7.1 bjo-ep-svc-017.dev.fwmrm.net Ready 1h v1.7.1
3. 测试集群是否正常工作
kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes.github.io/master/docs/concepts/workloads/controllers/nginx-deployment.yaml
通过kubectl get po -o wide查看，3个Nginx实例分别部署到3个node上
kubectl get po -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-deployment-431080787-2z167 1/1 Running 0 3m 10.244.0.15 bjo-ep-dep-039.dev.fwmrm.net nginx-deployment-431080787-55fl8 1/1 Running 0 3m 10.16.103.5 bjo-ep-svc-017.dev.fwmrm.net nginx-deployment-431080787-bcmfx 1/1 Running 0 3m 10.16.103.4 bjo-ep-svc-017.dev.fwmrm.net
集群外访问集群
默认情况下因安全原因，集群外部不可以直接操作集群，如需要集群外部操作，可通过如下方式
scp root@<master ip>:/etc/kubernetes/admin.conf . kubectl --kubeconfig ./admin.conf get nodes
示例如下：
xiazhang-mac:~ xiazhang$ kubectl --kubeconfig ./admin.conf -n kube-system get nodes NAME STATUS AGE VERSION bjo-ep-dep-039.dev.fwmrm.net Ready 2d v1.7.1 bjo-ep-dep-040.dev.fwmrm.net Ready 22h v1.7.1 bjo-ep-svc-017.dev.fwmrm.net Ready 23h v1.7.1
或通过kubectl proxy设置代理:
scp root@<master ip>:/etc/kubernetes/admin.conf . kubectl --kubeconfig ./admin.conf proxy
示例如下：
xiazhang-mac:~ xiazhang$ kubectl --kubeconfig ./admin.conf proxy Starting to serve on 127.0.0.1:8001 kubectl config set-cluster default-cluster --server=http://localhost:8001
另开一个Terminal，执行 kubectl get nodes
xiazhang-mac:~ xiazhang$ kubectl get nodes NAME STATUS AGE VERSION bjo-ep-dep-039.dev.fwmrm.net Ready 2d v1.7.1 bjo-ep-dep-040.dev.fwmrm.net Ready 22h v1.7.1 bjo-ep-svc-017.dev.fwmrm.net Ready 23h v1.7.1
Kubernetes Dashboard
1. 使用官方yml文件 https://github.com/kubernetes/dashboard，增加NodePort暴露service端口，本文以31000为例。
kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kube-system spec: type: NodePort ports: - port: 80 targetPort: 9090 nodePort: 31000 selector: k8s-app: kubernetes-dashboard
2. Kubernetes 1.6开始API Server启用了RBAC授权，kubernetes-dashboard.yaml中并未定义需要授权的ServiceAccount，如果访问Dashborad：http://ClusterIP:NodePort会被拒绝。
User "system:serviceaccount:kube-system:default" cannot list statefulsets.apps in the namespace "default". (get statefulsets.apps)
定义dashboard-rbac.yaml，并kubectl create -f dashboard-rbac.yaml
kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: dashboard-admin roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: default namespace: kube-system
3. 部署Heapster in a Kubernetes
首先下载如下目录文件
https://github.com/kubernetes/heapster/blob/master/deploy/kube-config/rbac/heapster-rbac.yaml https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb
拷贝heapster-rbac.yaml 到influxdb文件夹下，然后执行：
kubectl create -f deploy/kube-config/influxdb
注：部署grafana如报如下错误
Starting a utility program that will configure Grafana
Starting Grafana in foreground mode
t=2017-07-17T07:28:47+0000 lvl=crit msg="Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA []interface {}=[])"
可通过替换image解决，编辑deploy/kube-config/influxdb/grafana.yaml
spec: containers: - name: grafana #image: gcr.io/google_containers/heapster-grafana-amd64:v4.2.0 image: gcr.io/google_containers/heapster-grafana-amd64:v4.0.2
4. 访问Dashboard UI: http://192.168.17.139:31000/%2 … D_all
集群部署到此完成，有问题欢迎留言交流。
参考资料
https://kubernetes.io/docs/set … eadm/
https://kubernetes.io/docs/admin/kubeadm/
https://kubernetes.io/docs/set … eadm/
https://kubernetes.io/docs/tas … ectl/
https://stackoverflow.com/ques … board
http://www.cnblogs.com/caiwenhao/p/6196014.html