目录

1. k8s集群系统规划

1.1. kubernetes 1.10的依赖

k8s V1.10对一些相关的软件包,如etcd,docker并不是全版本支持或全版本测试,建议的版本如下:

  • docker: 1.11.2 to 1.13.1 and 17.03.x
  • etcd: 3.1.12
  • 全部信息如下:

参考:External Dependencies

  • The supported etcd server version is 3.1.12, as compared to 3.0.17 in v1.9 (#60988)
  • The validated docker versions are the same as for v1.9: 1.11.2 to 1.13.1 and 17.03.x (ref)
  • The Go version is go1.9.3, as compared to go1.9.2 in v1.9. (#59012)
  • The minimum supported go is the same as for v1.9: go1.9.1. (#55301)
  • CNI is the same as v1.9: v0.6.0 (#51250)
  • CSI is updated to 0.2.0 as compared to 0.1.0 in v1.9. (#60736)
  • The dashboard add-on has been updated to v1.8.3, as compared to 1.8.0 in v1.9. (#57326)
  • Heapster has is the same as v1.9: v1.5.0. It will be upgraded in v1.11. (ref)
  • Cluster Autoscaler has been updated to v1.2.0. (#60842, @mwielgus)
  • Updates kube-dns to v1.14.8 (#57918, @rramkumar1)
  • Influxdb is unchanged from v1.9: v1.3.3 (#53319)
  • Grafana is unchanged from v1.9: v4.4.3 (#53319)
  • CAdvisor is v0.29.1 (#60867)
  • fluentd-gcp-scaler is v0.3.0 (#61269)
  • Updated fluentd in fluentd-es-image to fluentd v1.1.0 (#58525, @monotek)
  • fluentd-elasticsearch is v2.0.4 (#58525)
  • Updated fluentd-gcp to v3.0.0. (#60722)
  • Ingress glbc is v1.0.0 (#61302)
  • OIDC authentication is coreos/go-oidc v2 (#58544)
  • Updated fluentd-gcp updated to v2.0.11. (#56927, @x13n)
  • Calico has been updated to v2.6.7 (#59130, @caseydavenport)

1.2 测试服务器准备及环境规划

服务器名IP功 能安装服务
bs-ops-test-docker-dev-01172.21.251.111mastermaster,etcd
bs-ops-test-docker-dev-02172.21.251.112mastermaster,etcd
bs-ops-test-docker-dev-03172.21.251.113nodenode,etcd
bs-ops-test-docker-dev-04172.21.248.242node,私有镜像仓库harbor,node
VIP172.21.255.100master vipnetmask:255.255.240.0

netmask都为:255.255.240.0 所有的测试服务器安装centos linux 7.4最新版本.

2. k8s集群安装前的准备工作

2.1 操作系统环境配置

在所有的服务器上操作:

  • 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
  • 关闭Swap
swapoff -a 
sed -i 's/.*swap.*/#&/' /etc/fstab
  • 关闭SELinux
setenforce  0 
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux 
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config 
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux 
sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config  
  • 开启网桥转发
modprobe br_netfilter
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf
ls /proc/sys/net/bridge
  • 配置K8S的yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
  • 安装依赖包
yum install -y epel-release
yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools conntrack-tools wget vim  ntpdate libseccomp libtool-ltdl 
  • 安装bash命令提示
yum install -y bash-argsparse bash-completion bash-completion-extras
  • 配置ntp
systemctl enable ntpdate.service
echo '*/30 * * * * /usr/sbin/ntpdate time7.aliyun.com >/dev/null 2>&1' > /tmp/crontab2.tmp
crontab /tmp/crontab2.tmp
systemctl start ntpdate.service
  • 调整文件打开数等配置
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nproc 65536"  >> /etc/security/limits.conf
echo "* hard nproc 65536"  >> /etc/security/limits.conf
echo "* soft  memlock  unlimited"  >> /etc/security/limits.conf
echo "* hard memlock  unlimited"  >> /etc/security/limits.conf

2.2 安装docker

在所有node节点上安装docker17.03,注意:k8s v1.10经过测试的最高docker版本为17.03.x,不建议用其它版本。

yum remove docker \
			docker-client \
			docker-client-latest \
			docker-common \
			docker-latest \
			docker-latest-logrotate \
			docker-logrotate \
			docker-selinux \
			docker-engine-selinux \
			docker-engine
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
#yum-config-manager --enable docker-ce-edge
#yum-config-manager --enable docker-ce-test
#docker官方YUM安装有冲突,需要加参数:--setopt=obsoletes=0
yum install -y --setopt=obsoletes=0 docker-ce-17.03.2.ce-1.el7.centos docker-ce-selinux-17.03.2.ce-1.el7.centos
systemctl enable docker
systemctl start docker

vim /usr/lib/systemd/system/docker.service
#找到 ExecStart= 这一行,加上加速地址及私有镜像等,修改如下:
ExecStart=/usr/bin/dockerd  -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --registry-mirror=https://iw5fhz08.mirror.aliyuncs.com  --insecure-registry 172.21.248.242

#重新加载配置
sudo systemctl daemon-reload

#重新启动Docker
sudo systemctl restart docker

2.3 安装harbor私有镜像仓库

以下操作只需要172.21.248.242服务器上执行:

yum -y install docker-compose
wget https://storage.googleapis.com/harbor-releases/release-1.4.0/harbor-offline-installer-v1.4.0.tgz
mkdir /data/
cd /data/
tar xvzf /root/harbor-offline-installer-v1.4.0.tgz
cd harbor
vim harbor.cfg
hostname = 172.21.248.242

sh install.sh

#如果要修改配置:
docker-compose down -v
vim harbor.cfg
prepare
docker-compose up -d

#重启:
docker-compose stop
docker-compose start
  • harbor的管理界面为:http://172.21.248.242 用户名:admin 密码:Harbor12345

  • 上传下拉镜像测试: 在其它机器上:

docker login 172.21.248.242

输入用户名和密码即可登入。 – 从网方镜像库下载一个centos的镜像并推送到私有仓库:

docker pull centos
docker tag centos:latest 172.21.248.242/library/centos:latest
docker push 172.21.248.242/library/centos:latest

在web界面上就可以看到镜像了 – 拉镜像:

docker pull 172.21.248.242/library/centos:latest

3. 安装及配置etcd

以下操作在bs-ops-test-docker-dev-01上执行即可:

3.1 创建etcd证书

3.1.1 设置cfssl环境

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
export PATH=/usr/local/bin:$PATH

3.1.2 创建 CA 配置文件

下面证书内配置的IP为etcd节点的IP,为方便以后扩容,可考虑多配置几个预留IP地址:

mkdir /root/ssl
cd /root/ssl
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "8760h"
      }
    }
  }
}
EOF

cat > ca-csr.json <<EOF
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "shanghai",
    "L": "shanghai",
    "O": "k8s",
    "OU": "System"
  }]
}
EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@bs-ops-test-docker-dev-01 ssl]# ll
total 20
-rw-r--r-- 1 root root  290 Apr 29 19:46 ca-config.json
-rw-r--r-- 1 root root 1005 Apr 29 19:49 ca.csr
-rw-r--r-- 1 root root  190 Apr 29 19:48 ca-csr.json
-rw------- 1 root root 1675 Apr 29 19:49 ca-key.pem
-rw-r--r-- 1 root root 1363 Apr 29 19:49 ca.pem

cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "172.21.251.111",
    "172.21.251.112",
    "172.21.251.113",
    "172.21.255.100",
    "172.21.248.242"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shanghai",
      "L": "shanghai",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF

cfssl gencert -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
  
[root@bs-ops-test-docker-dev-01 ssl]# ll
total 36
-rw-r--r-- 1 root root  290 Apr 29 19:46 ca-config.json
-rw-r--r-- 1 root root 1005 Apr 29 19:49 ca.csr
-rw-r--r-- 1 root root  190 Apr 29 19:48 ca-csr.json
-rw------- 1 root root 1675 Apr 29 19:49 ca-key.pem
-rw-r--r-- 1 root root 1363 Apr 29 19:49 ca.pem
-rw-r--r-- 1 root root 1082 Apr 29 19:53 etcd.csr
-rw-r--r-- 1 root root  348 Apr 29 19:53 etcd-csr.json
-rw------- 1 root root 1675 Apr 29 19:53 etcd-key.pem
-rw-r--r-- 1 root root 1456 Apr 29 19:53 etcd.pem

3.1.3 把证书放到3个etcd节点上

mkdir -p /etc/etcd/ssl
cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/
ssh -n bs-ops-test-docker-dev-02 "mkdir -p /etc/etcd/ssl && exit"
ssh -n bs-ops-test-docker-dev-03 "mkdir -p /etc/etcd/ssl && exit"
scp -r /etc/etcd/ssl/*.pem bs-ops-test-docker-dev-02:/etc/etcd/ssl/
scp -r /etc/etcd/ssl/*.pem bs-ops-test-docker-dev-03:/etc/etcd/ssl/

3.2 安装etcd

k8s v1.10测试的是etcd 3.1.12的版本,所以我们也使用这个版本,发现这个版本在YUM源上没有,只能下载二进制包安装了。

wget https://github.com/coreos/etcd/releases/download/v3.1.12/etcd-v3.1.12-linux-amd64.tar.gz
tar -xvf etcd-v3.1.12-linux-amd64.tar.gz
mv etcd-v3.1.12-linux-amd64/etcd* /usr/local/bin

#创建 etcd 的 systemd unit 文件。
cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd \\
  --name \${ETCD_NAME} \\
  --cert-file=\${ETCD_CERT_FILE} \\
  --key-file=\${ETCD_KEY_FILE} \\
  --peer-cert-file=\${ETCD_PEER_CERT_FILE} \\
  --peer-key-file=\${ETCD_PEER_KEY_FILE} \\
  --trusted-ca-file=\${ETCD_TRUSTED_CA_FILE} \\
  --peer-trusted-ca-file=\${ETCD_PEER_TRUSTED_CA_FILE} \\
  --initial-advertise-peer-urls \${ETCD_INITIAL_ADVERTISE_PEER_URLS} \\
  --listen-peer-urls \${ETCD_LISTEN_PEER_URLS} \\
  --listen-client-urls \${ETCD_LISTEN_CLIENT_URLS} \\
  --advertise-client-urls \${ETCD_ADVERTISE_CLIENT_URLS} \\
  --initial-cluster-token \${ETCD_INITIAL_CLUSTER_TOKEN} \\
  --initial-cluster \${ETCD_INITIAL_CLUSTER} \\
  --initial-cluster-state new \\
  --data-dir=\${ETCD_DATA_DIR}
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

#etcd 配置文件,这是172.21.251.111节点的配置,他两个etcd节点只要将上面的IP地址改成相应节点的IP地址即可。
#ETCD_NAME换成对应节点的infra1/2/3。
cat > /etc/etcd/etcd.conf <<EOF
ETCD_DATA_DIR="/var/lib/etcd/"
ETCD_LISTEN_PEER_URLS="https://172.21.251.111:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.21.251.111:2379,http://127.0.0.1:2379"
ETCD_NAME="infra1"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.21.251.111:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.21.251.111:2379"
ETCD_INITIAL_CLUSTER="infra1=https://172.21.251.111:2380,infra2=https://172.21.251.112:2380,infra3=https://172.21.251.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.pem"
EOF

mkdir /var/lib/etcd/
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

在三个etcd节点的服务都起来后,执行一下命令检查状态,如果像以下输出,说明配置成功。

etcdctl \
--ca-file=/etc/etcd/ssl/ca.pem \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
cluster-health
2018-04-29 23:55:29.773343 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
2018-04-29 23:55:29.774558 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
member 6c87a871854f88b is healthy: got healthy result from https://172.21.251.112:2379
member c5d2ced3329b39da is healthy: got healthy result from https://172.21.251.113:2379
member e6e88b355a379c5f is healthy: got healthy result from https://172.21.251.111:2379
cluster is healthy

4. 安装及配置keepalived

4.1 安装keepalived

在master节点bs-ops-test-docker-dev-01,bs-ops-test-docker-dev-02上安装keepalived:

yum install -y keepalived
systemctl enable keepalived

4.2 配置keepalived

  • 在bs-ops-test-docker-dev-01上生成配置文件
cat <<EOF > /etc/keepalived/keepalived.conf
global_defs {
   router_id LVS_k8s
}

vrrp_script CheckK8sMaster {
    script "curl -k https://172.21.255.100:6443"
    interval 3
    timeout 9
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 61
    priority 100
    advert_int 1
    mcast_src_ip 172.21.251.111
    nopreempt
    authentication {
        auth_type PASS
        auth_pass kwi28x7sx37uw72
    }
    unicast_peer {
        172.21.251.112
    }
    virtual_ipaddress {
        172.21.255.100/20
    }
    track_script {
        CheckK8sMaster
    }

}
EOF
  • 在bs-ops-test-docker-dev-02上生成配置文件
cat <<EOF > /etc/keepalived/keepalived.conf
global_defs {
   router_id LVS_k8s
}

vrrp_script CheckK8sMaster {
    script "curl -k https://172.21.255.100:6443"
    interval 3
    timeout 9
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 61
    priority 90
    advert_int 1
    mcast_src_ip 172.21.251.112
    nopreempt
    authentication {
        auth_type PASS
        auth_pass kwi28x7sx37uw72
    }
    unicast_peer {
        172.21.251.111
    }
    virtual_ipaddress {
        172.21.255.100/20
    }
    track_script {
        CheckK8sMaster
    }

}
EOF

4.3 启动验证keepalived

systemctl restart keepalived

[root@bs-ops-test-docker-dev-01 ~]# ip addr
1: lo: <loopback ,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <broadcast ,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 06:dd:4a:00:0a:70 brd ff:ff:ff:ff:ff:ff
    inet 172.21.251.111/20 brd 172.21.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.21.255.100/20 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::83a4:1f7:15d3:49eb/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::851c:69f8:270d:b510/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
    inet6 fe80::a8c8:d941:f3c5:c36a/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever
3: docker0: <no -CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:b8:b4:a0:b1 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever

可以看到VIP 172.21.255.100在bs-ops-test-docker-dev-01上。

5. 使用kubeadm安装及配置k8s集群

5.1 下载官方镜像

由于官方镜像在国内访问非常慢或不能访问,所以先把镜像拉到私有仓库:

docker pull anjia0532/pause-amd64:3.1
docker pull anjia0532/kube-apiserver-amd64:v1.10.2
docker pull anjia0532/kube-proxy-amd64:v1.10.2
docker pull anjia0532/kube-scheduler-amd64:v1.10.2
docker pull anjia0532/kube-controller-manager-amd64:v1.10.2
docker pull anjia0532/k8s-dns-sidecar-amd64:1.14.8
docker pull anjia0532/k8s-dns-kube-dns-amd64:1.14.8
docker pull anjia0532/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker pull anjia0532/kubernetes-dashboard-amd64:v1.8.3
docker pull anjia0532/heapster-influxdb-amd64:v1.3.3
docker pull anjia0532/heapster-grafana-amd64:v4.4.3
docker pull anjia0532/heapster-amd64:v1.5.2
docker pull anjia0532/etcd-amd64:3.1.12
docker pull anjia0532/fluentd-elasticsearch:v2.0.4
docker pull anjia0532/cluster-autoscaler:v1.2.0

docker tag anjia0532/pause-amd64:3.1 172.21.248.242/k8s/pause-amd64:3.1
docker tag anjia0532/kube-apiserver-amd64:v1.10.2 172.21.248.242/k8s/kube-apiserver-amd64:v1.10.2
docker tag anjia0532/kube-proxy-amd64:v1.10.2 172.21.248.242/k8s/kube-proxy-amd64:v1.10.2
docker tag anjia0532/kube-scheduler-amd64:v1.10.2 172.21.248.242/k8s/kube-scheduler-amd64:v1.10.2
docker tag anjia0532/kube-controller-manager-amd64:v1.10.2 172.21.248.242/k8s/kube-controller-manager-amd64:v1.10.2
docker tag anjia0532/k8s-dns-sidecar-amd64:1.14.8 172.21.248.242/k8s/k8s-dns-sidecar-amd64:1.14.8
docker tag anjia0532/k8s-dns-kube-dns-amd64:1.14.8 172.21.248.242/k8s/k8s-dns-kube-dns-amd64:1.14.8
docker tag anjia0532/k8s-dns-dnsmasq-nanny-amd64:1.14.8 172.21.248.242/k8s/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker tag anjia0532/kubernetes-dashboard-amd64:v1.8.3 172.21.248.242/k8s/kubernetes-dashboard-amd64:v1.8.3
docker tag anjia0532/heapster-influxdb-amd64:v1.3.3 172.21.248.242/k8s/heapster-influxdb-amd64:v1.3.3
docker tag anjia0532/heapster-grafana-amd64:v4.4.3 172.21.248.242/k8s/heapster-grafana-amd64:v4.4.3
docker tag anjia0532/heapster-amd64:v1.5.2 172.21.248.242/k8s/heapster-amd64:v1.5.2
docker tag anjia0532/etcd-amd64:3.1.12 172.21.248.242/k8s/etcd-amd64:3.1.12
docker tag anjia0532/fluentd-elasticsearch:v2.0.4 172.21.248.242/k8s/fluentd-elasticsearch:v2.0.4
docker tag anjia0532/cluster-autoscaler:v1.2.0 172.21.248.242/k8s/cluster-autoscaler:v1.2.0

docker push 172.21.248.242/k8s/pause-amd64:3.1
docker push 172.21.248.242/k8s/kube-apiserver-amd64:v1.10.2
docker push 172.21.248.242/k8s/kube-proxy-amd64:v1.10.2
docker push 172.21.248.242/k8s/kube-scheduler-amd64:v1.10.2
docker push 172.21.248.242/k8s/kube-controller-manager-amd64:v1.10.2
docker push 172.21.248.242/k8s/k8s-dns-sidecar-amd64:1.14.8
docker push 172.21.248.242/k8s/k8s-dns-kube-dns-amd64:1.14.8
docker push 172.21.248.242/k8s/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker push 172.21.248.242/k8s/kubernetes-dashboard-amd64:v1.8.3
docker push 172.21.248.242/k8s/heapster-influxdb-amd64:v1.3.3
docker push 172.21.248.242/k8s/heapster-grafana-amd64:v4.4.3
docker push 172.21.248.242/k8s/heapster-amd64:v1.5.2
docker push 172.21.248.242/k8s/etcd-amd64:3.1.12
docker push 172.21.248.242/k8s/fluentd-elasticsearch:v2.0.4
docker push 172.21.248.242/k8s/cluster-autoscaler:v1.2.0

5.2 安装kubelet kubeadm kubectl

在所有节点上安装:

yum install -y kubelet kubeadm kubectl
systemctl enable kubelet

所有节点修改kubelet配置文件

#修改这一行
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
#添加这一行
Environment="KUBELET_EXTRA_ARGS=--v=2 --fail-swap-on=false --pod-infra-container-image=172.21.248.242/k8s/pause-amd64:3.1"

让配置生效:

systemctl daemon-reload
systemctl enable kubelet

启用命令行自动补全:

yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source < (kubectl completion bash)
source <(kubeadm completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
echo "source < (kubeadm completion bash)" >> ~/.bashrc 

5.3 通过kubeadm配置集群

5.3.1 生成kubeadm配置文件

在master节点bs-ops-test-docker-dev-01,bs-ops-test-docker-dev-02上生成kubeadm配置文件:

cat <<EOF > config.yaml 
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:
  endpoints:
  - https://172.21.251.111:2379
  - https://172.21.251.112:2379
  - https://172.21.251.113:2379
  caFile: /etc/etcd/ssl/ca.pem
  certFile: /etc/etcd/ssl/etcd.pem
  keyFile: /etc/etcd/ssl/etcd-key.pem
  dataDir: /var/lib/etcd
networking:
  podSubnet: 10.244.0.0/16
kubernetesVersion: 1.10.2
api:
  advertiseAddress: "172.21.255.100"
token: "465wc3.r1u0vv92d3slu4tf"
tokenTTL: "0s"
apiServerCertSANs:
- bs-ops-test-docker-dev-01
- bs-ops-test-docker-dev-02
- bs-ops-test-docker-dev-03
- bs-ops-test-docker-dev-04
- 172.21.251.111
- 172.21.251.112
- 172.21.251.113
- 172.21.248.242
- 172.21.255.100
featureGates:
  CoreDNS: true
imageRepository: "172.21.248.242/k8s"
EOF

[root@bs-ops-test-docker-dev-01 ~]# kubeadm init --help | grep service
      --service-cidr string                  Use alternative range of IP address for service VIPs. (default "10.96.0.0/12")
      --service-dns-domain string            Use alternative domain for services, e.g. "myorg.internal". (default "cluster.local")

注意: 通过上面的kubeadm init –help,可以看到kubectl服务的网段是10.96.0.0/12,所以在/etc/systemd/system/kubelet.service.d/10-kubeadm.conf内的cluster ip为:10.96.0.10。 这与我们配置的podSubnet: 10.244.0.0/16并不冲突,一个是用户使用的,一个是k8s服务使用的。

5.3.2 在bs-ops-test-docker-dev-01上初始化集群

[root@bs-ops-test-docker-dev-01 ~]# kubeadm init --config config.yaml
[init] Using Kubernetes version: v1.10.2
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [bs-ops-test-docker-dev-01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local bs-ops-test-docker-dev-01 bs-ops-test-docker-dev-02 bs-ops-test-docker-dev-03 bs-ops-test-docker-dev-04] and IPs [10.96.0.1 172.21.255.100 172.21.251.111 172.21.251.112 172.21.251.113 172.21.248.242 172.21.255.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [bs-ops-test-docker-dev-01] and IPs [172.21.255.100]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 20.501769 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node bs-ops-test-docker-dev-01 as master by adding a label and a taint
[markmaster] Master bs-ops-test-docker-dev-01 tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 465wc3.r1u0vv92d3slu4tf
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 172.21.255.100:6443 --token 465wc3.r1u0vv92d3slu4tf --discovery-token-ca-cert-hash sha256:234294087f9598267de8f1b5f484c56463ae74d56c9484ecee3e6b6aa0f85fe1

按照提示,执行以下命令后,kubectl就可以直接运行了。

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

注意这个时候,node的状态会一直是NotReady,coredns也起不来,这是正常的,因为网络插件还没装。

[root@bs-ops-test-docker-dev-01 ~]# kubectl   get node -o wide
NAME                        STATUS     ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
bs-ops-test-docker-dev-01   NotReady   master    17m       v1.10.2   <none>        CentOS Linux 7 (Core)   3.10.0-693.21.1.el7.x86_64   docker://17.3.2

[root@bs-ops-test-docker-dev-01 ~]# kubectl   get pods --all-namespaces -o wide
NAMESPACE     NAME                                                READY     STATUS    RESTARTS   AGE       IP               NODE
kube-system   coredns-7997f8864c-mc899                            0/1       Pending   0          21m       </none><none>           </none><none>
kube-system   coredns-7997f8864c-xgtj9                            0/1       Pending   0          21m       </none><none>           </none><none>
kube-system   kube-apiserver-bs-ops-test-docker-dev-01            1/1       Running   0          21m       172.21.251.111   bs-ops-test-docker-dev-01
kube-system   kube-controller-manager-bs-ops-test-docker-dev-01   1/1       Running   0          22m       172.21.251.111   bs-ops-test-docker-dev-01
kube-system   kube-proxy-c9tcq                                    1/1       Running   0          21m       172.21.251.111   bs-ops-test-docker-dev-01
kube-system   kube-scheduler-bs-ops-test-docker-dev-01            1/1       Running   0          20m       172.21.251.111   bs-ops-test-docker-dev-01

如果有问题,可以通过kubeadm reset重置到初始状态,然后再重新初始化

[root@bs-ops-test-docker-dev-01 ~]# kubeadm reset
[preflight] Running pre-flight checks.
[reset] Stopping the kubelet service.
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers.
[reset] No etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd.
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

# kubeadm reset并不会初始化etcd上的数据信息,需要我们手动清一下,不清会有很多坑
systemctl stop etcd && rm -rf /var/lib/etcd/* && systemctl start etcd

[root@bs-ops-test-docker-dev-01 ~]# kubeadm init --config config.yaml
[init] Using Kubernetes version: v1.10.2
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [bs-ops-test-docker-dev-01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local bs-ops-test-docker-dev-01 bs-ops-test-docker-dev-02 bs-ops-test-docker-dev-03 bs-ops-test-docker-dev-04] and IPs [10.96.0.1 172.21.255.100 172.21.251.111 172.21.251.112 172.21.251.113 172.21.248.242 172.21.255.100]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost] and IPs [127.0.0.1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [bs-ops-test-docker-dev-01] and IPs [172.21.255.100]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 29.001939 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node bs-ops-test-docker-dev-01 as master by adding a label and a taint
[markmaster] Master bs-ops-test-docker-dev-01 tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 465wc3.r1u0vv92d3slu4tf
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 172.21.255.100:6443 --token 465wc3.r1u0vv92d3slu4tf --discovery-token-ca-cert-hash sha256:bbaea5bd68a1ba5002cf745aa21f1f42f0cef1d04ec2f10203500e4924932706

5.3.3 部署flannel

只需要bs-ops-test-docker-dev-01执行:

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
#版本信息:quay.io/coreos/flannel:v0.10.0-amd64
kubectl create -f  kube-flannel.yml

过个一两分钟后,我们可以看到节点和pods都起来了,在我们安装网络插件flannel之前,coredns是起不来的状态

[root@bs-ops-test-docker-dev-01 ~]# kubectl get node
NAME                        STATUS    ROLES     AGE       VERSION
bs-ops-test-docker-dev-01   Ready     master    6m        v1.10.2
[root@bs-ops-test-docker-dev-01 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                                READY     STATUS    RESTARTS   AGE
kube-system   coredns-7997f8864c-5wczb                            1/1       Running   0          5m
kube-system   coredns-7997f8864c-tvqrt                            1/1       Running   0          5m
kube-system   kube-apiserver-bs-ops-test-docker-dev-01            1/1       Running   0          5m
kube-system   kube-controller-manager-bs-ops-test-docker-dev-01   1/1       Running   0          5m
kube-system   kube-flannel-ds-n259k                               1/1       Running   0          2m
kube-system   kube-proxy-4cvhl                                    1/1       Running   0          5m
kube-system   kube-scheduler-bs-ops-test-docker-dev-01            1/1       Running   0          5m

5.3.3 第二台master节点初始化

先把刚刚在master节点上生成的证书分发到其它的master节点,node节点不需要:

scp -r /etc/kubernetes/pki  bs-ops-test-docker-dev-02:/etc/kubernetes/

在第二台master节点上初始化(文件内的token可使用’kubeadm token generate’生成):

cat <<EOF > config.yaml
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
etcd:
  endpoints:
  - https://172.21.251.111:2379
  - https://172.21.251.112:2379
  - https://172.21.251.113:2379
  caFile: /etc/etcd/ssl/ca.pem
  certFile: /etc/etcd/ssl/etcd.pem
  keyFile: /etc/etcd/ssl/etcd-key.pem
  dataDir: /var/lib/etcd
networking:
  podSubnet: 10.244.0.0/16
kubernetesVersion: 1.10.2
api:
  advertiseAddress: "172.21.255.100"
token: "465wc3.r1u0vv92d3slu4tf"
tokenTTL: "0s"
apiServerCertSANs:
- bs-ops-test-docker-dev-01
- bs-ops-test-docker-dev-02
- bs-ops-test-docker-dev-03
- bs-ops-test-docker-dev-04
- 172.21.251.111
- 172.21.251.112
- 172.21.251.113
- 172.21.248.242
- 172.21.255.100
featureGates:
  CoreDNS: true
imageRepository: "172.21.248.242/k8s"
EOF

[root@bs-ops-test-docker-dev-02 ~]# kubeadm init --config config.yaml
[init] Using Kubernetes version: v1.10.2
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[preflight] Starting the kubelet service
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Using the existing apiserver-kubelet-client certificate and key.
[certificates] Using the existing etcd/ca certificate and key.
[certificates] Using the existing etcd/server certificate and key.
[certificates] Using the existing etcd/peer certificate and key.
[certificates] Using the existing etcd/healthcheck-client certificate and key.
[certificates] Using the existing apiserver-etcd-client certificate and key.
[certificates] Using the existing sa key.
[certificates] Using the existing front-proxy-ca certificate and key.
[certificates] Using the existing front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 0.013231 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node bs-ops-test-docker-dev-02 as master by adding a label and a taint
[markmaster] Master bs-ops-test-docker-dev-02 tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 465wc3.r1u0vv92d3slu4tf
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 172.21.255.100:6443 --token 465wc3.r1u0vv92d3slu4tf --discovery-token-ca-cert-hash sha256:bbaea5bd68a1ba5002cf745aa21f1f42f0cef1d04ec2f10203500e4924932706

我们可以发现生成的hash值是一样的。

现在查看k8s状态,可以看到有2个master节点了,而除coredns外,所有的容器比刚刚都多了一个,coredns本来就有2个。

[root@bs-ops-test-docker-dev-02 ~]# kubectl get node
NAME                        STATUS    ROLES     AGE       VERSION
bs-ops-test-docker-dev-01   Ready     master    36m       v1.10.2
bs-ops-test-docker-dev-02   Ready     master    2m        v1.10.2
[root@bs-ops-test-docker-dev-02 ~]# kubectl get all --all-namespaces
NAMESPACE     NAME                                                    READY     STATUS    RESTARTS   AGE
kube-system   pod/coredns-7997f8864c-5wczb                            1/1       Running   0          36m
kube-system   pod/coredns-7997f8864c-tvqrt                            1/1       Running   0          36m
kube-system   pod/kube-apiserver-bs-ops-test-docker-dev-01            1/1       Running   0          35m
kube-system   pod/kube-apiserver-bs-ops-test-docker-dev-02            1/1       Running   0          2m
kube-system   pod/kube-controller-manager-bs-ops-test-docker-dev-01   1/1       Running   0          35m
kube-system   pod/kube-controller-manager-bs-ops-test-docker-dev-02   1/1       Running   0          2m
kube-system   pod/kube-flannel-ds-n259k                               1/1       Running   0          32m
kube-system   pod/kube-flannel-ds-r2b7k                               1/1       Running   0          2m
kube-system   pod/kube-proxy-4cvhl                                    1/1       Running   0          36m
kube-system   pod/kube-proxy-zmgzm                                    1/1       Running   0          2m
kube-system   pod/kube-scheduler-bs-ops-test-docker-dev-01            1/1       Running   0          35m
kube-system   pod/kube-scheduler-bs-ops-test-docker-dev-02            1/1       Running   0          2m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes   ClusterIP   10.96.0.1    </none><none>        443/TCP         36m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   </none><none>        53/UDP,53/TCP   36m

NAMESPACE     NAME                             DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE
kube-system   daemonset.apps/kube-flannel-ds   2         2         2         2            2           beta.kubernetes.io/arch=amd64   32m
kube-system   daemonset.apps/kube-proxy        2         2         2         2            2           </none><none>                          36m

NAMESPACE     NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2         2         2            2           36m

NAMESPACE     NAME                                 DESIRED   CURRENT   READY     AGE
kube-system   replicaset.apps/coredns-7997f8864c   2         2         2         36m

默认master不运行pod,通过以下命令可让master也运行pod,生产环境不建议这样使用。

[root@bs-ops-test-docker-dev-02 ~]# kubectl taint nodes --all node-role.kubernetes.io/master-
node "bs-ops-test-docker-dev-01" untainted
node "bs-ops-test-docker-dev-02" untainted

5.3.4 node节点加入集群

在bs-ops-test-docker-dev-03 node节点上,执行以下命令加入集群:

[root@bs-ops-test-docker-dev-03 ~]# kubeadm join 172.21.255.100:6443 --token 465wc3.r1u0vv92d3slu4tf --discovery-token-ca-cert-hash sha256:bbaea5bd68a1ba5002cf745aa21f1f42f0cef1d04ec2f10203500e4924932706
[preflight] Running pre-flight checks.
        [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "172.21.255.100:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://172.21.255.100:6443"
[discovery] Requesting info from "https://172.21.255.100:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.21.255.100:6443"
[discovery] Successfully established connection with API Server "172.21.255.100:6443"

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

[root@bs-ops-test-docker-dev-02 ~]# kubectl get node -o wide
NAME                        STATUS    ROLES     AGE       VERSION   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
bs-ops-test-docker-dev-01   Ready     master    48m       v1.10.2   </none><none>        CentOS Linux 7 (Core)   3.10.0-693.21.1.el7.x86_64   docker://17.3.2
bs-ops-test-docker-dev-02   Ready     master    13m       v1.10.2   </none><none>        CentOS Linux 7 (Core)   3.10.0-693.21.1.el7.x86_64   docker://17.3.2
bs-ops-test-docker-dev-03   Ready     </none><none>    1m        v1.10.2   </none><none>        CentOS Linux 7 (Core)   3.10.0-693.21.1.el7.x86_64   docker://17.3.2

可以看到bs-ops-test-docker-dev-03节点已经加入到集群内了,很方便。

6. 其它kubernetes-dashboard等插件的安装及配置

6.1 kubernetes-dashboard

kubernetes 1.10测试过的版本为:v1.8.3

wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
kubectl create -f kubernetes-dashboard.yaml

访问方式为:https://172.21.255.100:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/,打开后就报错了。

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "services \"https:kubernetes-dashboard:\" is forbidden: User \"system:anonymous\" cannot get services/proxy in the namespace \"kube-system\"",
  "reason": "Forbidden",
  "details": {
    "name": "https:kubernetes-dashboard:",
    "kind": "services"
  },
  "code": 403
}

用以下命令生成证书,并导入到浏览器上。

openssl pkcs12 -export -in apiserver-kubelet-client.crt  -out apiserver-kubelet-client.p12 -inkey apiserver-kubelet-client.key

但是打开还是会报错:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "no endpoints available for service \"https:kubernetes-dashboard:\"",
  "reason": "ServiceUnavailable",
  "code": 503
}

这个错误后面一查是由于容器挂了导致的,挂的原因是我把kubernetes-dashboard.yaml内的api-server那行取消注释了。再次注释掉那行,重新创建一下就OK了。

这次终于看到了页面:

下面是找到token的方法:

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

使用token登入后,发现页面上也有报错:

解决办法:

# 创建一个ServiceAccount
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
EOF

#再创建一个ClusterRoleBinding,给admin-user相应的权限
cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
EOF

# 查找admin-user的token
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

用admin-user的token登入,发现这一次没有报错,完美,如图:

6.2 heapster安装配置

wget https://github.com/kubernetes/heapster/archive/v1.5.2.tar.gz
tar xvzf v1.5.2.tar.gz
cd heapster-1.5.2/

部署文件目录:heapster-1.5.2/deploy/kube-config/influxdb 修改配置文件内的image为本地。 heapster写好了一个kube.sh脚本来导入配置,另外记得要导入rbac/heapster-rbac.yaml,否则heapster会没有权限:

cd /root/heapster/heapster-1.5.2/deploy
[root@bs-ops-test-docker-dev-01 deploy]# pwd
/root/heapster/heapster-1.5.2/deploy
[root@bs-ops-test-docker-dev-01 deploy]# ll
total 12
drwxrwxr-x 2 root root 4096 Mar 20 17:10 docker
drwxrwxr-x 8 root root 4096 Mar 20 17:10 kube-config
-rwxrwxr-x 1 root root  869 Mar 20 17:10 kube.sh
[root@bs-ops-test-docker-dev-01 deploy]# sh kube.sh
Usage: kube.sh {start|stop|restart}
[root@bs-ops-test-docker-dev-01 deploy]# sh kube.sh start
heapster pods have been setup
[root@bs-ops-test-docker-dev-01 deploy]# cd kube-config
[root@bs-ops-test-docker-dev-01 kube-config]# kubectl create -f rbac/heapster-rbac.yaml 
clusterrolebinding.rbac.authorization.k8s.io "heapster" created
[root@bs-ops-test-docker-dev-01 deploy]# kubectl get all -n=kube-system 
NAME                                                    READY     STATUS    RESTARTS   AGE
pod/coredns-7997f8864c-5wczb                            1/1       Running   0          6d
pod/coredns-7997f8864c-tvqrt                            1/1       Running   0          6d
pod/heapster-855c88b45-npft9                            1/1       Running   0          1m
pod/kube-apiserver-bs-ops-test-docker-dev-01            1/1       Running   0          6d
pod/kube-apiserver-bs-ops-test-docker-dev-02            1/1       Running   0          6d
pod/kube-controller-manager-bs-ops-test-docker-dev-01   1/1       Running   0          6d
pod/kube-controller-manager-bs-ops-test-docker-dev-02   1/1       Running   0          6d
pod/kube-flannel-ds-b2gss                               1/1       Running   0          6d
pod/kube-flannel-ds-n259k                               1/1       Running   0          6d
pod/kube-flannel-ds-r2b7k                               1/1       Running   0          6d
pod/kube-proxy-4cvhl                                    1/1       Running   0          6d
pod/kube-proxy-z6dkq                                    1/1       Running   0          6d
pod/kube-proxy-zmgzm                                    1/1       Running   0          6d
pod/kube-scheduler-bs-ops-test-docker-dev-01            1/1       Running   0          6d
pod/kube-scheduler-bs-ops-test-docker-dev-02            1/1       Running   0          6d
pod/kubernetes-dashboard-85d6464777-cr6sz               1/1       Running   0          6d
pod/monitoring-grafana-5c5b5656d4-stvkh                 1/1       Running   0          1m
pod/monitoring-influxdb-5f7597c95d-5g7g5                1/1       Running   0          1m

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
service/heapster               ClusterIP   10.100.16.218    <none>        80/TCP          1m
service/kube-dns               ClusterIP   10.96.0.10       </none><none>        53/UDP,53/TCP   6d
service/kubernetes-dashboard   ClusterIP   10.101.92.10     </none><none>        443/TCP         6d
service/monitoring-grafana     ClusterIP   10.100.199.102   </none><none>        80/TCP          1m
service/monitoring-influxdb    ClusterIP   10.107.211.129   </none><none>        8086/TCP        1m

NAME                             DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE
daemonset.apps/kube-flannel-ds   3         3         3         3            3           beta.kubernetes.io/arch=amd64   6d
daemonset.apps/kube-proxy        3         3         3         3            3           </none><none>                          6d

NAME                                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns                2         2         2            2           6d
deployment.apps/heapster               1         1         1            1           1m
deployment.apps/kubernetes-dashboard   1         1         1            1           6d
deployment.apps/monitoring-grafana     1         1         1            1           1m
deployment.apps/monitoring-influxdb    1         1         1            1           1m

NAME                                              DESIRED   CURRENT   READY     AGE
replicaset.apps/coredns-7997f8864c                2         2         2         6d
replicaset.apps/heapster-855c88b45                1         1         1         1m
replicaset.apps/kubernetes-dashboard-85d6464777   1         1         1         6d
replicaset.apps/monitoring-grafana-5c5b5656d4     1         1         1         1m
replicaset.apps/monitoring-influxdb-5f7597c95d    1         1         1         1m

效果图如下,可以看到比上面的图多了CPU及内存等监控信息: 访问 grafana:

[root@bs-ops-test-docker-dev-01 kube-config]# kubectl cluster-info
Kubernetes master is running at https://172.21.255.100:6443
Heapster is running at https://172.21.255.100:6443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://172.21.255.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
monitoring-grafana is running at https://172.21.255.100:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
monitoring-influxdb is running at https://172.21.255.100:6443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy

用浏览器打开:https://172.21.255.100:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy,可看到下图: 里面有K8s集群以及各个pod的监控信息.

好了,k8s的安装及配置就写到这里,后面准备对k8s进行增删节点,自动缩扩容等方面进行测试。