通过coredns跨k8s集群解析DNS

发表于2021年12月13日由阿辉

最近在弄AlertManager,发现其集群模式需要知道各个AlertManager节点的IP地址，而我们现在需要在2个K8s集群同时部署AlertManager上来创建一套AlertManager，以确保在某一个K8s集群挂了后报警还能发出来，同时确保当2个K8s集群都正常的时候，一个告警只发一次出来。

但怎么在A集群上找到B集群上的POD的IP，这是个问题。通过研究，我们发现CoreDns可以很好的解决这个问题。

假设我们现在有2套集群，2套集群使用的DNS后缀都是cluster.local，容器网段及service网段都不一样。

我们现在在A集群上，先看旧的CoreDNS的配置：

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        log . {combined} {
          class error
        }
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        reload
        loadbalance
        #rewrite name saas-fe-zhan.internal.com public-fe-zhan-node-dev.public-fe-node-dev.svc.cluster.local
    }

apiVersion: v1

kind: ConfigMap

metadata:

namespace: kube-system

data:

Corefile: |

.:53 {

log . {combined} {

class error

}

errors

health

ready

kubernetes cluster.local in-addr.arpa ip6.arpa {

pods insecure

fallthrough in-addr.arpa ip6.arpa

}

prometheus :9153

forward . /etc/resolv.conf

cache 30

reload

loadbalance

#rewrite name saas-fe-zhan.internal.com public-fe-zhan-node-dev.public-fe-node-dev.svc.cluster.local

}

继续阅读 →

linux kernel 4.8以上内核NR_KERNEL_STACK的改变

发表于2021年3月31日由阿辉

1. NR_KERNEL_STACK是干什么用的？

通过下面的命令，我们可以查看内核栈的数量：

cat /proc/vmstat  | grep stack
nr_kernel_stack 567

cat /proc/vmstat | grep stack

nr_kernel_stack 567

这个值在内核中的宏为：NR_KERNEL_STACK，表示当前内核中有多少个内核栈。

我们使用这个数字来监控K8S节点上使用PID的数量，以避免PID被耗尽。

2. 那么PID与内核栈有什么关系呢？

在linux系统中，每个进程、子进程和线程在运行时都会有一个PID。这些进程或线程在运行时，因为CPU需要进行任务切换，在任务切换时就需要上下文交换，在上下文交换时就需要把当前进程的上下文压到内核栈内去，以便下次再运行时取出继续执行。

所以可以确定：每个进程、子进程和线程都会有一个内核栈。内核栈的数量与PID的数量大致相当。

注：基于linux内核的线程，比如java的线程与linux的线程是一一对应的，nodejs只使用了linux的进程，线程模型是其自己实现的，golang最特别，使用了多进程，每个进程上有多线程（基于内核），线程上还是自己实现的协程或者说goroute(可以理解为自己实现的线程)

继续阅读 →

PVC在K8S集群迁移或恢复需注意地方

发表于2021年3月29日由阿辉

目前在做同城双活，每个机房都有一个k8s集群，其中一些共享卷需要做到双活的2个集群上都可用。

比如我们现在在主机房的集群上有一个PVC：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.cfs
  creationTimestamp: "2020-11-25T07:24:15Z"
  name: public-devops-teststatefulset-nginx-online-pvc
  namespace: test
  uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 30Gi
  storageClassName: cfs-hp
  volumeMode: Filesystem
  volumeName: pvc-3d65a636-f4c3-4c49-b957-c45673f48bc6

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

annotations:

pv.kubernetes.io/bind-completed: "yes"

pv.kubernetes.io/bound-by-controller: "yes"

volume.beta.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.cfs

creationTimestamp: "2020-11-25T07:24:15Z"

namespace: test

uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6

spec:

accessModes:

- ReadWriteMany

resources:

requests:

storage: 30Gi

storageClassName: cfs-hp

volumeMode: Filesystem

volumeName: pvc-3d65a636-f4c3-4c49-b957-c45673f48bc6

以及对应的PV：

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: com.tencent.cloud.csi.cfs
  creationTimestamp: "2020-11-25T07:24:22Z"
  name: pvc-3d65a636-f4c3-4c49-b957-c45673f48bc6
  uid: 301c903e-10fc-40ee-98e1-0655948425fc
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 30Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: public-devops-teststatefulset-nginx-online-pvc
    namespace: test
    resourceVersion: "657951844"
    uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6
  csi:
    driver: com.tencent.cloud.csi.cfs
    fsType: ext4
    volumeAttributes:
      fsid: cfs-3c7e956f6
      host: 10.10.144.12
      storage.kubernetes.io/csiProvisionerIdentity: 1597949457398-8081-com.tencent.cloud.csi.cfs
      storagetype: HP
      subnetid: subnet-mg9pp5o5
      vpcid: vpc-aqpp2l40
      zone: ap-shanghai-4
    volumeHandle: cfs-3c7e956f6
  persistentVolumeReclaimPolicy: Delete
  storageClassName: cfs-hp
  volumeMode: Filesystem

apiVersion: v1

kind: PersistentVolume

metadata:

annotations:

pv.kubernetes.io/provisioned-by: com.tencent.cloud.csi.cfs

creationTimestamp: "2020-11-25T07:24:22Z"

uid: 301c903e-10fc-40ee-98e1-0655948425fc

spec:

accessModes:

- ReadWriteMany

capacity:

storage: 30Gi

claimRef:

apiVersion: v1

kind: PersistentVolumeClaim

namespace: test

resourceVersion: "657951844"

uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6

csi:

driver: com.tencent.cloud.csi.cfs

fsType: ext4

volumeAttributes:

fsid: cfs-3c7e956f6

host: 10.10.144.12

storage.kubernetes.io/csiProvisionerIdentity: 1597949457398-8081-com.tencent.cloud.csi.cfs

storagetype: HP

subnetid: subnet-mg9pp5o5

vpcid: vpc-aqpp2l40

zone: ap-shanghai-4

volumeHandle: cfs-3c7e956f6

persistentVolumeReclaimPolicy: Delete

storageClassName: cfs-hp

volumeMode: Filesystem

继续阅读 →

istio 1.5版本升级

发表于2020年5月9日由阿辉

之前测试的istio是1.5.1，现在升级到1.5.2。可以参考之前的istio定制安装的文章。http://www.huilog.com/?p=1299

下载新版本，并准备好之前安装时的配置文件

wget https://github.com/istio/istio/releases/download/1.5.2/istio-1.5.2-linux.tar.gz
tar xvzf istio-1.5.2-linux.tar.gz 
cd istio-1.5.1/
cp -rf *.yaml ../istio-1.5.2/
cd ../istio-1.5.2/

cp bin/istioctl /usr/local/bin/
cp tools/istioctl.bash /usr/local/bin/

wget https://github.com/istio/istio/releases/download/1.5.2/istio-1.5.2-linux.tar.gz

tar xvzf istio-1.5.2-linux.tar.gz

cd istio-1.5.1/

cp -rf *.yaml ../istio-1.5.2/

cd ../istio-1.5.2/

cp bin/istioctl /usr/local/bin/

cp tools/istioctl.bash /usr/local/bin/

查看支持的版本列表，验证 istoctl 命令是否支持从当前版本升级

[root@sh-saas-k8s1-master-dev-01 istio-1.5.2]# istioctl manifest versions

Operator version is 1.5.2.

The following installation package versions are recommended for use with this version of the operator:
  1.5.0

The following installation package versions are supported for upgrade by this version of the operator:
  >=1.4.0
   <1.6

[root@sh-saas-k8s1-master-dev-01 istio-1.5.2]# istioctl manifest versions

Operator version is 1.5.2.

The following installation package versions are recommended for use with this version of the operator:

1.5.0

The following installation package versions are supported for upgrade by this version of the operator:

>=1.4.0

<1.6

继续阅读 →

istio-init容器Init:CrashLoopBackOff故障解决

发表于2020年5月8日由阿辉

最近在测试istio时，经常发现注入过sidecar的pod过段时间就变成了Init:CrashLoopBackOff状态。如：

[root@sh-saas-k8s1-master-dev-01 ~]# kubectl get pod --all-namespaces -o wide | grep  'Init'
public-ops-tomcat-dev           public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-64qwr              0/2     Init:CrashLoopBackOff   7          21h     10.253.3.166   10.12.97.23   <none>           <none>

[root@sh-saas-k8s1-master-dev-01 ~]# kubectl get pod --all-namespaces -o wide | grep 'Init'

public-ops-tomcat-dev public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-64qwr 0/2 Init:CrashLoopBackOff 7 21h 10.253.3.166 10.12.97.23 <none> <none>

我的kubernetes版本为1.14.10，istio版本为：1.5.1
查看istio-init容器的日志，发现有如下的报错：

[root@sh-saas-k8s1-master-dev-01 ~]# kubectl logs -n public-ops-tomcat-dev           public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-64qwr istio-init 
Environment:
------------
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false

Writing following contents to rules file:  /tmp/iptables-rules-1588923880490327697.txt562915423
* nat
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_INBOUND
-N ISTIO_OUTPUT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT

iptables-restore --noflush /tmp/iptables-rules-1588923880490327697.txt562915423
iptables-restore: line 2 failed
iptables-save 
# Generated by iptables-save v1.6.1 on Fri May  8 07:44:40 2020
*mangle
:PREROUTING ACCEPT [643414:2344563772]
:INPUT ACCEPT [643414:2344563772]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [616124:4267707048]
:POSTROUTING ACCEPT [616124:4267707048]
COMMIT
# Completed on Fri May  8 07:44:40 2020
# Generated by iptables-save v1.6.1 on Fri May  8 07:44:40 2020
*raw
:PREROUTING ACCEPT [643414:2344563772]
:OUTPUT ACCEPT [616124:4267707048]
COMMIT
# Completed on Fri May  8 07:44:40 2020
# Generated by iptables-save v1.6.1 on Fri May  8 07:44:40 2020
*nat
:PREROUTING ACCEPT [38474:2000648]
:INPUT ACCEPT [40999:2131948]
:OUTPUT ACCEPT [7987:560379]
:POSTROUTING ACCEPT [8763:600731]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Fri May  8 07:44:40 2020
# Generated by iptables-save v1.6.1 on Fri May  8 07:44:40 2020
*filter
:INPUT ACCEPT [643414:2344563772]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [616124:4267707048]
COMMIT
# Completed on Fri May  8 07:44:40 2020
panic: exit status 1

goroutine 1 [running]:
istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0xd819c0, 0x9739b8, 0x10, 0xc00000cbc0, 0x2, 0x2)
        istio.io/istio@/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc000109d30, 0x7faeecd9a001, 0x0, 0x0)
        istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:474 +0x3aa
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc000109d30)
        istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:481 +0x45
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc000109d30)
        istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:428 +0x24e2
istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0xd5c740, 0xc0000ee700, 0x0, 0x10)
        istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:56 +0x14e
github.com/spf13/cobra.(*Command).execute(0xd5c740, 0xc00001e130, 0x10, 0x11, 0xd5c740, 0xc00001e130)
        github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xd5c740, 0x40574f, 0xc00009e058, 0x0)
        github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v0.0.5/command.go:864
istio.io/istio/tools/istio-iptables/pkg/cmd.Execute()
        istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:284 +0x2d
main.main()
        istio.io/istio@/tools/istio-iptables/main.go:22 +0x20

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

[root@sh-saas-k8s1-master-dev-01 ~]# kubectl logs -n public-ops-tomcat-dev public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-64qwr istio-init

Environment:

------------

ENVOY_PORT=

INBOUND_CAPTURE_PORT=

ISTIO_INBOUND_INTERCEPTION_MODE=

ISTIO_INBOUND_TPROXY_MARK=

ISTIO_INBOUND_TPROXY_ROUTE_TABLE=

ISTIO_INBOUND_PORTS=

ISTIO_LOCAL_EXCLUDE_PORTS=

ISTIO_SERVICE_CIDR=

ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:

----------

PROXY_PORT=15001

PROXY_INBOUND_CAPTURE_PORT=15006

PROXY_UID=1337

PROXY_GID=1337

INBOUND_INTERCEPTION_MODE=REDIRECT

INBOUND_TPROXY_MARK=1337

INBOUND_TPROXY_ROUTE_TABLE=133

INBOUND_PORTS_INCLUDE=*

INBOUND_PORTS_EXCLUDE=15090,15020

OUTBOUND_IP_RANGES_INCLUDE=*

OUTBOUND_IP_RANGES_EXCLUDE=

OUTBOUND_PORTS_EXCLUDE=

KUBEVIRT_INTERFACES=

ENABLE_INBOUND_IPV6=false

Writing following contents to rules file: /tmp/iptables-rules-1588923880490327697.txt562915423

* nat

-N ISTIO_REDIRECT

-N ISTIO_IN_REDIRECT

-N ISTIO_INBOUND

-N ISTIO_OUTPUT

-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port 15001

-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port 15006

-A PREROUTING -p tcp -j ISTIO_INBOUND

-A ISTIO_INBOUND -p tcp --dport 22 -j RETURN

-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN

-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN

-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT

-A OUTPUT -p tcp -j ISTIO_OUTPUT

-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN

-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT

-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT

-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN

-A ISTIO_OUTPUT -j ISTIO_REDIRECT

COMMIT

iptables-restore --noflush /tmp/iptables-rules-1588923880490327697.txt562915423

iptables-restore: line 2 failed

iptables-save

# Generated by iptables-save v1.6.1 on Fri May 8 07:44:40 2020

*mangle

:PREROUTING ACCEPT [643414:2344563772]

:INPUT ACCEPT [643414:2344563772]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [616124:4267707048]

:POSTROUTING ACCEPT [616124:4267707048]

COMMIT

# Completed on Fri May 8 07:44:40 2020

# Generated by iptables-save v1.6.1 on Fri May 8 07:44:40 2020

*raw

:PREROUTING ACCEPT [643414:2344563772]

:OUTPUT ACCEPT [616124:4267707048]

COMMIT

# Completed on Fri May 8 07:44:40 2020

# Generated by iptables-save v1.6.1 on Fri May 8 07:44:40 2020

*nat

:PREROUTING ACCEPT [38474:2000648]

:INPUT ACCEPT [40999:2131948]

:OUTPUT ACCEPT [7987:560379]

:POSTROUTING ACCEPT [8763:600731]

:ISTIO_INBOUND - [0:0]

:ISTIO_IN_REDIRECT - [0:0]

:ISTIO_OUTPUT - [0:0]

:ISTIO_REDIRECT - [0:0]

-A PREROUTING -p tcp -j ISTIO_INBOUND

-A OUTPUT -p tcp -j ISTIO_OUTPUT

-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN

-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN

-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN

-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT

-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006

-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN

-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT

-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN

-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT

-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN

-A ISTIO_OUTPUT -j ISTIO_REDIRECT

-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001

COMMIT

# Completed on Fri May 8 07:44:40 2020

# Generated by iptables-save v1.6.1 on Fri May 8 07:44:40 2020

*filter

:INPUT ACCEPT [643414:2344563772]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [616124:4267707048]

COMMIT

# Completed on Fri May 8 07:44:40 2020

panic: exit status 1

goroutine 1 [running]:

istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0xd819c0, 0x9739b8, 0x10, 0xc00000cbc0, 0x2, 0x2)

istio.io/istio@/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96

istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc000109d30, 0x7faeecd9a001, 0x0, 0x0)

istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:474 +0x3aa

istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc000109d30)

istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:481 +0x45

istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc000109d30)

istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:428 +0x24e2

istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0xd5c740, 0xc0000ee700, 0x0, 0x10)

istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:56 +0x14e

github.com/spf13/cobra.(*Command).execute(0xd5c740, 0xc00001e130, 0x10, 0x11, 0xd5c740, 0xc00001e130)

github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa

github.com/spf13/cobra.(*Command).ExecuteC(0xd5c740, 0x40574f, 0xc00009e058, 0x0)

github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb

github.com/spf13/cobra.(*Command).Execute(...)

github.com/spf13/cobra@v0.0.5/command.go:864

istio.io/istio/tools/istio-iptables/pkg/cmd.Execute()

istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:284 +0x2d

main.main()

istio.io/istio@/tools/istio-iptables/main.go:22 +0x20

继续阅读 →

通过docker overlay2 目录名查找容器名

发表于2019年7月18日由阿辉

有时候经常会有个别容器占用磁盘空间特别大，这个时候就需要通过docker overlay2 目录名查找容器名：

先进入overlay2的目录，这个目录可以通过docker的配置文件（/etc/docker/daemon.json）内找到。然后看看谁占用空间比较多。

[root@sh-saas-k8s1-node-qa-04 overlay2]# du -sc * | sort -rn  | more
33109420        total
1138888 20049e2e445181fc742b9e74a8819edf0e7ee8f0c0041fb2d1c9d321f73d8f5b
1066548 010d0a26a1fe5b00e330d0d87649fc73af45f9333fd824bf0f9d91a37276af18
943208  030c0f111675f6ed534eaa6e4183ec91d4c065dd4bdb5a289f4b572357667378
825116  0ad9e737795dd367bb72f7735fb69a65db3d8907305b305ec21232505241d044
824756  bf3c698966bc19318f3263631bc285bde07c6a1a4eaea25c4ecd3b7b8f29b3fd
661000  15763b72802e1e71cc943e09cba8b747779bf80fa35d56318cf1b89f7b1f1e71
575564  02eaa52e2f999dc387a9dee543028bada0762022cef1400596b5cc18a6223635
486780  4353c30611d7f51932d9af24bb1330db0fdb86faa9d9cae02ed618fa975c697a
486420  562a8874cc345b8ea830c1486c42211b288c886c5dca08e14d7057cacab984c1
486420  4f897e8cd355320b0e7ee1ecc9db5c43d5151f9afa29f1925fe264c88429be4c
448652  a8d0596d123fcc59983ce63df3f3acd40d5c930ed72874ce0a9efbc3234466de
448296  851cc4912edb9989e120acf241f26c82e9715e7fcb1d2bd19a002fcfb894f1f4
417780  20608baacae6bafcd4230a18a272857bc75703a6eefef5c9b40ba4ea19496b11
387388  43a8a76de3b5531e4c12f956f7bfcfcdb8fd38548faf20812cafa9a39813abc5

[root@sh-saas-k8s1-node-qa-04 overlay2]# du -sc * | sort -rn | more

33109420 total

1138888 20049e2e445181fc742b9e74a8819edf0e7ee8f0c0041fb2d1c9d321f73d8f5b

1066548 010d0a26a1fe5b00e330d0d87649fc73af45f9333fd824bf0f9d91a37276af18

943208 030c0f111675f6ed534eaa6e4183ec91d4c065dd4bdb5a289f4b572357667378

825116 0ad9e737795dd367bb72f7735fb69a65db3d8907305b305ec21232505241d044

824756 bf3c698966bc19318f3263631bc285bde07c6a1a4eaea25c4ecd3b7b8f29b3fd

661000 15763b72802e1e71cc943e09cba8b747779bf80fa35d56318cf1b89f7b1f1e71

575564 02eaa52e2f999dc387a9dee543028bada0762022cef1400596b5cc18a6223635

486780 4353c30611d7f51932d9af24bb1330db0fdb86faa9d9cae02ed618fa975c697a

486420 562a8874cc345b8ea830c1486c42211b288c886c5dca08e14d7057cacab984c1

486420 4f897e8cd355320b0e7ee1ecc9db5c43d5151f9afa29f1925fe264c88429be4c

448652 a8d0596d123fcc59983ce63df3f3acd40d5c930ed72874ce0a9efbc3234466de

448296 851cc4912edb9989e120acf241f26c82e9715e7fcb1d2bd19a002fcfb894f1f4

417780 20608baacae6bafcd4230a18a272857bc75703a6eefef5c9b40ba4ea19496b11

387388 43a8a76de3b5531e4c12f956f7bfcfcdb8fd38548faf20812cafa9a39813abc5

再通过目录名查找容器名：

[root@sh-saas-k8s1-node-qa-04 overlay2]#  docker ps -q | xargs docker inspect --format '{{.State.Pid}}, {{.Name}}, {{.GraphDriver.Data.WorkDir}}' | grep "20049e2e445181fc742b9e74a8819edf0e7ee8f0c0041fb2d1c9d321f73d8f5b"
4884, /k8s_taskmanager_flink-taskmanager-java-qa-7879d55f45-xbd74_public-tjpm-flink-java-qa_ad8bf915-a23f-11e9-be66-52540088db9a_0, /data/kubernetes/docker/overlay2/20049e2e445181fc742b9e74a8819edf0e7ee8f0c0041fb2d1c9d321f73d8f5b/work

[root@sh-saas-k8s1-node-qa-04 overlay2]# docker ps -q | xargs docker inspect --format '{{.State.Pid}}, {{.Name}}, {{.GraphDriver.Data.WorkDir}}' | grep "20049e2e445181fc742b9e74a8819edf0e7ee8f0c0041fb2d1c9d321f73d8f5b"

4884, /k8s_taskmanager_flink-taskmanager-java-qa-7879d55f45-xbd74_public-tjpm-flink-java-qa_ad8bf915-a23f-11e9-be66-52540088db9a_0, /data/kubernetes/docker/overlay2/20049e2e445181fc742b9e74a8819edf0e7ee8f0c0041fb2d1c9d321f73d8f5b/work

如果发现有目录查不到，通常是因为容器已经被删掉了，目录没有清理，这时直接清理便可：

docker system prune -a -f

1 2	docker system prune -a -f

k8s创建集群只读service account

发表于2019年6月4日由阿辉

有时需要在k8s 集群上给比如开发人员创建一个只读的service account，在这里记录一下创建方法：

先创建oms-viewonly.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: oms-viewonly
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - persistentvolumeclaims
  - pods
  - replicationcontrollers
  - replicationcontrollers/scale
  - serviceaccounts
  - services
  - nodes
  - persistentvolumeclaims
  - persistentvolumes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - bindings
  - events
  - limitranges
  - namespaces/status
  - pods/log
  - pods/status
  - replicationcontrollers/status
  - resourcequotas
  - resourcequotas/status
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - daemonsets
  - deployments
  - deployments/scale
  - replicasets
  - replicasets/scale
  - statefulsets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - daemonsets
  - deployments
  - deployments/scale
  - ingresses
  - networkpolicies
  - replicasets
  - replicasets/scale
  - replicationcontrollers/scale
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  - volumeattachments
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterrolebindings
  - clusterroles
  - roles
  - rolebindings
  verbs:
  - get
  - list
  - watch

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: oms-read 
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: oms-read
  labels: 
    k8s-app: oms-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: oms-viewonly
subjects:
- kind: ServiceAccount
  name: oms-read
  namespace: kube-system

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

rules:

- apiGroups:

- ""

resources:

- configmaps

- endpoints

- persistentvolumeclaims

- pods

- replicationcontrollers

- replicationcontrollers/scale

- serviceaccounts

- services

- nodes

- persistentvolumeclaims

- persistentvolumes

verbs:

- get

- list

- watch

- apiGroups:

- ""

resources:

- bindings

- events

- limitranges

- namespaces/status

- pods/log

- pods/status

- replicationcontrollers/status

- resourcequotas

- resourcequotas/status

verbs:

- get

- list

- watch

- apiGroups:

- ""

resources:

- namespaces

verbs:

- get

- list

- watch

- apiGroups:

- apps

resources:

- daemonsets

- deployments

- deployments/scale

- replicasets

- replicasets/scale

- statefulsets

verbs:

- get

- list

- watch

- apiGroups:

- autoscaling

resources:

- horizontalpodautoscalers

verbs:

- get

- list

- watch

- apiGroups:

- batch

resources:

- cronjobs

- jobs

verbs:

- get

- list

- watch

- apiGroups:

- extensions

resources:

- daemonsets

- deployments

- deployments/scale

- ingresses

- networkpolicies

- replicasets

- replicasets/scale

- replicationcontrollers/scale

verbs:

- get

- list

- watch

- apiGroups:

- policy

resources:

- poddisruptionbudgets

verbs:

- get

- list

- watch

- apiGroups:

- networking.k8s.io

resources:

- networkpolicies

verbs:

- get

- list

- watch

- apiGroups:

- storage.k8s.io

resources:

- storageclasses

- volumeattachments

verbs:

- get

- list

- watch

- apiGroups:

- rbac.authorization.k8s.io

resources:

- clusterrolebindings

- clusterroles

- roles

- rolebindings

verbs:

- get

- list

- watch

---

apiVersion: v1

kind: ServiceAccount

metadata:

namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

labels:

k8s-app: oms-read

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

subjects:

- kind: ServiceAccount

namespace: kube-system

然后创建：
kubectl apply -f oms-viewonly.yaml

最后就可以使用以下命令查找刚刚创建SA的token:
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep oms-read | awk '{print $1}')

通过zabbix监控kubernetes集群

发表于2019年2月13日由阿辉

日前写了一个zabbix的监控脚本来监控kubernetes集群，主要用于报警的功能。性能监控还是使用其它方式来实现。

github URL

https://github.com/farmerluo/k8s_zabbix

k8s_zabbix说明

k8s_zabbix实现了使用zabbix监控kubernetes的ingress,hpa,pod状态等功能。

Template Check K8S Cluster Status.xml：zabbix模板，可通过此文件导入到zabbix

check_k8s_status.py：kubernetes的监控脚本

userparameter_k8s.conf：zabbix agent端的配置文件，需要注意脚本的路径

check_k8s_status.py说明

监控的k8s集群配置：

conf.host = "https://10.10.88.20:8443"
conf.api_key['authorization'] = "xxxxxx.xxxxxxx.x-x-xxx-xxx-x"

conf.host = "https://10.10.88.20:8443"

conf.api_key['authorization'] = "xxxxxx.xxxxxxx.x-x-xxx-xxx-x"

脚本会监控traefik ingress的访问状态，将对400~599的非正常状态进行报警，需事先将traefik的访问日志通过fluentd或filebeat等导入到elasticsearch集群，脚本将定时通过查询访问日志来监控ingress的访问状态。监控脚本内的配置：

# elasticsearch server config
es_server = [{"host": "10.16.252.50", "port": 9200},
             {"host": "10.16.252.50", "port": 9200},
             {"host": "10.16.252.50", "port": 9200}
             ]
# 索引名
es_index = "logstash-traefik-ingress-lb-*"
# es查询间隔,ms
es_query_duration = 60000

# 状态码报警及阈值配置
# xxx.com为自定域名的例子
status_code_config = {
    'default': {'403': '90', '404': '90', '500': '2', '502': '2', '499': '70', '406': '70', '503': '5',
                '504': '5', '599': '2', 'other': '30', '429': '5', '430': '1'
                },
    'xxx.com': {'403': '100', '404': '100', '500': '70', '502': '70', '499': '100', '406': '80', '503': '70',
                '504': '70', '599': '0', 'other': '60', '429': '5', '430': '1'
                }
}

# elasticsearch server config

es_server = [{"host": "10.16.252.50", "port": 9200},

{"host": "10.16.252.50", "port": 9200},

{"host": "10.16.252.50", "port": 9200}

]

# 索引名

es_index = "logstash-traefik-ingress-lb-*"

# es查询间隔,ms

es_query_duration = 60000

# 状态码报警及阈值配置

# xxx.com为自定域名的例子

status_code_config = {

'default': {'403': '90', '404': '90', '500': '2', '502': '2', '499': '70', '406': '70', '503': '5',

'504': '5', '599': '2', 'other': '30', '429': '5', '430': '1'

'xxx.com': {'403': '100', '404': '100', '500': '70', '502': '70', '499': '100', '406': '80', '503': '70',

'504': '70', '599': '0', 'other': '60', '429': '5', '430': '1'

}

腾讯云环境K8S安装及配置测试

发表于2018年7月5日由阿辉

1. k8s集群系统规划

1.1. kubernetes 1.10的依赖

k8s V1.10对一些相关的软件包，如etcd,docker并不是全版本支持或全版本测试，建议的版本如下：

docker: 1.11.2 to 1.13.1 and 17.03.x
etcd: 3.1.12
全部信息如下：

参考：External Dependencies
* The supported etcd server version is 3.1.12, as compared to 3.0.17 in v1.9 (#60988)
* The validated docker versions are the same as for v1.9: 1.11.2 to 1.13.1 and 17.03.x (ref)
* The Go version is go1.9.3, as compared to go1.9.2 in v1.9. (#59012)
* The minimum supported go is the same as for v1.9: go1.9.1. (#55301)
* CNI is the same as v1.9: v0.6.0 (#51250)
* CSI is updated to 0.2.0 as compared to 0.1.0 in v1.9. (#60736)
* The dashboard add-on has been updated to v1.8.3, as compared to 1.8.0 in v1.9. (#57326)
* Heapster has is the same as v1.9: v1.5.0. It will be upgraded in v1.11. (ref)
* Cluster Autoscaler has been updated to v1.2.0. (#60842, @mwielgus)
* Updates kube-dns to v1.14.8 (#57918, @rramkumar1)
* Influxdb is unchanged from v1.9: v1.3.3 (#53319)
* Grafana is unchanged from v1.9: v4.4.3 (#53319)
* CAdvisor is v0.29.1 (#60867)
* fluentd-gcp-scaler is v0.3.0 (#61269)
* Updated fluentd in fluentd-es-image to fluentd v1.1.0 (#58525, @monotek)
* fluentd-elasticsearch is v2.0.4 (#58525)
* Updated fluentd-gcp to v3.0.0. (#60722)
* Ingress glbc is v1.0.0 (#61302)
* OIDC authentication is coreos/go-oidc v2 (#58544)
* Updated fluentd-gcp updated to v2.0.11. (#56927, @x13n)
* Calico has been updated to v2.6.7 (#59130, @caseydavenport)
*

1.2 测试服务器准备及环境规划

服务器名	IP	功能	安装服务
sh-saas-cvmk8s-master-01	10.12.96.3	master	master,etcd
sh-saas-cvmk8s-master-02	10.12.96.5	master	master,etcd
sh-saas-cvmk8s-master-03	10.12.96.13	master	master,etcd
sh-saas-cvmk8s-node-01	10.12.96.2	node	node
sh-saas-cvmk8s-node-02	10.12.96.4	node	node
sh-saas-cvmk8s-node-03	10.12.96.6	node	node
bs-ops-test-docker-dev-04	172.21.248.242	私有镜像仓库	harbor
VIP	10.12.96.100	master vip	netmask:255.255.255.0

netmask都为:255.255.255.0

所有的测试服务器安装centos linux 7.4最新版本.

VIP：10.12.96.100只是用于keepalived的测试，实际本文使用的是腾讯云LB+haproxy的模式，使用的腾讯云LB VIP为：10.12.16.101

容器网段：10.254.0.0/16
容器网段需要避免这些冲突:

同vpc的其它集群的集群网络cidr
所在vpc的cidr
所在vpc的子网路由的cidr
route-ctl list 能看到的所有route table 的 cidr
容器网段不要在VPC内创建，也要不在VPC的路由表内，使用一个VPC内不存在的网络。

k8s service cluster网络：10.254.255.0/24

继续阅读 →

kubernetes 日常使用及操作测试

发表于2018年5月21日由阿辉

前文已经安装好了一套kubernetes 1.10，下面我们来进行日常使用测试

1. 创建部署及服务

编辑一个yaml文件：

vim nginx.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-router
  namespace: test
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx-router
    spec:
      containers:
      - name: nginx-router
        image: 172.21.248.242/base/nginx
        ports:
        - containerPort: 80

---
kind: Service
apiVersion: v1
metadata:
  name: nginx-router
  namespace: test
spec:
  selector:
    app: nginx-router
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-router-ingress
  namespace: test
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: "nginx.k8s.dev.huilog.com"
    http:
      paths:
      - backend:
          serviceName: nginx-router
          servicePort: 80

# 创建部署及服务：
[root@bs-ops-test-docker-dev-01 dev]# kubectl create -f nginx.yaml 
deployment.extensions "nginx-router" created
service "nginx-router" created
ingress.extensions "nginx-router-ingress" created
[root@bs-ops-test-docker-dev-01 dev]#

vim nginx.yaml

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

namespace: test

spec:

replicas: 2

template:

metadata:

labels:

app: nginx-router

spec:

containers:

- name: nginx-router

image: 172.21.248.242/base/nginx

ports:

- containerPort: 80

---

kind: Service

apiVersion: v1

metadata:

namespace: test

spec:

selector:

app: nginx-router

ports:

- protocol: TCP

port: 80

targetPort: 80

---

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

namespace: test

annotations:

kubernetes.io/ingress.class: traefik

spec:

rules:

- host: "nginx.k8s.dev.huilog.com"

http:

paths:

- backend:

serviceName: nginx-router

servicePort: 80

# 创建部署及服务：

[root@bs-ops-test-docker-dev-01 dev]# kubectl create -f nginx.yaml

deployment.extensions "nginx-router" created

service "nginx-router" created

ingress.extensions "nginx-router-ingress" created

[root@bs-ops-test-docker-dev-01 dev]#

继续阅读 →

阿辉的博客

docker kubernetes istio等分布式系统和云计算等技术研究

标签归档：kubernetes

通过coredns跨k8s集群解析DNS

linux kernel 4.8以上内核NR_KERNEL_STACK的改变

1. NR_KERNEL_STACK是干什么用的？

2. 那么PID与内核栈有什么关系呢？

PVC在K8S集群迁移或恢复需注意地方

istio 1.5版本升级

istio-init容器Init:CrashLoopBackOff故障解决

通过docker overlay2 目录名查找容器名

k8s创建集群只读service account

通过zabbix监控kubernetes集群

github URL

k8s_zabbix说明

check_k8s_status.py说明

腾讯云环境K8S安装及配置测试

1. k8s集群系统规划

1.1. kubernetes 1.10的依赖

1.2 测试服务器准备及环境规划

kubernetes 日常使用及操作测试

1. 创建部署及服务

2022年 5月
一	二	三	四	五	六	日
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30	31