linux kernel 4.8以上内核NR_KERNEL_STACK的改变

发表于2021年3月31日由阿辉

1. NR_KERNEL_STACK是干什么用的？

通过下面的命令，我们可以查看内核栈的数量：

cat /proc/vmstat  | grep stack
nr_kernel_stack 567

cat /proc/vmstat | grep stack

nr_kernel_stack 567

这个值在内核中的宏为：NR_KERNEL_STACK，表示当前内核中有多少个内核栈。

我们使用这个数字来监控K8S节点上使用PID的数量，以避免PID被耗尽。

2. 那么PID与内核栈有什么关系呢？

在linux系统中，每个进程、子进程和线程在运行时都会有一个PID。这些进程或线程在运行时，因为CPU需要进行任务切换，在任务切换时就需要上下文交换，在上下文交换时就需要把当前进程的上下文压到内核栈内去，以便下次再运行时取出继续执行。

所以可以确定：每个进程、子进程和线程都会有一个内核栈。内核栈的数量与PID的数量大致相当。

注：基于linux内核的线程，比如java的线程与linux的线程是一一对应的，nodejs只使用了linux的进程，线程模型是其自己实现的，golang最特别，使用了多进程，每个进程上有多线程（基于内核），线程上还是自己实现的协程或者说goroute(可以理解为自己实现的线程)

继续阅读 →

PVC在K8S集群迁移或恢复需注意地方

发表于2021年3月29日由阿辉

目前在做同城双活，每个机房都有一个k8s集群，其中一些共享卷需要做到双活的2个集群上都可用。

比如我们现在在主机房的集群上有一个PVC：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.cfs
  creationTimestamp: "2020-11-25T07:24:15Z"
  name: public-devops-teststatefulset-nginx-online-pvc
  namespace: test
  uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 30Gi
  storageClassName: cfs-hp
  volumeMode: Filesystem
  volumeName: pvc-3d65a636-f4c3-4c49-b957-c45673f48bc6

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

annotations:

pv.kubernetes.io/bind-completed: "yes"

pv.kubernetes.io/bound-by-controller: "yes"

volume.beta.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.cfs

creationTimestamp: "2020-11-25T07:24:15Z"

namespace: test

uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6

spec:

accessModes:

- ReadWriteMany

resources:

requests:

storage: 30Gi

storageClassName: cfs-hp

volumeMode: Filesystem

volumeName: pvc-3d65a636-f4c3-4c49-b957-c45673f48bc6

以及对应的PV：

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: com.tencent.cloud.csi.cfs
  creationTimestamp: "2020-11-25T07:24:22Z"
  name: pvc-3d65a636-f4c3-4c49-b957-c45673f48bc6
  uid: 301c903e-10fc-40ee-98e1-0655948425fc
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 30Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: public-devops-teststatefulset-nginx-online-pvc
    namespace: test
    resourceVersion: "657951844"
    uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6
  csi:
    driver: com.tencent.cloud.csi.cfs
    fsType: ext4
    volumeAttributes:
      fsid: cfs-3c7e956f6
      host: 10.10.144.12
      storage.kubernetes.io/csiProvisionerIdentity: 1597949457398-8081-com.tencent.cloud.csi.cfs
      storagetype: HP
      subnetid: subnet-mg9pp5o5
      vpcid: vpc-aqpp2l40
      zone: ap-shanghai-4
    volumeHandle: cfs-3c7e956f6
  persistentVolumeReclaimPolicy: Delete
  storageClassName: cfs-hp
  volumeMode: Filesystem

apiVersion: v1

kind: PersistentVolume

metadata:

annotations:

pv.kubernetes.io/provisioned-by: com.tencent.cloud.csi.cfs

creationTimestamp: "2020-11-25T07:24:22Z"

uid: 301c903e-10fc-40ee-98e1-0655948425fc

spec:

accessModes:

- ReadWriteMany

capacity:

storage: 30Gi

claimRef:

apiVersion: v1

kind: PersistentVolumeClaim

namespace: test

resourceVersion: "657951844"

uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6

csi:

driver: com.tencent.cloud.csi.cfs

fsType: ext4

volumeAttributes:

fsid: cfs-3c7e956f6

host: 10.10.144.12

storage.kubernetes.io/csiProvisionerIdentity: 1597949457398-8081-com.tencent.cloud.csi.cfs

storagetype: HP

subnetid: subnet-mg9pp5o5

vpcid: vpc-aqpp2l40

zone: ap-shanghai-4

volumeHandle: cfs-3c7e956f6

persistentVolumeReclaimPolicy: Delete

storageClassName: cfs-hp

volumeMode: Filesystem

继续阅读 →

k8s源码分析：kube-apiserver –admission-control及–enable-admission-plugins –disable-admission-plugins参数差异

发表于2021年2月1日由阿辉

kubernetes源码分支：1.18

先说结论，kube-apiserver启动时：

–admission-control参数带的插件将是apiserver启动的插件，不包括默认插件
–admission-control和–enable-admission-plugins –disable-admission-plugins不能同时使用
–enable-admission-plugins参数不需要按加载顺序填写
不使用–admission-control参数时，api server会同时启动默认插件
–enable-admission-plugins参数启用时，api server会同时启动默认插件，除非使用了–disable-admission-plugins显示的关闭某个插件
–enable-admission-plugins和–disable-admission-plugins如果同时填写了某一个插件，这个插件将会被加载

继续阅读 →

istio 1.6新特性：istio自身灰度发布功能测试

发表于2020年5月27日由阿辉

1. istio 1.6灰度功能测试

istio 1.6新增了istio自身的灰度测试特性，我先测试其灰度功能。

在官网下载并解压istio 1.6，并把istioctl复制到对应目录：

[root@sh-saas-k8s1-master-dev-01 istio]# tar xfzf istio-1.6.0-linux-amd64.tar.gz
[root@sh-saas-k8s1-master-dev-01 istio]# cd istio-1.6.0
[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# cp bin/istioctl /usr/local/bin/
cp: overwrite ‘/usr/local/bin/istioctl’? y
[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# ll tools/
total 188
-rw-r--r-- 1 root root  2031 May 21 07:12 convert_RbacConfig_to_ClusterRbacConfig.sh
-rw-r--r-- 1 root root 10669 May 21 07:12 dump_kubernetes.sh
-rw-r--r-- 1 root root 88599 May 21 07:12 _istioctl
-rw-r--r-- 1 root root 85301 May 21 07:12 istioctl.bash
[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# cp tools/istioctl.bash /usr/local/bin/
cp: overwrite ‘/usr/local/bin/istioctl.bash’? y

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# istioctl version
client version: 1.6.0
control plane version: 1.5.2
data plane version: 1.5.2 (6 proxies), 1.5.1 (4 proxies)

[root@sh-saas-k8s1-master-dev-01 istio]# tar xfzf istio-1.6.0-linux-amd64.tar.gz

[root@sh-saas-k8s1-master-dev-01 istio]# cd istio-1.6.0

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# cp bin/istioctl /usr/local/bin/

cp: overwrite ‘/usr/local/bin/istioctl’? y

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# ll tools/

total 188

-rw-r--r-- 1 root root 2031 May 21 07:12 convert_RbacConfig_to_ClusterRbacConfig.sh

-rw-r--r-- 1 root root 10669 May 21 07:12 dump_kubernetes.sh

-rw-r--r-- 1 root root 88599 May 21 07:12 _istioctl

-rw-r--r-- 1 root root 85301 May 21 07:12 istioctl.bash

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# cp tools/istioctl.bash /usr/local/bin/

cp: overwrite ‘/usr/local/bin/istioctl.bash’? y

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# istioctl version

client version: 1.6.0

control plane version: 1.5.2

data plane version: 1.5.2 (6 proxies), 1.5.1 (4 proxies)

继续阅读 →

istio 1.5版本升级

发表于2020年5月9日由阿辉

之前测试的istio是1.5.1，现在升级到1.5.2。可以参考之前的istio定制安装的文章。http://www.huilog.com/?p=1299

下载新版本，并准备好之前安装时的配置文件

wget https://github.com/istio/istio/releases/download/1.5.2/istio-1.5.2-linux.tar.gz
tar xvzf istio-1.5.2-linux.tar.gz 
cd istio-1.5.1/
cp -rf *.yaml ../istio-1.5.2/
cd ../istio-1.5.2/

cp bin/istioctl /usr/local/bin/
cp tools/istioctl.bash /usr/local/bin/

wget https://github.com/istio/istio/releases/download/1.5.2/istio-1.5.2-linux.tar.gz

tar xvzf istio-1.5.2-linux.tar.gz

cd istio-1.5.1/

cp -rf *.yaml ../istio-1.5.2/

cd ../istio-1.5.2/

cp bin/istioctl /usr/local/bin/

cp tools/istioctl.bash /usr/local/bin/

查看支持的版本列表，验证 istoctl 命令是否支持从当前版本升级

[root@sh-saas-k8s1-master-dev-01 istio-1.5.2]# istioctl manifest versions

Operator version is 1.5.2.

The following installation package versions are recommended for use with this version of the operator:
  1.5.0

The following installation package versions are supported for upgrade by this version of the operator:
  >=1.4.0
   <1.6

[root@sh-saas-k8s1-master-dev-01 istio-1.5.2]# istioctl manifest versions

Operator version is 1.5.2.

The following installation package versions are recommended for use with this version of the operator:

1.5.0

The following installation package versions are supported for upgrade by this version of the operator:

>=1.4.0

<1.6

继续阅读 →

istio-init容器Init:CrashLoopBackOff故障解决

发表于2020年5月8日由阿辉

最近在测试istio时，经常发现注入过sidecar的pod过段时间就变成了Init:CrashLoopBackOff状态。如：

[root@sh-saas-k8s1-master-dev-01 ~]# kubectl get pod --all-namespaces -o wide | grep  'Init'
public-ops-tomcat-dev           public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-64qwr              0/2     Init:CrashLoopBackOff   7          21h     10.253.3.166   10.12.97.23   <none>           <none>

[root@sh-saas-k8s1-master-dev-01 ~]# kubectl get pod --all-namespaces -o wide | grep 'Init'

public-ops-tomcat-dev public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-64qwr 0/2 Init:CrashLoopBackOff 7 21h 10.253.3.166 10.12.97.23 <none> <none>

我的kubernetes版本为1.14.10，istio版本为：1.5.1
查看istio-init容器的日志，发现有如下的报错：

[root@sh-saas-k8s1-master-dev-01 ~]# kubectl logs -n public-ops-tomcat-dev           public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-64qwr istio-init 
Environment:
------------
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false

Writing following contents to rules file:  /tmp/iptables-rules-1588923880490327697.txt562915423
* nat
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_INBOUND
-N ISTIO_OUTPUT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT

iptables-restore --noflush /tmp/iptables-rules-1588923880490327697.txt562915423
iptables-restore: line 2 failed
iptables-save 
# Generated by iptables-save v1.6.1 on Fri May  8 07:44:40 2020
*mangle
:PREROUTING ACCEPT [643414:2344563772]
:INPUT ACCEPT [643414:2344563772]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [616124:4267707048]
:POSTROUTING ACCEPT [616124:4267707048]
COMMIT
# Completed on Fri May  8 07:44:40 2020
# Generated by iptables-save v1.6.1 on Fri May  8 07:44:40 2020
*raw
:PREROUTING ACCEPT [643414:2344563772]
:OUTPUT ACCEPT [616124:4267707048]
COMMIT
# Completed on Fri May  8 07:44:40 2020
# Generated by iptables-save v1.6.1 on Fri May  8 07:44:40 2020
*nat
:PREROUTING ACCEPT [38474:2000648]
:INPUT ACCEPT [40999:2131948]
:OUTPUT ACCEPT [7987:560379]
:POSTROUTING ACCEPT [8763:600731]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Fri May  8 07:44:40 2020
# Generated by iptables-save v1.6.1 on Fri May  8 07:44:40 2020
*filter
:INPUT ACCEPT [643414:2344563772]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [616124:4267707048]
COMMIT
# Completed on Fri May  8 07:44:40 2020
panic: exit status 1

goroutine 1 [running]:
istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0xd819c0, 0x9739b8, 0x10, 0xc00000cbc0, 0x2, 0x2)
        istio.io/istio@/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc000109d30, 0x7faeecd9a001, 0x0, 0x0)
        istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:474 +0x3aa
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc000109d30)
        istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:481 +0x45
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc000109d30)
        istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:428 +0x24e2
istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0xd5c740, 0xc0000ee700, 0x0, 0x10)
        istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:56 +0x14e
github.com/spf13/cobra.(*Command).execute(0xd5c740, 0xc00001e130, 0x10, 0x11, 0xd5c740, 0xc00001e130)
        github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xd5c740, 0x40574f, 0xc00009e058, 0x0)
        github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/cobra@v0.0.5/command.go:864
istio.io/istio/tools/istio-iptables/pkg/cmd.Execute()
        istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:284 +0x2d
main.main()
        istio.io/istio@/tools/istio-iptables/main.go:22 +0x20

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

[root@sh-saas-k8s1-master-dev-01 ~]# kubectl logs -n public-ops-tomcat-dev public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-64qwr istio-init

Environment:

------------

ENVOY_PORT=

INBOUND_CAPTURE_PORT=

ISTIO_INBOUND_INTERCEPTION_MODE=

ISTIO_INBOUND_TPROXY_MARK=

ISTIO_INBOUND_TPROXY_ROUTE_TABLE=

ISTIO_INBOUND_PORTS=

ISTIO_LOCAL_EXCLUDE_PORTS=

ISTIO_SERVICE_CIDR=

ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:

----------

PROXY_PORT=15001

PROXY_INBOUND_CAPTURE_PORT=15006

PROXY_UID=1337

PROXY_GID=1337

INBOUND_INTERCEPTION_MODE=REDIRECT

INBOUND_TPROXY_MARK=1337

INBOUND_TPROXY_ROUTE_TABLE=133

INBOUND_PORTS_INCLUDE=*

INBOUND_PORTS_EXCLUDE=15090,15020

OUTBOUND_IP_RANGES_INCLUDE=*

OUTBOUND_IP_RANGES_EXCLUDE=

OUTBOUND_PORTS_EXCLUDE=

KUBEVIRT_INTERFACES=

ENABLE_INBOUND_IPV6=false

Writing following contents to rules file: /tmp/iptables-rules-1588923880490327697.txt562915423

* nat

-N ISTIO_REDIRECT

-N ISTIO_IN_REDIRECT

-N ISTIO_INBOUND

-N ISTIO_OUTPUT

-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-port 15001

-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-port 15006

-A PREROUTING -p tcp -j ISTIO_INBOUND

-A ISTIO_INBOUND -p tcp --dport 22 -j RETURN

-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN

-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN

-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT

-A OUTPUT -p tcp -j ISTIO_OUTPUT

-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN

-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT

-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT

-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN

-A ISTIO_OUTPUT -j ISTIO_REDIRECT

COMMIT

iptables-restore --noflush /tmp/iptables-rules-1588923880490327697.txt562915423

iptables-restore: line 2 failed

iptables-save

# Generated by iptables-save v1.6.1 on Fri May 8 07:44:40 2020

*mangle

:PREROUTING ACCEPT [643414:2344563772]

:INPUT ACCEPT [643414:2344563772]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [616124:4267707048]

:POSTROUTING ACCEPT [616124:4267707048]

COMMIT

# Completed on Fri May 8 07:44:40 2020

# Generated by iptables-save v1.6.1 on Fri May 8 07:44:40 2020

*raw

:PREROUTING ACCEPT [643414:2344563772]

:OUTPUT ACCEPT [616124:4267707048]

COMMIT

# Completed on Fri May 8 07:44:40 2020

# Generated by iptables-save v1.6.1 on Fri May 8 07:44:40 2020

*nat

:PREROUTING ACCEPT [38474:2000648]

:INPUT ACCEPT [40999:2131948]

:OUTPUT ACCEPT [7987:560379]

:POSTROUTING ACCEPT [8763:600731]

:ISTIO_INBOUND - [0:0]

:ISTIO_IN_REDIRECT - [0:0]

:ISTIO_OUTPUT - [0:0]

:ISTIO_REDIRECT - [0:0]

-A PREROUTING -p tcp -j ISTIO_INBOUND

-A OUTPUT -p tcp -j ISTIO_OUTPUT

-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN

-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN

-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN

-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT

-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006

-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN

-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT

-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN

-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT

-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN

-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN

-A ISTIO_OUTPUT -j ISTIO_REDIRECT

-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001

COMMIT

# Completed on Fri May 8 07:44:40 2020

# Generated by iptables-save v1.6.1 on Fri May 8 07:44:40 2020

*filter

:INPUT ACCEPT [643414:2344563772]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [616124:4267707048]

COMMIT

# Completed on Fri May 8 07:44:40 2020

panic: exit status 1

goroutine 1 [running]:

istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0xd819c0, 0x9739b8, 0x10, 0xc00000cbc0, 0x2, 0x2)

istio.io/istio@/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96

istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc000109d30, 0x7faeecd9a001, 0x0, 0x0)

istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:474 +0x3aa

istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc000109d30)

istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:481 +0x45

istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc000109d30)

istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:428 +0x24e2

istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0xd5c740, 0xc0000ee700, 0x0, 0x10)

istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:56 +0x14e

github.com/spf13/cobra.(*Command).execute(0xd5c740, 0xc00001e130, 0x10, 0x11, 0xd5c740, 0xc00001e130)

github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa

github.com/spf13/cobra.(*Command).ExecuteC(0xd5c740, 0x40574f, 0xc00009e058, 0x0)

github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb

github.com/spf13/cobra.(*Command).Execute(...)

github.com/spf13/cobra@v0.0.5/command.go:864

istio.io/istio/tools/istio-iptables/pkg/cmd.Execute()

istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:284 +0x2d

main.main()

istio.io/istio@/tools/istio-iptables/main.go:22 +0x20

继续阅读 →

istio 1.5 定制安装

发表于2020年4月4日由阿辉

在istio 1.5，已经不支持helm方式安装，因为helm已经弃用。helm部分的代码已不在更新。只支持istioctl的方式安装istio。

istioctl在安装包内，可通过下面的链接下载安装包。
https://istio.io/docs/setup/getting-started/#download

安装包解压后，直接将istioctl复制到执行目录就可以用了：
cp bin/istioctl /usr/local/bin/

以下命令可以默认配置安装istio:
istioctl manifest apply

如果想安装demo：
istioctl manifest apply --set profile=demo

但是默认配置有些部件是没有选中的。参考下图：

继续阅读 →

Api Gateway和服务网格的对比

发表于2020年3月29日由阿辉

在服务网络出来之前，就已经有很多各式各样的Api Gateway了。他们之间有什么不同呢？

1. Api Gateway架构

Api Gateway有很多，现阶段做的比较好的有ambassador和kong。还有一些其它的，目前看来这两个用的人比较多。
– Ambassador架构：

Kong架构：

从流量转发的角度来看，Api Gateway主要做的事是外部到内部的流量转发，也就是通常所说的南北流量。基于kubernetes的架构内来说，其主要是实现ingress的功能。

应用内部之间的流量（东西流量）并不是它关注的重点，虽然部分上也可以实现，但是相关功能并不多。

2. API Gateway解决了什么问题？

API Gateway主要是解决了API管理的问题。

根据Wikipedia的定义：API管理的过程包括创建和发布Web API、执行调用策略、访问控制、订阅管理、收集和分析调用统计以及报告性能。

根据以上定义，API Gateway主要的功能如下：
– api的创建和发布，比如说api的自动注册
– 访问控制和调用策略（流控等），谁能访问？API被调用时，怎么确保安全和稳定的访问？
– 订阅管理，哪些人可以访问我的API
– 调用性能分析及信息收集等

继续阅读 →

traefik 重写配置

发表于2020年3月11日由阿辉

traefik ingress同样可以配置URL的重写：

traefik 1.x配置方法

下面是一个完整例子：

 apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      kubernetes.io/ingress.class: traefik
      traefik.ingress.kubernetes.io/request-modifier: 'ReplacePathRegex: ^/api3/(.*)
        /api/$1'
    creationTimestamp: "2020-03-09T03:27:40Z"
    generation: 2
    labels:
      app: public-fe-zhan-operation-node-qa
    name: public-fe-zhan-operation-node-qa-7091-2-ingress
    namespace: public-fe-node-qa
    resourceVersion: "2310342"
    selfLink: /apis/extensions/v1beta1/namespaces/public-fe-node-qa/ingresses/public-fe-zhan-operation-node-qa-7091-2-ingress
    uid: ee6f6696-61b5-11ea-a82f-52540088db9a
  spec:
    rules:
    - host: www.xxxx.com
      http:
        paths:
        - backend:
            serviceName: public-fe-zhan-operation-node-qa
            servicePort: 7091
          path: /api3

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

annotations:

kubernetes.io/ingress.class: traefik

traefik.ingress.kubernetes.io/request-modifier: 'ReplacePathRegex: ^/api3/(.*)

/api/$1'

creationTimestamp: "2020-03-09T03:27:40Z"

generation: 2

labels:

app: public-fe-zhan-operation-node-qa

namespace: public-fe-node-qa

resourceVersion: "2310342"

selfLink: /apis/extensions/v1beta1/namespaces/public-fe-node-qa/ingresses/public-fe-zhan-operation-node-qa-7091-2-ingress

uid: ee6f6696-61b5-11ea-a82f-52540088db9a

spec:

rules:

- host: www.xxxx.com

http:

paths:

- backend:

serviceName: public-fe-zhan-operation-node-qa

servicePort: 7091

path: /api3

参考文档：
https://s0docs0traefik0io.icopy.site/v1.7/basics/#path-matcher-usage-guidelines

https://docs.traefik.io/v1.7/basics/#rules-order

继续阅读 →

nginx的add_header及proxy_set_header命令需注意的地方

发表于2019年11月15日由阿辉

nginx的add_header及proxy_set_header命令有三个地方可以配置，分别是http,server,location。我们一般对其优先级的理解是location->server->http。

这没有错，但是这个优先级是指令级别的（不包括指令后面的参数），也就是说：假设在http和server内都有add_header,那么只有server内的生效，忽略http内的add_header，而不管在这两个地方配置新增的header是不是一样。

proxy_set_header也是一样的效果。所以个人推测有很多nginx指令可能都是这样的规则。

另个在同一级别也有坑，对于add_header，如果同一个域名的两个不同location加了不同的add_header，那么只有最后一个location内的生效。这个我没有亲测，详细的可以看下面的参考链接。

继续阅读 →

阿辉的博客

docker kubernetes istio等分布式系统和云计算等技术研究

linux kernel 4.8以上内核NR_KERNEL_STACK的改变

1. NR_KERNEL_STACK是干什么用的？

2. 那么PID与内核栈有什么关系呢？

PVC在K8S集群迁移或恢复需注意地方

k8s源码分析：kube-apiserver –admission-control及–enable-admission-plugins –disable-admission-plugins参数差异

istio 1.6新特性：istio自身灰度发布功能测试

1. istio 1.6灰度功能测试

istio 1.5版本升级

istio-init容器Init:CrashLoopBackOff故障解决

istio 1.5 定制安装

Api Gateway和服务网格的对比

1. Api Gateway架构

2. API Gateway解决了什么问题？

traefik 重写配置

nginx的add_header及proxy_set_header命令需注意的地方

2021年 9月
一	二	三	四	五	六	日
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30