AWS VPC CNI路由分析

发表于2022年5月13日由阿辉

amazon-vpc-cni-k8s是AWS基于VPC CNI的k8s网络插件，有着高性能及高度灵活的优点。项目地址：
https://github.com/aws/amazon-vpc-cni-k8s

下面我们通过分析其源码，查看实际上相关的CNI网络是怎么实现的。

1. AWS VPC CNI Agent node部分

Node上的Agent程序与路由相关的，主要做2块工作，一个是配置主网卡（SetupHostNetwork），一个是配置辅助网卡（setupENINetwork）。

1.1 SetupHostNetwork

针对host网络和主网卡Primary ENI做一些配置，Primary ENI: 主机的默认网卡,默认一般为eth0。

如果启用ipv4和支持node port，设置RP Filter：

// 启用ConfigureRpFilter，设置net.ipv4.conf.{primaryIntf}.rp_filter为2
primaryIntfRPFilter := "net/ipv4/conf/" + primaryIntf + "/rp_filter"
echo 2 > primaryIntfRPFilter

// 启用ConfigureRpFilter，设置net.ipv4.conf.{primaryIntf}.rp_filter为2

primaryIntfRPFilter := "net/ipv4/conf/" + primaryIntf + "/rp_filter"

echo 2 > primaryIntfRPFilter

设置Primary eni网卡的mtu：

ip link set dev <link> mtu MTU

1 2	ip link set dev <link> mtu MTU

先删除main表里优先级为1024的rule

// If this is a restart, cleanup previous rule first
ip rule del fwmark 0x80/0x80 pref 1024 table main

// If this is a restart, cleanup previous rule first

ip rule del fwmark 0x80/0x80 pref 1024 table main

如果启用node port支持，添加基于fwmark的路由策略:

// defaultConnmark is the default value for the connmark described above. Note: the mark space is a little crowded,
// - kube-proxy uses 0x0000c000
// - Calico uses 0xffff0000.
// defaultConnmark = 0x80
ip rule add fwmark 0x80/0x80 pref 1024 table main

// defaultConnmark is the default value for the connmark described above. Note: the mark space is a little crowded,

// - kube-proxy uses 0x0000c000

// - Calico uses 0xffff0000.

// defaultConnmark = 0x80

ip rule add fwmark 0x80/0x80 pref 1024 table main

如果启用ipv4且启用pod eni，则：

// Add new rule with higher priority
// 添加table为255的优先级为20策略路由
ip rule add pref 20 table local

// 删除table为255的优先级为0的策略路由
ip rule del pref 0 table local

// Add new rule with higher priority

// 添加table为255的优先级为20策略路由

ip rule add pref 20 table local

// 删除table为255的优先级为0的策略路由

ip rule del pref 0 table local

创建snat的iptables规则，这里省略

1.2 setupENINetwork

setupENINetwork为非Primary ENI的弹性网卡配置网络，每块弹性网卡都有一个主IP+N个辅助IP

eniName: 主机的默认网卡,默认一般为eth0+x，X为网卡序号，如eth1

设置eni网卡的mtu

ip link set dev <eniName> mtu MTU

1 2	ip link set dev <eniName> mtu MTU

让eni网卡up

ip link set dev <eniName> up

1 2	ip link set dev <eniName> up

删除eni网卡上所有已经存在的ip地址

ip add del <eniIP> dev <eniName> (if necessary)

1 2	ip add del <eniIP> dev <eniName> (if necessary)

设置eni网卡的ip

// eniIP： primary IP of that ENI
ip add add <eniIP> dev <eniName>

// eniIP： primary IP of that ENI

ip add add <eniIP> dev <eniName>

在deviceNumber + 1路由表里，删除已经存在默认路由，添加默认路由指向eni子网的网关

// tableNumber := deviceNumber + 1
// gw: 网关为Eni IP子网的第二个IP地址
// Add a direct link route for the host is ENI IP only
ip route add <gw>/32 scope link  dev <eniName> table tableNumber

// Route all other traffic via the host is ENI IP
// tableNumber := deviceNumber + 1
// gw: 网关为Eni IP子网的第二个IP地址
ip route add 0.0.0.0/0 scope 0 via <gw> dev <eniName> table tableNumber

// 在main路由表里删除源地址为eniIP，目标地址为eniSubnetCIDR的路由
ip route del eniSubnetCIDR src eniIP scope link table main

// tableNumber := deviceNumber + 1

// gw: 网关为Eni IP子网的第二个IP地址

// Add a direct link route for the host is ENI IP only

ip route add <gw>/32 scope link dev <eniName> table tableNumber

// Route all other traffic via the host is ENI IP

// tableNumber := deviceNumber + 1

// gw: 网关为Eni IP子网的第二个IP地址

ip route add 0.0.0.0/0 scope 0 via <gw> dev <eniName> table tableNumber

// 在main路由表里删除源地址为eniIP，目标地址为eniSubnetCIDR的路由

ip route del eniSubnetCIDR src eniIP scope link table main

2. AWS CNI二进制文件

CNI二进制文件主要是实现CNI标准规则的几个接口：

2.1 cmdAdd

grpc请求”/rpc.CNIBackend/AddNetwork”调用ipam分配ip
构建hostVethName

// build hostVethName
// Note: the maximum length for linux interface name is 15
// 返回veth网卡名字，{conf.VethPrefix}+{hash({namespace}.{pod name})前10位}
// VethPrefix is the prefix to use when constructing the host-side
// veth device name. It should be no more than four characters, and
// defaults to 'eni'.
// 如: eni1234567890

// build hostVethName

// Note: the maximum length for linux interface name is 15

// 返回veth网卡名字，{conf.VethPrefix}+{hash({namespace}.{pod name})前10位}

// VethPrefix is the prefix to use when constructing the host-side

// veth device name. It should be no more than four characters, and

// defaults to 'eni'.

// 如: eni1234567890

创建veth和容器内网卡，绑定容器网卡ip地址，容器内设置路由和其他内核参数

// Clean up if hostVeth exists.
// 如果hostVethName网卡存在，删除宿主机网卡hostVethName
ip link del <oldlink>

// Clean up if hostVeth exists.

// 如果hostVethName网卡存在，删除宿主机网卡hostVethName

ip link del <oldlink>

4.以下步骤在POD network namespce内执行

1) 创建网卡

// createVethContext.contVethName:调用CNI二进制文件时传过来的网卡名：一般为eth0
ip link add {createVethContext.contVethName}
ip link add {createVethContext.contVethName} type veth peer name {createVethContext.hostVethName} /* on host namespace */

2) 启动网卡

ip link set dev {createVethContext.hostVethName} up
ip link set dev {createVethContext.contVethName} up

3) 添加路由

// Add a connected route to a dummy next hop (169.254.1.1 or fe80::1)
// # ip route show
// default via 169.254.1.1 dev eth0
// 169.254.1.1 dev eth0
// scope link = scope 253
ip route add 169.254.1.1 scope link dev {createVethContext.contVethName}

// Add a default route via dummy next hop(169.254.1.1 or fe80::1). Then all outgoing traffic will be routed by this
// default route via dummy next hop (169.254.1.1 or fe80::1)
// 添加默认路由
// scope 0 = scope universe
ip route add 0.0.0.0/0 scope 0 via 169.254.1.1

4) 给容器里的网卡绑定ip

// $addr: 调用ipam分配ip
ip addr add $addr dev <createVethContext.contVethName>

5) 添加arp表

// $addr: gateway ip 169.254.1.1
// $mac: hostVeth.Attrs().HardwareAddr
ip neigh add $addr lladdr $mac nud permanent dev <createVethContext.contVethName>

6) 将host端网卡移动到host命名空间

// veth into the host namespace.
// * move {createVethContext.hostVethName} to Pod's namespace hostNS */
ip link set {createVethContext.hostVethName} netns {hostNS}

1) 创建网卡

// createVethContext.contVethName:调用CNI二进制文件时传过来的网卡名：一般为eth0

ip link add {createVethContext.contVethName}

ip link add {createVethContext.contVethName} type veth peer name {createVethContext.hostVethName} /* on host namespace */

2) 启动网卡

ip link set dev {createVethContext.hostVethName} up

ip link set dev {createVethContext.contVethName} up

3) 添加路由

// Add a connected route to a dummy next hop (169.254.1.1 or fe80::1)

// # ip route show

// default via 169.254.1.1 dev eth0

// 169.254.1.1 dev eth0

// scope link = scope 253

ip route add 169.254.1.1 scope link dev {createVethContext.contVethName}

// Add a default route via dummy next hop(169.254.1.1 or fe80::1). Then all outgoing traffic will be routed by this

// default route via dummy next hop (169.254.1.1 or fe80::1)

// 添加默认路由

// scope 0 = scope universe

ip route add 0.0.0.0/0 scope 0 via 169.254.1.1

4) 给容器里的网卡绑定ip

// $addr: 调用ipam分配ip

ip addr add $addr dev <createVethContext.contVethName>

5) 添加arp表

// $addr: gateway ip 169.254.1.1

// $mac: hostVeth.Attrs().HardwareAddr

ip neigh add $addr lladdr $mac nud permanent dev <createVethContext.contVethName>

6) 将host端网卡移动到host命名空间

// veth into the host namespace.

// * move {createVethContext.hostVethName} to Pod's namespace hostNS */

ip link set {createVethContext.hostVethName} netns {hostNS}

启动host网卡

// Explicitly set the veth to UP state, because netlink doesn't always do that on all the platforms with net.FlagUp.
// veth won't get a link local address unless it's set to UP state
ip link set dev {createVethContext.hostVethName} up

// Explicitly set the veth to UP state, because netlink doesn't always do that on all the platforms with net.FlagUp.

// veth won't get a link local address unless it's set to UP state

ip link set dev {createVethContext.hostVethName} up

添加用于host上访问pod的路由

// Add or replace route
// $addr: 调用ipam分配ip
ip route add $addr/32 dev {hostVethName} scope link /* add host route */

// Add or replace route

// $addr: 调用ipam分配ip

ip route add $addr/32 dev {hostVethName} scope link /* add host route */

添加用于访问pod流量的路由策略

// $addr: 调用ipam分配ip
// 512: from all to 10.200.202.222 lookup main
ip rule del to $addr/32 pref 512 table main
ip rule add to $addr/32 pref 512 table main

// add from-pod rule, only need it when it is not primary ENI
// <tableNumber>里的路由应该是在ipam中添加的，不是cni插件
if deviceNumber > 0 {
    // add rule: 1536: from <podIP> use table <table>
    // $addr: 调用ipam分配ip
    // tableNumber := deviceNumber + 1
    // deviceNumber: 调用ipam分配ip时取到的
    // 1536: from 10.200.202.222 to all lookup <tableNumber>
    ip rule del from $addr/32 pref 1536 table tableNumber
    ip rule add from $addr/32 pref 1536 table tableNumber
}

// $addr: 调用ipam分配ip

// 512: from all to 10.200.202.222 lookup main

ip rule del to $addr/32 pref 512 table main

ip rule add to $addr/32 pref 512 table main

// add from-pod rule, only need it when it is not primary ENI

// <tableNumber>里的路由应该是在ipam中添加的，不是cni插件

if deviceNumber > 0 {

// add rule: 1536: from <podIP> use table <table>

// $addr: 调用ipam分配ip

// tableNumber := deviceNumber + 1

// deviceNumber: 调用ipam分配ip时取到的

// 1536: from 10.200.202.222 to all lookup <tableNumber>

ip rule del from $addr/32 pref 1536 table tableNumber

ip rule add from $addr/32 pref 1536 table tableNumber

}

3. 参考文件

AWS 弹性网卡参考

https://aws.amazon.com/cn/premiumsupport/knowledge-center/ec2-ubuntu-secondary-network-interface/

cni文档

https://github.com/containernetworking/cni/blob/spec-v1.0.0/SPEC.md
https://www.cni.dev/docs/

libcni库

https://github.com/containernetworking/cni/tree/main/scripts
https://www.cni.dev/docs/spec-upgrades/#specific-guidance-for-plugins-written-in-go

cni调用原理

http://www.noobyard.com/article/p-mjmyxamv-ob.html
https://blog.csdn.net/shida_csdn/article/details/79752411
https://segmentfault.com/a/1190000019956620
https://morningspace.github.io/tech/k8s-net-cni/

k8s apiserver 自签名证书过期导致集群故障

发表于2022年5月10日由阿辉

1. 故障处理过程

今天接到同事反馈发现有一套k8s apiserver集群出现如下报错：

Failed to create new replica set "recommend-alg-service-74c6bc97cd": Get https://10.13.96.12:6443/api/v1/namespaces/saas-ec-tomcat-pl/resourcequotas: x509: certificate has expired or is not yet valid

1 2	Failed to create new replica set "recommend-alg-service-74c6bc97cd": Get https://10.13.96.12:6443/api/v1/namespaces/saas-ec-tomcat-pl/resourcequotas: x509: certificate has expired or is not yet valid

随后去api server节点上查询api server日志，发现也有大量报错：

I0510 17:43:56.889617  790992 reflector.go:211] Listing and watching *v1.MutatingWebhookConfiguration from k8s.io/client-go/informers/factory.go:135
I0510 17:43:56.892900  790992 log.go:172] http: TLS handshake error from 10.13.96.11:36528: remote error: tls: bad certificate
E0510 17:43:56.892945  790992 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.MutatingWebhookConfiguration: Get https://10.13.96.11:6443/apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations?resourceVersion=331916875: x509: certificate has expired or is not yet valid
I0510 17:43:57.087301  790992 reflector.go:211] Listing and watching *v1.ClusterRole from k8s.io/client-go/informers/factory.go:135
I0510 17:43:57.090560  790992 log.go:172] http: TLS handshake error from 10.13.96.11:36530: remote error: tls: bad certificate
E0510 17:43:57.090625  790992 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.ClusterRole: Get https://10.13.96.11:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?resourceVersion=397554978: x509: certificate has expired or is not yet valid

I0510 17:43:56.889617 790992 reflector.go:211] Listing and watching *v1.MutatingWebhookConfiguration from k8s.io/client-go/informers/factory.go:135

I0510 17:43:56.892900 790992 log.go:172] http: TLS handshake error from 10.13.96.11:36528: remote error: tls: bad certificate

E0510 17:43:56.892945 790992 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.MutatingWebhookConfiguration: Get https://10.13.96.11:6443/apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations?resourceVersion=331916875: x509: certificate has expired or is not yet valid

I0510 17:43:57.087301 790992 reflector.go:211] Listing and watching *v1.ClusterRole from k8s.io/client-go/informers/factory.go:135

I0510 17:43:57.090560 790992 log.go:172] http: TLS handshake error from 10.13.96.11:36530: remote error: tls: bad certificate

E0510 17:43:57.090625 790992 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.ClusterRole: Get https://10.13.96.11:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?resourceVersion=397554978: x509: certificate has expired or is not yet valid

让人奇怪的是，并不是所有请求都报证书错误，通过日志发现，大量的请求到api server都是200的，出现500的为少数。

同时我们通过上面的报错发现，报错的是10.13.96.11:36528，这个IP是api server自己。

最后我们将错误定位到以下范围：

api server自己访问自己报证书过期的错误，而其它组件访问都是正常的。

同时我们检查了服务器上的所有k8s 集群使用的证书，并没有过期。

在无计可施时，怀着试试看的想法，我们将api server服务重启了，发现重启后恢复正常。果然是重启能解决99%的问题。

2. 故障原因排查

故障恢复后，我们一直在排查原因，排查原因时我们发现下面3点：

api server服务正好启动了一年没有重启
api server自己访问自己报证书过期的错误
api server访问自己使用的是IP地址，然我们api server对其它组件及外部都是使用域名的

下面就只能看源码了，通过api server的源码，我们发现api server启动时会自动生成一个自签名证书：

k8s.io/apiserver/pkg/server/options/serving_with_loopback.go:

// ApplyTo fills up serving information in the server configuration.
func (s *SecureServingOptionsWithLoopback) ApplyTo(secureServingInfo **server.SecureServingInfo, loopbackClientConfig **rest.Config) error {
    if s == nil || s.SecureServingOptions == nil || secureServingInfo == nil {
        return nil
    }

    if err := s.SecureServingOptions.ApplyTo(secureServingInfo); err != nil {
        return err
    }

    if *secureServingInfo == nil || loopbackClientConfig == nil {
        return nil
    }

    // create self-signed cert+key with the fake server.LoopbackClientServerNameOverride and
    // let the server return it when the loopback client connects.
    certPem, keyPem, err := certutil.GenerateSelfSignedCertKey(server.LoopbackClientServerNameOverride, nil, nil)
    if err != nil {
        return fmt.Errorf("failed to generate self-signed certificate for loopback connection: %v", err)
    }
    certProvider, err := dynamiccertificates.NewStaticSNICertKeyContent("self-signed loopback", certPem, keyPem, server.LoopbackClientServerNameOverride)
    if err != nil {
        return fmt.Errorf("failed to generate self-signed certificate for loopback connection: %v", err)
    }

    secureLoopbackClientConfig, err := (*secureServingInfo).NewLoopbackClientConfig(uuid.New().String(), certPem)
    switch {
    // if we failed and there's no fallback loopback client config, we need to fail
    case err != nil && *loopbackClientConfig == nil:
        return err

    // if we failed, but we already have a fallback loopback client config (usually insecure), allow it
    case err != nil && *loopbackClientConfig != nil:

    default:
        *loopbackClientConfig = secureLoopbackClientConfig
        // Write to the front of SNICerts so that this overrides any other certs with the same name
        (*secureServingInfo).SNICerts = append([]dynamiccertificates.SNICertKeyContentProvider{certProvider}, (*secureServingInfo).SNICerts...)
    }

    return nil
}

// ApplyTo fills up serving information in the server configuration.

func (s *SecureServingOptionsWithLoopback) ApplyTo(secureServingInfo **server.SecureServingInfo, loopbackClientConfig **rest.Config) error {

if s == nil || s.SecureServingOptions == nil || secureServingInfo == nil {

return nil

}

if err := s.SecureServingOptions.ApplyTo(secureServingInfo); err != nil {

return err

}

if *secureServingInfo == nil || loopbackClientConfig == nil {

return nil

}

// create self-signed cert+key with the fake server.LoopbackClientServerNameOverride and

// let the server return it when the loopback client connects.

certPem, keyPem, err := certutil.GenerateSelfSignedCertKey(server.LoopbackClientServerNameOverride, nil, nil)

if err != nil {

return fmt.Errorf("failed to generate self-signed certificate for loopback connection: %v", err)

}

certProvider, err := dynamiccertificates.NewStaticSNICertKeyContent("self-signed loopback", certPem, keyPem, server.LoopbackClientServerNameOverride)

if err != nil {

return fmt.Errorf("failed to generate self-signed certificate for loopback connection: %v", err)

}

secureLoopbackClientConfig, err := (*secureServingInfo).NewLoopbackClientConfig(uuid.New().String(), certPem)

switch {

// if we failed and there's no fallback loopback client config, we need to fail

case err != nil && *loopbackClientConfig == nil:

return err

// if we failed, but we already have a fallback loopback client config (usually insecure), allow it

case err != nil && *loopbackClientConfig != nil:

default:

*loopbackClientConfig = secureLoopbackClientConfig

// Write to the front of SNICerts so that this overrides any other certs with the same name

(*secureServingInfo).SNICerts = append([]dynamiccertificates.SNICertKeyContentProvider{certProvider}, (*secureServingInfo).SNICerts...)

}

return nil

}

而且这个自签名证书的时间为一年。

k8s.io/client-go/util/cert/cert.go:

// GenerateSelfSignedCertKey creates a self-signed certificate and key for the given host.
// Host may be an IP or a DNS name
// You may also specify additional subject alt names (either ip or dns names) for the certificate.
func GenerateSelfSignedCertKey(host string, alternateIPs []net.IP, alternateDNS []string) ([]byte, []byte, error) {
    return GenerateSelfSignedCertKeyWithFixtures(host, alternateIPs, alternateDNS, "")
}

// GenerateSelfSignedCertKeyWithFixtures creates a self-signed certificate and key for the given host.
// Host may be an IP or a DNS name. You may also specify additional subject alt names (either ip or dns names)
// for the certificate.
//
// If fixtureDirectory is non-empty, it is a directory path which can contain pre-generated certs. The format is:
// <host>_<ip>-<ip>_<alternateDNS>-<alternateDNS>.crt
// <host>_<ip>-<ip>_<alternateDNS>-<alternateDNS>.key
// Certs/keys not existing in that directory are created.
func GenerateSelfSignedCertKeyWithFixtures(host string, alternateIPs []net.IP, alternateDNS []string, fixtureDirectory string) ([]byte, []byte, error) {
    validFrom := time.Now().Add(-time.Hour) // valid an hour earlier to avoid flakes due to clock skew
    maxAge := time.Hour * 24 * 365          // one year self-signed certs

    baseName := fmt.Sprintf("%s_%s_%s", host, strings.Join(ipsToStrings(alternateIPs), "-"), strings.Join(alternateDNS, "-"))
    certFixturePath := path.Join(fixtureDirectory, baseName+".crt")
    keyFixturePath := path.Join(fixtureDirectory, baseName+".key")
    if len(fixtureDirectory) > 0 {
        cert, err := ioutil.ReadFile(certFixturePath)
        if err == nil {
            key, err := ioutil.ReadFile(keyFixturePath)
            if err == nil {
                return cert, key, nil
            }
            return nil, nil, fmt.Errorf("cert %s can be read, but key %s cannot: %v", certFixturePath, keyFixturePath, err)
        }
        maxAge = 100 * time.Hour * 24 * 365 // 100 years fixtures
    }

    caKey, err := rsa.GenerateKey(cryptorand.Reader, 2048)
    if err != nil {
        return nil, nil, err
    }

// GenerateSelfSignedCertKey creates a self-signed certificate and key for the given host.

// Host may be an IP or a DNS name

// You may also specify additional subject alt names (either ip or dns names) for the certificate.

func GenerateSelfSignedCertKey(host string, alternateIPs []net.IP, alternateDNS []string) ([]byte, []byte, error) {

return GenerateSelfSignedCertKeyWithFixtures(host, alternateIPs, alternateDNS, "")

}

// GenerateSelfSignedCertKeyWithFixtures creates a self-signed certificate and key for the given host.

// Host may be an IP or a DNS name. You may also specify additional subject alt names (either ip or dns names)

// for the certificate.

// If fixtureDirectory is non-empty, it is a directory path which can contain pre-generated certs. The format is:

// <host>_<ip>-<ip>_<alternateDNS>-<alternateDNS>.crt

// <host>_<ip>-<ip>_<alternateDNS>-<alternateDNS>.key

// Certs/keys not existing in that directory are created.

func GenerateSelfSignedCertKeyWithFixtures(host string, alternateIPs []net.IP, alternateDNS []string, fixtureDirectory string) ([]byte, []byte, error) {

validFrom := time.Now().Add(-time.Hour) // valid an hour earlier to avoid flakes due to clock skew

maxAge := time.Hour * 24 * 365 // one year self-signed certs

baseName := fmt.Sprintf("%s_%s_%s", host, strings.Join(ipsToStrings(alternateIPs), "-"), strings.Join(alternateDNS, "-"))

certFixturePath := path.Join(fixtureDirectory, baseName+".crt")

keyFixturePath := path.Join(fixtureDirectory, baseName+".key")

if len(fixtureDirectory) > 0 {

cert, err := ioutil.ReadFile(certFixturePath)

if err == nil {

key, err := ioutil.ReadFile(keyFixturePath)

if err == nil {

return cert, key, nil

}

return nil, nil, fmt.Errorf("cert %s can be read, but key %s cannot: %v", certFixturePath, keyFixturePath, err)

}

maxAge = 100 * time.Hour * 24 * 365 // 100 years fixtures

}

caKey, err := rsa.GenerateKey(cryptorand.Reader, 2048)

if err != nil {

return nil, nil, err

}

3. 故障结论

故障原因很明显：api server启动时会自动生成一个自签名证书，而且这个自签名证书的时间为一年。

故障处理方法：重启api server（建议在一年内重启）

github上有人提过issue，官方以k8s本来就应该一年内升级且会重启为由，不认为这是一个问题：https://github.com/kubernetes/kubernetes/issues/86552

通过coredns跨k8s集群解析DNS

发表于2021年12月13日由阿辉

最近在弄AlertManager,发现其集群模式需要知道各个AlertManager节点的IP地址，而我们现在需要在2个K8s集群同时部署AlertManager上来创建一套AlertManager，以确保在某一个K8s集群挂了后报警还能发出来，同时确保当2个K8s集群都正常的时候，一个告警只发一次出来。

但怎么在A集群上找到B集群上的POD的IP，这是个问题。通过研究，我们发现CoreDns可以很好的解决这个问题。

假设我们现在有2套集群，2套集群使用的DNS后缀都是cluster.local，容器网段及service网段都不一样。

我们现在在A集群上，先看旧的CoreDNS的配置：

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        log . {combined} {
          class error
        }
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        reload
        loadbalance
        #rewrite name saas-fe-zhan.internal.com public-fe-zhan-node-dev.public-fe-node-dev.svc.cluster.local
    }

apiVersion: v1

kind: ConfigMap

metadata:

namespace: kube-system

data:

Corefile: |

.:53 {

log . {combined} {

class error

}

errors

health

ready

kubernetes cluster.local in-addr.arpa ip6.arpa {

pods insecure

fallthrough in-addr.arpa ip6.arpa

}

prometheus :9153

forward . /etc/resolv.conf

cache 30

reload

loadbalance

#rewrite name saas-fe-zhan.internal.com public-fe-zhan-node-dev.public-fe-node-dev.svc.cluster.local

}

继续阅读 →

通过Jenkins Pipeline在Dockerfile内自动构建包含私有模块的golang项目

发表于2021年11月22日由阿辉

以前写的golang项目一直使用的都是公开的module，最近使用私有gitlab仓库的module时，发现go get mod时需要输入用户名密码，开发环境还好解决，手动输一次让他记住就好了。但是项目正式上线，通过公司统一的jenkins Pipeline执行构建时，没有办法去手动输用户名密码。

Dockerfile配置

要解决这个问题，可以通过.netrc文件来处理。Dockerfile如下：

FROM mirror.ccs.tencentyun.com/library/golang:1.17 as builder

ARG GIT_USR
ARG GIT_PWD

ENV GOPROXY=https://goproxy.cn,direct
ENV GOPRIVATE=stash.xxxx.com
ENV GOOS=linux
ENV GIT_TERMINAL_PROMPT=1

WORKDIR /build
COPY . ./

RUN go env -w GOPRIVATE=stash.xxxx.com \
    && echo "machine stash.xxxx.com login ${GIT_USR} password ${GIT_PWD}" > ~/.netrc \
    && GO111MODULE=on CGO_ENABLED=0 GOOS=${GOOS} GOPROXY=${GOPROXY} go build -o=knative-audit-log

################################################################################
##                               MAIN STAGE                                   ##
################################################################################
# Copy the manager into the distroless image.
FROM mirror.ccs.tencentyun.com/library/alpine:3.13

RUN echo 'https://mirrors.cloud.tencent.com/alpine/v3.13/main' > /etc/apk/repositories \
    && echo 'https://mirrors.cloud.tencent.com/alpine/v3.13/community' >>/etc/apk/repositories \
    && apk update && apk add tzdata && ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
    && echo "Asia/Shanghai" > /etc/timezone

WORKDIR /usr/local/app
COPY --from=builder /build/knative-audit-log /usr/local/app/knative-audit-log
RUN chmod 755 /usr/local/app/knative-audit-log

ENTRYPOINT ["/usr/local/app/knative-audit-log"]

FROM mirror.ccs.tencentyun.com/library/golang:1.17 as builder

ARG GIT_USR

ARG GIT_PWD

ENV GOPROXY=https://goproxy.cn,direct

ENV GOPRIVATE=stash.xxxx.com

ENV GOOS=linux

ENV GIT_TERMINAL_PROMPT=1

WORKDIR /build

COPY . ./

RUN go env -w GOPRIVATE=stash.xxxx.com \

&& echo "machine stash.xxxx.com login ${GIT_USR} password ${GIT_PWD}" > ~/.netrc \

&& GO111MODULE=on CGO_ENABLED=0 GOOS=${GOOS} GOPROXY=${GOPROXY} go build -o=knative-audit-log

################################################################################

## MAIN STAGE ##

################################################################################

# Copy the manager into the distroless image.

FROM mirror.ccs.tencentyun.com/library/alpine:3.13

RUN echo 'https://mirrors.cloud.tencent.com/alpine/v3.13/main' > /etc/apk/repositories \

&& echo 'https://mirrors.cloud.tencent.com/alpine/v3.13/community' >>/etc/apk/repositories \

&& apk update && apk add tzdata && ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \

&& echo "Asia/Shanghai" > /etc/timezone

WORKDIR /usr/local/app

COPY --from=builder /build/knative-audit-log /usr/local/app/knative-audit-log

RUN chmod 755 /usr/local/app/knative-audit-log

ENTRYPOINT ["/usr/local/app/knative-audit-log"]

重点在3,4行和15行，3，4行通过传入GIT_USR和GIT_PWD两个参数到docker构建内，第15行生成.netrc配置。

现在我们就可以通过传参的方式来构建了：

docker build -t hub.docker.com/xxx/knative-audit-log:v0.1 --build-arg GIT_USR=**** --build-arg 'GIT_PWD=****' -f ./Dockerfile .

继续阅读 →

如何删除一个tcp长连接

发表于2021年11月15日由阿辉

有时端口被占用导致服务起不来，发现本机端口被另一个程序使用了。这个时候我们可以使用tcpkill命令来删除这个长连接，让其自动重连换一个端口。然后再启动想启动的服务。

安装tcpkill:

yum -y install dsniff --enablerepo=epel

1 2	yum -y install dsniff --enablerepo=epel

查询这个连接：

[root@sh5-saas-k8scs1-node-online-167 ~]# netstat -anp | grep 4080
tcp        0      0 10.10.182.167:4080      10.10.182.3:8443        ESTABLISHED 71932/kubelet       
unix  2      [ ACC ]     STREAM     LISTENING     2363408000 61767/containerd     /run/containerd/s/e3515d3603c24c9fdf4f6e16c493fe8953ab73538685ca8a52331356a3951649
unix  3      [ ]         STREAM     CONNECTED     2363408007 61767/containerd

[root@sh5-saas-k8scs1-node-online-167 ~]# netstat -anp | grep 4080

tcp 0 0 10.10.182.167:4080 10.10.182.3:8443 ESTABLISHED 71932/kubelet

unix 2 [ ACC ] STREAM LISTENING 2363408000 61767/containerd /run/containerd/s/e3515d3603c24c9fdf4f6e16c493fe8953ab73538685ca8a52331356a3951649

unix 3 [ ] STREAM CONNECTED 2363408007 61767/containerd

删除连接：

[root@sh5-saas-k8scs1-node-online-167 ~]# tcpkill -9 port 4080
tcpkill: listening on eth0 [port 4080]

10.10.182.167:4080 > 10.10.182.3:8443: R 1598753917:1598753917(0) win 0
10.10.182.167:4080 > 10.10.182.3:8443: R 1598754418:1598754418(0) win 0
10.10.182.167:4080 > 10.10.182.3:8443: R 1598755420:1598755420(0) win 0
10.10.182.167:4080 > 10.10.182.3:8443: R 1598756923:1598756923(0) win 0
10.10.182.167:4080 > 10.10.182.3:8443: R 1598758927:1598758927(0) win 0
10.10.182.167:4080 > 10.10.182.3:8443: R 1598761432:1598761432(0) win 0
10.10.182.167:4080 > 10.10.182.3:8443: R 1598764438:1598764438(0) win 0
10.10.182.167:4080 > 10.10.182.3:8443: R 1598767945:1598767945(0) win 0
10.10.182.167:4080 > 10.10.182.3:8443: R 1598771953:1598771953(0) win 0
10.10.182.3:8443 > 10.10.182.167:4080: R 1279911727:1279911727(0) win 0
10.10.182.3:8443 > 10.10.182.167:4080: R 1279913189:1279913189(0) win 0
10.10.182.3:8443 > 10.10.182.167:4080: R 1279916113:1279916113(0) win 0
10.10.182.3:8443 > 10.10.182.167:4080: R 1279920499:1279920499(0) win 0
10.10.182.3:8443 > 10.10.182.167:4080: R 1279926347:1279926347(0) win 0
10.10.182.3:8443 > 10.10.182.167:4080: R 1279933657:1279933657(0) win 0
10.10.182.3:8443 > 10.10.182.167:4080: R 1279942429:1279942429(0) win 0
10.10.182.3:8443 > 10.10.182.167:4080: R 1279952663:1279952663(0) win 0
10.10.182.3:8443 > 10.10.182.167:4080: R 1279964359:1279964359(0) win 0

[root@sh5-saas-k8scs1-node-online-167 ~]# tcpkill -9 port 4080

tcpkill: listening on eth0 [port 4080]

10.10.182.167:4080 > 10.10.182.3:8443: R 1598753917:1598753917(0) win 0

10.10.182.167:4080 > 10.10.182.3:8443: R 1598754418:1598754418(0) win 0

10.10.182.167:4080 > 10.10.182.3:8443: R 1598755420:1598755420(0) win 0

10.10.182.167:4080 > 10.10.182.3:8443: R 1598756923:1598756923(0) win 0

10.10.182.167:4080 > 10.10.182.3:8443: R 1598758927:1598758927(0) win 0

10.10.182.167:4080 > 10.10.182.3:8443: R 1598761432:1598761432(0) win 0

10.10.182.167:4080 > 10.10.182.3:8443: R 1598764438:1598764438(0) win 0

10.10.182.167:4080 > 10.10.182.3:8443: R 1598767945:1598767945(0) win 0

10.10.182.167:4080 > 10.10.182.3:8443: R 1598771953:1598771953(0) win 0

10.10.182.3:8443 > 10.10.182.167:4080: R 1279911727:1279911727(0) win 0

10.10.182.3:8443 > 10.10.182.167:4080: R 1279913189:1279913189(0) win 0

10.10.182.3:8443 > 10.10.182.167:4080: R 1279916113:1279916113(0) win 0

10.10.182.3:8443 > 10.10.182.167:4080: R 1279920499:1279920499(0) win 0

10.10.182.3:8443 > 10.10.182.167:4080: R 1279926347:1279926347(0) win 0

10.10.182.3:8443 > 10.10.182.167:4080: R 1279933657:1279933657(0) win 0

10.10.182.3:8443 > 10.10.182.167:4080: R 1279942429:1279942429(0) win 0

10.10.182.3:8443 > 10.10.182.167:4080: R 1279952663:1279952663(0) win 0

10.10.182.3:8443 > 10.10.182.167:4080: R 1279964359:1279964359(0) win 0

继续阅读 →

Traefik 2.5 源码分析1 – 程序启动部分

发表于2021年9月21日由阿辉

1. 前言

Traefik版本为2.5.2，从今天开始，计划陆续看完Traefik的核心部分的源码。
本文是第一部分，Traefik的程序启动分析。

2. 主流程

main.go在cmd/traefik/traefik.go内，其主要流程为：

先构造3个Command启动器，分别是traefik,healthcheck,version，Command结构如下：

源码文件：github.com/traefik/paerser@v0.1.4/cli/commands.go

package cli

type Command struct {
    Name           string
    Description    string
    Configuration  interface{}
    Resources      []ResourceLoader
    Run            func([]string) error
    CustomHelpFunc func(io.Writer, *Command) error
    Hidden         bool
    // AllowArg if not set, disallows any argument that is not a known command or a sub-command.
    AllowArg    bool
    subCommands []*Command
}

package cli

type Command struct {

Name string

Description string

Configuration interface{}

Resources []ResourceLoader

Run func([]string) error

CustomHelpFunc func(io.Writer, *Command) error

Hidden bool

// AllowArg if not set, disallows any argument that is not a known command or a sub-command.

AllowArg bool

subCommands []*Command

}

可以看到，每个Command都可以配置名字，资源，Run函数等。

然后通过cli.Execute(cmdTraefik)来启动上面3个服务,也就是调用Command内的Run函数。

ResourceLoader是用于配置的加载，有文件，参数，环境变量3种加载方法，下面会提到。

继续阅读 →

linux kernel 4.8以上内核NR_KERNEL_STACK的改变

发表于2021年3月31日由阿辉

1. NR_KERNEL_STACK是干什么用的？

通过下面的命令，我们可以查看内核栈的数量：

cat /proc/vmstat  | grep stack
nr_kernel_stack 567

cat /proc/vmstat | grep stack

nr_kernel_stack 567

这个值在内核中的宏为：NR_KERNEL_STACK，表示当前内核中有多少个内核栈。

我们使用这个数字来监控K8S节点上使用PID的数量，以避免PID被耗尽。

2. 那么PID与内核栈有什么关系呢？

在linux系统中，每个进程、子进程和线程在运行时都会有一个PID。这些进程或线程在运行时，因为CPU需要进行任务切换，在任务切换时就需要上下文交换，在上下文交换时就需要把当前进程的上下文压到内核栈内去，以便下次再运行时取出继续执行。

所以可以确定：每个进程、子进程和线程都会有一个内核栈。内核栈的数量与PID的数量大致相当。

注：基于linux内核的线程，比如java的线程与linux的线程是一一对应的，nodejs只使用了linux的进程，线程模型是其自己实现的，golang最特别，使用了多进程，每个进程上有多线程（基于内核），线程上还是自己实现的协程或者说goroute(可以理解为自己实现的线程)

继续阅读 →

PVC在K8S集群迁移或恢复需注意地方

发表于2021年3月29日由阿辉

目前在做同城双活，每个机房都有一个k8s集群，其中一些共享卷需要做到双活的2个集群上都可用。

比如我们现在在主机房的集群上有一个PVC：

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.cfs
  creationTimestamp: "2020-11-25T07:24:15Z"
  name: public-devops-teststatefulset-nginx-online-pvc
  namespace: test
  uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 30Gi
  storageClassName: cfs-hp
  volumeMode: Filesystem
  volumeName: pvc-3d65a636-f4c3-4c49-b957-c45673f48bc6

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

annotations:

pv.kubernetes.io/bind-completed: "yes"

pv.kubernetes.io/bound-by-controller: "yes"

volume.beta.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.cfs

creationTimestamp: "2020-11-25T07:24:15Z"

namespace: test

uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6

spec:

accessModes:

- ReadWriteMany

resources:

requests:

storage: 30Gi

storageClassName: cfs-hp

volumeMode: Filesystem

volumeName: pvc-3d65a636-f4c3-4c49-b957-c45673f48bc6

以及对应的PV：

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: com.tencent.cloud.csi.cfs
  creationTimestamp: "2020-11-25T07:24:22Z"
  name: pvc-3d65a636-f4c3-4c49-b957-c45673f48bc6
  uid: 301c903e-10fc-40ee-98e1-0655948425fc
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 30Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: public-devops-teststatefulset-nginx-online-pvc
    namespace: test
    resourceVersion: "657951844"
    uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6
  csi:
    driver: com.tencent.cloud.csi.cfs
    fsType: ext4
    volumeAttributes:
      fsid: cfs-3c7e956f6
      host: 10.10.144.12
      storage.kubernetes.io/csiProvisionerIdentity: 1597949457398-8081-com.tencent.cloud.csi.cfs
      storagetype: HP
      subnetid: subnet-mg9pp5o5
      vpcid: vpc-aqpp2l40
      zone: ap-shanghai-4
    volumeHandle: cfs-3c7e956f6
  persistentVolumeReclaimPolicy: Delete
  storageClassName: cfs-hp
  volumeMode: Filesystem

apiVersion: v1

kind: PersistentVolume

metadata:

annotations:

pv.kubernetes.io/provisioned-by: com.tencent.cloud.csi.cfs

creationTimestamp: "2020-11-25T07:24:22Z"

uid: 301c903e-10fc-40ee-98e1-0655948425fc

spec:

accessModes:

- ReadWriteMany

capacity:

storage: 30Gi

claimRef:

apiVersion: v1

kind: PersistentVolumeClaim

namespace: test

resourceVersion: "657951844"

uid: 3d65a636-f4c3-4c49-b957-c45673f48bc6

csi:

driver: com.tencent.cloud.csi.cfs

fsType: ext4

volumeAttributes:

fsid: cfs-3c7e956f6

host: 10.10.144.12

storage.kubernetes.io/csiProvisionerIdentity: 1597949457398-8081-com.tencent.cloud.csi.cfs

storagetype: HP

subnetid: subnet-mg9pp5o5

vpcid: vpc-aqpp2l40

zone: ap-shanghai-4

volumeHandle: cfs-3c7e956f6

persistentVolumeReclaimPolicy: Delete

storageClassName: cfs-hp

volumeMode: Filesystem

继续阅读 →

k8s源码分析：kube-apiserver –admission-control及–enable-admission-plugins –disable-admission-plugins参数差异

发表于2021年2月1日由阿辉

kubernetes源码分支：1.18

先说结论，kube-apiserver启动时：

–admission-control参数带的插件将是apiserver启动的插件，不包括默认插件
–admission-control和–enable-admission-plugins –disable-admission-plugins不能同时使用
–enable-admission-plugins参数不需要按加载顺序填写
不使用–admission-control参数时，api server会同时启动默认插件
–enable-admission-plugins参数启用时，api server会同时启动默认插件，除非使用了–disable-admission-plugins显示的关闭某个插件
–enable-admission-plugins和–disable-admission-plugins如果同时填写了某一个插件，这个插件将会被加载

继续阅读 →

istio 1.6新特性：istio自身灰度发布功能测试

发表于2020年5月27日由阿辉

1. istio 1.6灰度功能测试

istio 1.6新增了istio自身的灰度测试特性，我先测试其灰度功能。

在官网下载并解压istio 1.6，并把istioctl复制到对应目录：

[root@sh-saas-k8s1-master-dev-01 istio]# tar xfzf istio-1.6.0-linux-amd64.tar.gz
[root@sh-saas-k8s1-master-dev-01 istio]# cd istio-1.6.0
[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# cp bin/istioctl /usr/local/bin/
cp: overwrite ‘/usr/local/bin/istioctl’? y
[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# ll tools/
total 188
-rw-r--r-- 1 root root  2031 May 21 07:12 convert_RbacConfig_to_ClusterRbacConfig.sh
-rw-r--r-- 1 root root 10669 May 21 07:12 dump_kubernetes.sh
-rw-r--r-- 1 root root 88599 May 21 07:12 _istioctl
-rw-r--r-- 1 root root 85301 May 21 07:12 istioctl.bash
[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# cp tools/istioctl.bash /usr/local/bin/
cp: overwrite ‘/usr/local/bin/istioctl.bash’? y

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# istioctl version
client version: 1.6.0
control plane version: 1.5.2
data plane version: 1.5.2 (6 proxies), 1.5.1 (4 proxies)

[root@sh-saas-k8s1-master-dev-01 istio]# tar xfzf istio-1.6.0-linux-amd64.tar.gz

[root@sh-saas-k8s1-master-dev-01 istio]# cd istio-1.6.0

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# cp bin/istioctl /usr/local/bin/

cp: overwrite ‘/usr/local/bin/istioctl’? y

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# ll tools/

total 188

-rw-r--r-- 1 root root 2031 May 21 07:12 convert_RbacConfig_to_ClusterRbacConfig.sh

-rw-r--r-- 1 root root 10669 May 21 07:12 dump_kubernetes.sh

-rw-r--r-- 1 root root 88599 May 21 07:12 _istioctl

-rw-r--r-- 1 root root 85301 May 21 07:12 istioctl.bash

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# cp tools/istioctl.bash /usr/local/bin/

cp: overwrite ‘/usr/local/bin/istioctl.bash’? y

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# istioctl version

client version: 1.6.0

control plane version: 1.5.2

data plane version: 1.5.2 (6 proxies), 1.5.1 (4 proxies)

继续阅读 →

阿辉的博客

docker kubernetes istio等分布式系统和云计算等技术研究

AWS VPC CNI路由分析

1. AWS VPC CNI Agent node部分

1.1 SetupHostNetwork

1.2 setupENINetwork

2. AWS CNI二进制文件

2.1 cmdAdd

3. 参考文件

k8s apiserver 自签名证书过期导致集群故障

1. 故障处理过程

2. 故障原因排查

3. 故障结论

通过coredns跨k8s集群解析DNS

通过Jenkins Pipeline在Dockerfile内自动构建包含私有模块的golang项目

Dockerfile配置

如何删除一个tcp长连接

Traefik 2.5 源码分析1 – 程序启动部分

1. 前言

2. 主流程

linux kernel 4.8以上内核NR_KERNEL_STACK的改变

1. NR_KERNEL_STACK是干什么用的？

2. 那么PID与内核栈有什么关系呢？

PVC在K8S集群迁移或恢复需注意地方

k8s源码分析：kube-apiserver –admission-control及–enable-admission-plugins –disable-admission-plugins参数差异

istio 1.6新特性：istio自身灰度发布功能测试

1. istio 1.6灰度功能测试

2022年 5月
一	二	三	四	五	六	日
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30	31