我发现k8s内coredns的解析结果有点问题。经常解析不出来。
/ # nslookup kubernetes-dashboard.kube-system.svc.cluster.local
Server: 10.253.255.10
Address: 10.253.255.10:53
Non-authoritative answer:
*** Can't find kubernetes-dashboard.kube-system.svc.cluster.local: No answer
/ # nslookup kubernetes-dashboard.kube-system.svc.cluster.local
Server: 10.253.255.10
Address: 10.253.255.10:53
Name: kubernetes-dashboard.kube-system.svc.cluster.local
Address: 10.253.255.40
*** Can't find kubernetes-dashboard.kube-system.svc.cluster.local: No answer
/ # nslookup kubernetes-dashboard.kube-system.svc.cluster.local
Server: 10.253.255.10
Address: 10.253.255.10:53
Name: kubernetes-dashboard.kube-system.svc.cluster.local
Address: 10.253.255.40
但是ping kubernetes-dashboard的话，一直是可以的。实际上我通过tcpdump抓包发现每次都是有响应的。非常奇怪。
[root@-k8s1-node-dev-03 ~]# tcpdump -i eth0 udp port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:06:03.867778 IP sh-k8s1-node-dev-03.40556 > 10.253.3.9.domain: 13824+ A? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:06:03.867802 IP sh-k8s1-node-dev-03.40556 > 10.253.3.9.domain: 13824+ AAAA? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:06:03.868388 IP 10.253.3.9.domain > sh-k8s1-node-dev-03.40556: 13824* 0/1/0 (161)
14:06:03.868507 IP 10.253.3.9.domain > sh-k8s1-node-dev-03.40556: 13824* 1/0/0 A 10.253.255.40 (134)
14:06:06.367967 IP sh-k8s1-node-dev-03.40556 > 10.253.3.9.domain: 13824+ AAAA? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:06:06.368411 IP 10.253.3.9.domain > sh-k8s1-node-dev-03.40556: 13824 0/1/0 (161)
14:07:02.364036 IP sh-k8s1-node-dev-03.42791 > 10.253.4.12.domain: 14080+ A? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:07:02.364070 IP sh-k8s1-node-dev-03.42791 > 10.253.4.12.domain: 14080+ AAAA? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:07:02.364664 IP 10.253.4.12.domain > sh-k8s1-node-dev-03.42791: 14080* 0/1/0 (161)
14:07:02.364703 IP 10.253.4.12.domain > sh-k8s1-node-dev-03.42791: 14080* 1/0/0 A 10.253.255.40 (134)
14:07:04.866459 IP sh-k8s1-node-dev-03.42791 > 10.253.4.12.domain: 14080+ AAAA? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:07:04.866927 IP 10.253.4.12.domain > sh-k8s1-node-dev-03.42791: 14080 0/1/0 (161)
14:09:34.147612 IP sh-k8s1-node-dev-03.47420 > 10.253.4.12.domain: 14336+ A? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:09:34.147649 IP sh-k8s1-node-dev-03.47420 > 10.253.4.12.domain: 14336+ AAAA? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:09:34.148225 IP 10.253.4.12.domain > sh-k8s1-node-dev-03.47420: 14336* 0/1/0 (161)
14:09:34.148330 IP 10.253.4.12.domain > sh-k8s1-node-dev-03.47420: 14336* 1/0/0 A 10.253.255.40 (134)
14:09:36.649964 IP sh-k8s1-node-dev-03.47420 > 10.253.4.12.domain: 14336+ AAAA? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:09:36.650457 IP 10.253.4.12.domain > sh-k8s1-node-dev-03.47420: 14336 0/1/0 (161)
14:09:41.000317 IP sh-k8s1-node-dev-03.58056 > 10.253.3.9.domain: 14592+ A? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:09:41.000353 IP sh-k8s1-node-dev-03.58056 > 10.253.3.9.domain: 14592+ AAAA? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:09:41.000921 IP 10.253.3.9.domain > sh-k8s1-node-dev-03.58056: 14592* 0/1/0 (161)
14:09:41.001041 IP 10.253.3.9.domain > sh-k8s1-node-dev-03.58056: 14592* 1/0/0 A 10.253.255.40 (134)
14:09:43.500986 IP sh-k8s1-node-dev-03.58056 > 10.253.3.9.domain: 14592+ AAAA? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:09:43.501421 IP 10.253.3.9.domain > sh-k8s1-node-dev-03.58056: 14592 0/1/0 (161)
14:10:06.964685 IP sh-k8s1-node-dev-03.53864 > 10.253.3.9.domain: 15104+ A? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:10:06.964711 IP sh-k8s1-node-dev-03.53864 > 10.253.3.9.domain: 15104+ AAAA? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:10:06.965148 IP 10.253.3.9.domain > sh-k8s1-node-dev-03.53864: 15104 0/1/0 (161)
14:10:06.965322 IP 10.253.3.9.domain > sh-k8s1-node-dev-03.53864: 15104* 1/0/0 A 10.253.255.40 (134)
14:10:09.466963 IP sh-k8s1-node-dev-03.53864 > 10.253.3.9.domain: 15104+ AAAA? kubernetes-dashboard.kube-system.svc.cluster.local. (68)
14:10:09.467394 IP 10.253.3.9.domain > sh-k8s1-node-dev-03.53864: 15104 0/1/0 (161)
后来发现是busybox nslookup版本的问题，新版本应该是有问题。使用老版本busybox:1.28.3就好了。
[root@sh-k8s1-master-dev-01 tasks]# kubectl run busybox --rm -it --image=busybox:1.28.3 /bin/sh
If you don't see a command prompt, try pressing enter.
/ # nslookup metrics-server
Server: 10.253.255.10
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
nslookup: can't resolve 'metrics-server'
/ # nslookup kube-dns
Server: 10.253.255.10
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
nslookup: can't resolve 'kube-dns'
/ # nslookup kube-dns.kube-system.svc.cluster.local
Server: 10.253.255.10
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
Name: kube-dns.kube-system.svc.cluster.local
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
/ # nslookup kube-dns.kube-system.svc.cluster.local
Server: 10.253.255.10
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
Name: kube-dns.kube-system.svc.cluster.local
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
/ # nslookup kube-dns.kube-system.svc.cluster.local
Server: 10.253.255.10
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
Name: kube-dns.kube-system.svc.cluster.local
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
/ # nslookup kube-dns.kube-system.svc.cluster.local
Server: 10.253.255.10
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
Name: kube-dns.kube-system.svc.cluster.local
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
/ # nslookup kube-dns.kube-system.svc.cluster.local
Server: 10.253.255.10
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
Name: kube-dns.kube-system.svc.cluster.local
Address 1: 10.253.255.10 kube-dns.kube-system.svc.cluster.local
备注下，另外要注意的是新版k8s或coredns对短域名支持不是很好，使用全域名就没问题。但是不影响直接使用短域名。