1. istio 1.6灰度功能测试

istio 1.6新增了istio自身的灰度测试特性，我先测试其灰度功能。

在官网下载并解压istio 1.6，并把istioctl复制到对应目录：

[root@sh-saas-k8s1-master-dev-01 istio]# tar xfzf istio-1.6.0-linux-amd64.tar.gz
[root@sh-saas-k8s1-master-dev-01 istio]# cd istio-1.6.0
[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# cp bin/istioctl /usr/local/bin/
cp: overwrite ‘/usr/local/bin/istioctl’? y
[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# ll tools/
total 188
-rw-r--r-- 1 root root  2031 May 21 07:12 convert_RbacConfig_to_ClusterRbacConfig.sh
-rw-r--r-- 1 root root 10669 May 21 07:12 dump_kubernetes.sh
-rw-r--r-- 1 root root 88599 May 21 07:12 _istioctl
-rw-r--r-- 1 root root 85301 May 21 07:12 istioctl.bash
[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# cp tools/istioctl.bash /usr/local/bin/
cp: overwrite ‘/usr/local/bin/istioctl.bash’? y

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# istioctl version
client version: 1.6.0
control plane version: 1.5.2
data plane version: 1.5.2 (6 proxies), 1.5.1 (4 proxies)

1.1 安装控制面板

通过设置revision=canary变量来安装灰度的控制面板：

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# istioctl install --set revision=canary
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Addons installed
✔ Installation complete

安装完后，可以看到带有canary的istiod：

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# kubectl get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
grafana-556b649566-twdk2               1/1     Running   0          46d
istio-egressgateway-6fdd4f8558-7mnzw   1/1     Running   0          18d
istio-ingressgateway-9fd45f487-grz8g   1/1     Running   0          3m55s
istio-tracing-7cf5f46848-lhjf7         1/1     Running   0          52d
istiod-77fd8dc44c-2krmf                1/1     Running   0          3m42s
istiod-77fd8dc44c-9wd95                1/1     Running   0          15d
istiod-canary-65fd47bf6f-tcl9n         1/1     Running   0          4m10s
kiali-66bcf44569-68nxm                 1/1     Running   0          18d
prometheus-7ff5fb8799-2hlwr            2/2     Running   0          3m55s

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# kubectl -n istio-system get svc -lapp=istiod
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                AGE
istiod          ClusterIP   10.253.252.222   <none>        15012/TCP,443/TCP                                      52d
istiod-canary   ClusterIP   10.253.204.108   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP,53/UDP,853/TCP   4m33s

sidecar injector也有了一个新的灰度版本：

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# kubectl get mutatingwebhookconfigurations
NAME                            CREATED AT
istio-sidecar-injector          2020-04-04T18:05:07Z
istio-sidecar-injector-canary   2020-05-27T07:58:59Z

1.2 配置数据面板

下面我们使用public-ops-tomcat-dev这个命名空间来做灰度测试。

先看一下命名空间的配置：

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# kubectl describe ns public-ops-tomcat-dev 
Name:         public-ops-tomcat-dev
Labels:       istio-injection=enabled
Annotations:  <none>
Status:       Active

No resource quota.

Resource Limits
 Type       Resource           Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
 ----       --------           ---  ---  ---------------  -------------  -----------------------
 Container  ephemeral-storage  -    -    256Mi            7Gi            -

可以看到一个标准的注入标签：istio-injection=enabled，要想配置为灰度测试，需要删除istio-injection标签，打上新标签：istio.io/rev=canary

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# kubectl label namespace public-ops-tomcat-dev istio-injection- istio.io/rev=canary
namespace/public-ops-tomcat-dev labeled

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# kubectl describe ns public-ops-tomcat-dev 
Name:         public-ops-tomcat-dev
Labels:       istio.io/rev=canary
Annotations:  <none>
Status:       Active

No resource quota.

Resource Limits
 Type       Resource           Min  Max  Default Request  Default Limit  Max Limit/Request Ratio
 ----       --------           ---  ---  ---------------  -------------  -----------------------
 Container  ephemeral-storage  -    -    256Mi            7Gi            -

重启pod，让配置生效，如果kubectl 的版本 >= 1.15:

kubectl rollout restart deployment -n public-ops-tomcat-dev

否则可以一个一个的删除pod：

[root@sh-saas-k8s1-master-dev-01 ~]# kubectl delete pod -n public-ops-tomcat-dev public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-nwcsg 
pod "public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-nwcsg" deleted
[root@sh-saas-k8s1-master-dev-01 ~]# kubectl delete pod -n public-ops-tomcat-dev public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-wfgsr 
pod "public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-wfgsr" deleted
[root@sh-saas-k8s1-master-dev-01 ~]# 

查看是否是新的版本的sidecar:

[root@sh-saas-k8s1-master-dev-01 ~]# kubectl get pods -n public-ops-tomcat-dev  -l istio.io/rev=canary       
NAME                                                   READY   STATUS    RESTARTS   AGE
public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-m4nm4   2/2     Running   0          79s
public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-m6x72   2/2     Running   0          60s

[root@sh-saas-k8s1-master-dev-01 ~]# istioctl proxy-config -n public-ops-tomcat-dev endpoints public-ops-dubbo-demo-web-tomcat-dev-79f758dcf-m6x72.public-ops-tomcat-dev --cluster xds-grpc -ojson | grep hostname            
                "hostname": "istiod-canary.istio-system.svc",

1.3 灰度版本的问题

• 怎么把灰度版本变成主版本？
• 怎么清理旧版本或通过升级旧版本后清理灰度版本？
• 目前发现Ingress Gateway在安装灰度版本时是直接升级了的，需要特别注意。

2. istio 升级到1.6

在创建了灰度的版本后，再升级就有些问题，会报错：

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# istioctl manifest versions

Binary version is 1.6.0.

This version of istioctl can:
  Install Istio 1.6.0
  Update Istio from >=1.5.0 to 1.6.0
  Update Istio from  <1.7 to 1.6.0

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# istioctl upgrade -y -f custom_profile.yaml 
2020-05-28T08:18:30.890158Z     info    proto: tag has too few fields: "-"
! global.proxy.accessLogFile is deprecated; use meshConfig.accessLogFile instead
Control Plane - egressgateway pod - istio-egressgateway-6fdd4f8558-7mnzw - version: 1.5.2
Control Plane - ingressgateway pod - istio-ingressgateway-9fd45f487-grz8g - version: 1.6.0
Control Plane - pilot pod - istiod-77fd8dc44c-9wd95 - version: 1.5.2
Control Plane - istiod pod - istiod-canary-65fd47bf6f-tcl9n - version: 1.6.0

2020-05-28T08:18:30.969886Z     info    Error: failed to read the current Istio version, error: different versions of Istio components found: [egressgateway pod - istio-egressgateway-6fdd4f8558-7mnzw - version: 1.5.2 ingressgateway pod - istio-ingressgateway-9fd45f487-grz8g - version: 1.6.0 pilot pod - istiod-77fd8dc44c-9wd95 - version: 1.5.2 istiod pod - istiod-canary-65fd47bf6f-tcl9n - version: 1.6.0]

Error: failed to read the current Istio version, error: different versions of Istio components found: [egressgateway pod - istio-egressgateway-6fdd4f8558-7mnzw - version: 1.5.2 ingressgateway pod - istio-ingressgateway-9fd45f487-grz8g - version: 1.6.0 pilot pod - istiod-77fd8dc44c-9wd95 - version: 1.5.2 istiod pod - istiod-canary-65fd47bf6f-tcl9n - version: 1.6.0]

istio官网已经有人提交了issue,https://github.com/istio/istio/issues/23923 有人说是istio 1.6+才能使用灰度。原版本1.5.x是不支持的。这样的话只能先删除，再重新安装istio 1.6

删除旧的istio:

istioctl manifest generate --set revision=canary  | kubectl delete -f -

istioctl manifest generate -f custom_profile.yaml  | kubectl delete -f -

重新安装：

[root@sh-saas-k8s1-master-dev-01 istio-1.6.0]# istioctl manifest apply -f custom_profile.yaml
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
! global.proxy.accessLogFile is deprecated; use meshConfig.accessLogFile instead
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Egress gateways installed
✔ Addons installed
✔ Installation complete