nekop's blog

OpenShift / JBoss / WildFly / Infinispanの中の人 http://twitter.com/nekop

Red Hat OpenShift Service Meshをインストールしてみる

Red Hat OpenShift Service Meshがプレビューリリースしたのでインストールしてみます。OpenShift向け製品版のIstioです。

セットアップはこんな感じ。

# Enable admission webhooks in master
MASTER_CONFIG_PATCH="admissionConfig:
  pluginConfig:
    MutatingAdmissionWebhook:
      configuration:
        apiVersion: v1
        disable: false
        kind: DefaultAdmissionConfig
    ValidatingAdmissionWebhook:
      configuration:
        apiVersion: v1
        disable: false
        kind: DefaultAdmissionConfig"
sudo cp -a /etc/origin/master/master-config.yaml{,.prepatch}
sudo oc ex config patch /etc/origin/master/master-config.yaml.prepatch -p "$MASTER_CONFIG_PATCH" | sudo tee /etc/origin/master/master-config.yaml
sudo /usr/local/bin/master-restart api
sudo /usr/local/bin/master-restart controllers

# sysctl vm.max_map_count=262144 on each node
echo "vm.max_map_count = 262144" | sudo tee -a /etc/sysctl.d/99-elasticsearch.conf
sudo sysctl vm.max_map_count=262144

# Install istio
curl -LO https://raw.githubusercontent.com/Maistra/openshift-ansible/maistra-0.1.0-ocp-3.1.0-istio-1.0.0/istio/istio_product_operator_template.yaml
oc new-project istio-operator
oc new-app -f istio_product_operator_template.yaml --param=OPENSHIFT_ISTIO_MASTER_PUBLIC_URL=https://s310.example.com:8443
cat <<EOF | oc create -n istio-operator -f -
apiVersion: "istio.openshift.com/v1alpha1"
kind: "Installation"
metadata:
  name: "istio-installation"
spec:
  jaeger:
    elasticsearch_memory: 1Gi
EOF

セットアップ直後の状態はこうなります。

$ oc get all -n istio-operator
NAME                                  READY     STATUS    RESTARTS   AGE
pod/istio-operator-5df6cbf496-tlrfn   1/1       Running   0          18m

NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/istio-operator   ClusterIP   172.30.186.9   <none>        60000/TCP   18m

NAME                             DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/istio-operator   1         1         1            1           18m

NAME                                        DESIRED   CURRENT   READY     AGE
replicaset.apps/istio-operator-5df6cbf496   1         1         1         18m


$ oc get all -n istio-system
NAME                                              READY     STATUS              RESTARTS   AGE
pod/openshift-ansible-istio-installer-job-j2c6m   0/1       ContainerCreating   0          58s

NAME                                              DESIRED   SUCCESSFUL   AGE
job.batch/openshift-ansible-istio-installer-job   1         0            58s

openshift-ansible-istio-installer-job podはAnsibleを実行しているようです。この実行が終わったらセットアップ完了で以下の状態になります。

$ oc get all -n istio-system
NAME                                              READY     STATUS      RESTARTS   AGE
pod/elasticsearch-0                               1/1       Running     0          8m
pod/grafana-6d5c5477-rbskl                        1/1       Running     0          19m
pod/istio-citadel-6f9c778bb6-trf6k                1/1       Running     0          21m
pod/istio-egressgateway-957857444-dx26j           1/1       Running     0          21m
pod/istio-galley-c47f5dffc-dn25p                  1/1       Running     0          21m
pod/istio-ingressgateway-7db86747b7-l86zp         1/1       Running     0          21m
pod/istio-pilot-5646d7786b-s29kv                  2/2       Running     0          21m
pod/istio-policy-7d694596c6-698v5                 2/2       Running     0          21m
pod/istio-sidecar-injector-57466d9bb-z6vdv        1/1       Running     0          21m
pod/istio-statsd-prom-bridge-7f44bb5ddb-2d75m     1/1       Running     0          21m
pod/istio-telemetry-7cf7b4b77c-6vxn4              2/2       Running     0          21m
pod/jaeger-agent-9f4xz                            1/1       Running     0          18m
pod/jaeger-collector-9c9f8bc66-7278h              1/1       Running     7          18m
pod/jaeger-query-fdc6dcd74-v9t5c                  1/1       Running     7          18m
pod/openshift-ansible-istio-installer-job-j2c6m   0/1       Completed   0          25m
pod/prometheus-84bd4b9796-wwfms                   1/1       Running     0          21m

NAME                               TYPE           CLUSTER-IP       EXTERNAL-IP                   PORT(S)                                                                                                     AGE
service/elasticsearch              ClusterIP      172.30.38.243    <none>                        9200/TCP                                                                                                    19m
service/elasticsearch-cluster      ClusterIP      172.30.117.160   <none>                        9300/TCP                                                                                                    19m
service/grafana                    ClusterIP      172.30.182.60    <none>                        3000/TCP                                                                                                    19m
service/istio-citadel              ClusterIP      172.30.169.27    <none>                        8060/TCP,9093/TCP                                                                                           21m
service/istio-egressgateway        ClusterIP      172.30.177.77    <none>                        80/TCP,443/TCP                                                                                              21m
service/istio-galley               ClusterIP      172.30.22.227    <none>                        443/TCP,9093/TCP                                                                                            21m
service/istio-ingressgateway       LoadBalancer   172.30.253.232   172.29.93.241,172.29.93.241   80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:32718/TCP,8060:30594/TCP,15030:30606/TCP,15031:32105/TCP   21m
service/istio-pilot                ClusterIP      172.30.111.153   <none>                        15010/TCP,15011/TCP,8080/TCP,9093/TCP                                                                       21m
service/istio-policy               ClusterIP      172.30.49.238    <none>                        9091/TCP,15004/TCP,9093/TCP                                                                                 21m
service/istio-sidecar-injector     ClusterIP      172.30.11.70     <none>                        443/TCP                                                                                                     21m
service/istio-statsd-prom-bridge   ClusterIP      172.30.48.15     <none>                        9102/TCP,9125/UDP                                                                                           21m
service/istio-telemetry            ClusterIP      172.30.72.184    <none>                        9091/TCP,15004/TCP,9093/TCP,42422/TCP                                                                       21m
service/jaeger-collector           ClusterIP      172.30.129.165   <none>                        14267/TCP,14268/TCP,9411/TCP                                                                                18m
service/jaeger-query               LoadBalancer   172.30.244.29    172.29.154.76,172.29.154.76   80:32087/TCP                                                                                                18m
service/prometheus                 ClusterIP      172.30.49.188    <none>                        9090/TCP                                                                                                    21m
service/tracing                    LoadBalancer   172.30.115.191   172.29.75.1,172.29.75.1       80:30290/TCP                                                                                                17m
service/zipkin                     ClusterIP      172.30.202.148   <none>                        9411/TCP                                                                                                    18m

NAME                          DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/jaeger-agent   1         1         1         1            1           <none>          18m

NAME                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana                    1         1         1            1           19m
deployment.apps/istio-citadel              1         1         1            1           21m
deployment.apps/istio-egressgateway        1         1         1            1           21m
deployment.apps/istio-galley               1         1         1            1           21m
deployment.apps/istio-ingressgateway       1         1         1            1           21m
deployment.apps/istio-pilot                1         1         1            1           21m
deployment.apps/istio-policy               1         1         1            1           21m
deployment.apps/istio-sidecar-injector     1         1         1            1           21m
deployment.apps/istio-statsd-prom-bridge   1         1         1            1           21m
deployment.apps/istio-telemetry            1         1         1            1           21m
deployment.apps/jaeger-collector           1         1         1            1           18m
deployment.apps/jaeger-query               1         1         1            1           18m
deployment.apps/prometheus                 1         1         1            1           21m

NAME                                                  DESIRED   CURRENT   READY     AGE
replicaset.apps/grafana-6d5c5477                      1         1         1         19m
replicaset.apps/istio-citadel-6f9c778bb6              1         1         1         21m
replicaset.apps/istio-egressgateway-957857444         1         1         1         21m
replicaset.apps/istio-galley-c47f5dffc                1         1         1         21m
replicaset.apps/istio-ingressgateway-7db86747b7       1         1         1         21m
replicaset.apps/istio-pilot-5646d7786b                1         1         1         21m
replicaset.apps/istio-policy-7d694596c6               1         1         1         21m
replicaset.apps/istio-sidecar-injector-57466d9bb      1         1         1         21m
replicaset.apps/istio-statsd-prom-bridge-7f44bb5ddb   1         1         1         21m
replicaset.apps/istio-telemetry-7cf7b4b77c            1         1         1         21m
replicaset.apps/jaeger-collector-9c9f8bc66            1         1         1         18m
replicaset.apps/jaeger-query-fdc6dcd74                1         1         1         18m
replicaset.apps/prometheus-84bd4b9796                 1         1         1         21m

NAME                             DESIRED   CURRENT   AGE
statefulset.apps/elasticsearch   1         1         19m

NAME                                              DESIRED   SUCCESSFUL   AGE
job.batch/openshift-ansible-istio-installer-job   1         1            25m

NAME                                            HOST/PORT                                              PATH      SERVICES               PORT              TERMINATION   WILDCARD
route.route.openshift.io/grafana                grafana-istio-system.apps.s310.example.com                       grafana                http                            None
route.route.openshift.io/istio-ingressgateway   istio-ingressgateway-istio-system.apps.s310.example.com          istio-ingressgateway   http2                           None
route.route.openshift.io/jaeger-query           jaeger-query-istio-system.apps.s310.example.com                  jaeger-query           jaeger-query      edge          None
route.route.openshift.io/prometheus             prometheus-istio-system.apps.s310.example.com                    prometheus             http-prometheus                 None
route.route.openshift.io/tracing                tracing-istio-system.apps.s310.example.com                       tracing                tracing           edge          None

アプリケーションを作ってみます。Istioではinit containerでiptablesを利用するのでprivilegedコンテナにする必要があるのですが、コミュニティ版IstioをOpenShiftで利用するときにinit containerをprivilegedとしてinjectionしてくれない、という問題がありました。OpenShift製品版のIstioはprivilegedとしてinjectionしてくれるようです。

oc new-project test-istio
oc adm policy add-scc-to-user privileged -z default
oc new-app https://github.com/nekop/hello-sinatra
oc patch dc/hello-sinatra -p 'spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"'

namespaceのlabel istio-injection=enabledでのinjectionも軽く試してみたのですが、そちらは動作していないように見えます。OpenShiftではbuild podにinjectionされると問題となる(ビルドに必要な通信がIstioで許可されていなくて失敗するなど)ので、その関係で無効化されていそうです。

再デプロイされたpodはinit containerとproxy sidecarがinjectionされています。

$ oc get pod
NAME                    READY     STATUS      RESTARTS   AGE
hello-sinatra-1-build   0/1       Completed   0          9m
hello-sinatra-2-9p6gx   2/2       Running     0          7m
$ curl hello-sinatra.test-istio.svc:8080
hello

Prometheusを開いてhello_sinatraと入力したときに各種メトリクスが見えるようになっていれば問題なく動作しています。