Probe Envoy without using requests

[JRBANCEL]

Describe the feature

Today, Istio Envoy Pods are probed using a special requests ("K-Network-Probe" == "probe").
This works well but breaks in the following scenarios:

• Istio Policy can block those requests: see Deploying sample helloworld.go with Istio 1.3.1 (SDS on) results in route and knative service with status IngressNotConfigured #6793
  There is no way to filter policies on header content. Filtering on path is possible and we could GET a specific path, e.g. /.well-known/knative-probe, but it is exposing internals to users.
• Requests can be forwarded: see Istio + Dex (used by Kubeflow)

One way to achieve this is to fetch the Envoy configuration (GET /config_dump) directly and inspect it to find the hash that is in appendHeader today.

Pros:

• no requests going though the regular request path
• no probing handling code in activator and queue-proxy: cleaner abstraction
• no need to deal with weird Istio configuration (like Red Hat listening on a port not matching the Gateway)
• no need to handle the multiple protocol configurations (HTTP, HTTP redirect, HTTP2, HTTPS)

Cons:

• parsing /config_dump output is a lot heavier than checking for HTTP 200, but we probably don't need to parse it, we can simply scan it for the string we want

/cc @mattmoor @tcnghia @markusthoemmes @yuzisun

[JRBANCEL]

/assign

[mattmoor]

no requests going though the regular request path

Are we sure this is a Pro and not a Con?

and inspect it to find the hash that is in appendHeader today.

This is insufficient, you also need to verify that the endpoints referenced aren't empty.

[JRBANCEL]

no requests going though the regular request path

Are we sure this is a Pro and not a Con?

It is a Pro to not have to deal with Istio Policies and Envoy filters.

and inspect it to find the hash that is in appendHeader today.

This is insufficient, you also need to verify that the endpoints referenced aren't empty.

It depends, either we want probing to:

1. validate that the Envoy configuration is correct: in this case going through the regular request path is required, but not sufficient. That's not what probing does today since as soon as one request succeeds (across protocols and across hosts), the Envoy Pod is considered ready.
2. check if the Envoy configuration has been applied: in this case, going through the regular request path is not required and finding the version hash in the config (without parsing it) is sufficient.

We do 2) today, and achieving 1) is tricky.

[dprotaso]

FYI - Pilot has an endpoint that tracks envoy configuration rollout - that might be worth experimenting with
https://istio.io/docs/reference/commands/istioctl/#istioctl-experimental-wait

[JRBANCEL]

I looked at how istioctl works and it is a bit more complicated than simple HTTP requests on the Envoy admin port because the admin port is not exposed outside of the Pod, only on loopback.
istioctl uses exec on the Pod to extract information.
For example: https:/istio/istio/blob/master/istioctl/pkg/kubernetes/client.go#L100

We can reuse this, but it is not exactly lightweight and we'll probably run into permission issues.

[JRBANCEL]

This is no possible to do efficiently and easily (see previous comment)
#6962 should be enough for most scenarios.
/close