Deploying sample helloworld.go with Istio 1.3.1 (SDS on) results in route and knative service with status IngressNotConfigured

[krishnadurai]

What version of Knative?

0.11.1

Expected Behavior

The KNative service helloworld.go should deploy with READY status True.

Actual Behavior

kubectl get ksvc,configuration,revision,route

NAME                                                           URL                                                           LATESTCREATED                            LATESTREADY                              READY     REASON
service.serving.knative.dev/helloworld-go                      http://helloworld-go.default.example.com                      helloworld-go-m9tdq                      helloworld-go-m9tdq                      Unknown   IngressNotConfigured

NAME                                                                 LATESTCREATED                            LATESTREADY                              READY   REASON
configuration.serving.knative.dev/helloworld-go                      helloworld-go-m9tdq                      helloworld-go-m9tdq                      True

NAME                                                                  CONFIG NAME                        K8S SERVICE NAME                         GENERATION   READY   REASON
revision.serving.knative.dev/helloworld-go-m9tdq                      helloworld-go                      helloworld-go-m9tdq                      1            True

NAME                                                         URL                                                           READY     REASON
route.serving.knative.dev/helloworld-go                      http://helloworld-go.default.example.com                      Unknown   IngressNotConfigured

kubectl describe ingresses.networking.internal.knative.dev helloworld-go

Name:         helloworld-go
Namespace:    default
Labels:       serving.knative.dev/route=helloworld-go
              serving.knative.dev/routeNamespace=default
              serving.knative.dev/service=helloworld-go
Annotations:  networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
API Version:  networking.internal.knative.dev/v1alpha1
Kind:         Ingress
Metadata:
  Creation Timestamp:  2020-02-10T10:15:08Z
  Generation:          1
  Owner References:
    API Version:           serving.knative.dev/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Route
    Name:                  helloworld-go
    UID:                   3400e985-4bee-11ea-9755-42010a8e0150
  Resource Version:        1540235
  Self Link:               /apis/networking.internal.knative.dev/v1alpha1/namespaces/default/ingresses/helloworld-go
  UID:                     3708b5f8-4bee-11ea-9755-42010a8e0150
Spec:
  Rules:
    Hosts:
      helloworld-go.default.svc.cluster.local
      helloworld-go.default.example.com
    Http:
      Paths:
        Retries:
          Attempts:         3
          Per Try Timeout:  10m0s
        Splits:
          Append Headers:
            Knative - Serving - Namespace:  default
            Knative - Serving - Revision:   helloworld-go-m9tdq
          Percent:                          100
          Service Name:                     helloworld-go-m9tdq
          Service Namespace:                default
          Service Port:                     80
        Timeout:                            10m0s
    Visibility:                             ExternalIP
  Visibility:                               ExternalIP
Status:
  Conditions:
    Last Transition Time:  2020-02-10T10:15:08Z
    Message:               Waiting for VirtualService to be ready
    Reason:                Uninitialized
    Status:                Unknown
    Type:                  LoadBalancerReady
    Last Transition Time:  2020-02-10T10:15:08Z
    Status:                True
    Type:                  NetworkConfigured
    Last Transition Time:  2020-02-10T10:15:08Z
    Message:               Waiting for VirtualService to be ready
    Reason:                Uninitialized
    Status:                Unknown
    Type:                  Ready
  Observed Generation:     1
Events:                    <none>

kubectl logs -n knative-serving networking-istio-7fd5c6bd64-pm6f5

{"level":"info","ts":"2020-02-10T12:52:01.153Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://helloworld-go.default.svc.cluster.local:80/, IP: 10.88.1.13:80 (depth: 2)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-02-10T12:52:01.153Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://helloworld-go.default.svc:80/, IP: 10.88.1.13:80 (depth: 1)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"info","ts":"2020-02-10T12:52:01.154Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://helloworld-go.default:80/, IP: 10.88.1.13:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"error","ts":"2020-02-10T12:52:01.161Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://helloworld-go.default.svc.cluster.local:80/ failed, IP: 10.88.1.13:80, ready: false, error: unexpected status code: want [200], got 404 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(*StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(*StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
{"level":"error","ts":"2020-02-10T12:52:01.161Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://helloworld-go.default:80/ failed, IP: 10.88.1.13:80, ready: false, error: unexpected status code: want [200], got 404 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(*StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(*StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
{"level":"error","ts":"2020-02-10T12:52:01.161Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://helloworld-go.default.svc:80/ failed, IP: 10.88.1.13:80, ready: false, error: unexpected status code: want [200], got 404 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(*StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(*StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}
{"level":"info","ts":"2020-02-10T12:52:01.218Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:331","msg":"Processing probe for http://helloworld-go.default.example.com:80/, IP: 10.88.1.13:80 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller"}
{"level":"error","ts":"2020-02-10T12:52:01.251Z","logger":"istiocontroller.ingress-controller.status-manager","caller":"ingress/status.go:366","msg":"Probing of http://helloworld-go.default.example.com:80/ failed, IP: 10.88.1.13:80, ready: false, error: unexpected status code: want [200], got 302 (depth: 0)","commit":"6b0e5c6","knative.dev/controller":"ingress-controller","stacktrace":"knative.dev/serving/pkg/reconciler/ingress.(*StatusProber).processWorkItem\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:366\nknative.dev/serving/pkg/reconciler/ingress.(*StatusProber).Start.func1\n\t/home/prow/go/src/knative.dev/serving/pkg/reconciler/ingress/status.go:268"}

kubectl get virtualservice

NAME                                    GATEWAYS                                                            HOSTS                                                                                                                                                                                                     AGE
helloworld-go                           [knative-serving/cluster-local-gateway kubeflow/kubeflow-gateway]   [helloworld-go.default helloworld-go.default.example.com helloworld-go.default.svc helloworld-go.default.svc.cluster.local]                                                                               160m
helloworld-go-mesh                      [mesh]                                                              [helloworld-go.default helloworld-go.default.svc helloworld-go.default.svc.cluster.local]                                                                                                                 160m

Inspite of the virtualservice being created for helloworld-go, the ingress fails with Waiting for VirtualService to be ready.

Steps to Reproduce the Problem

Install KNative v0.11.1
Install Istio 1.3.1 with kustomize manifests mentioned here: https:/kubeflow/manifests/tree/master/istio-1-3-1

Run helloworld.go example from: https:/knative/docs/tree/master/docs/serving/samples/hello-world/helloworld-go

[nak3]

@krishnadurai as per kserve/kserve#668 (comment), it seems that you are trying to fix it in your side. Please let us know if you need any help.

cc: @JRBANCEL

[krishnadurai]

@nak3 Sorry to have not updated the status here.

We are looking for a way to whitelist the probe requests on our side. Our Istio gateway requests are filtered through an auth mechanism and whitelisting these requests will disable the Auth checks.

[ryandawsonuk]

Do the probes use the cluster-local-gateway? Is there a knative community call we (kubeflow contributors) could join and discuss this issue?

[ZhiminXiang]

@krishnadurai could you share how the Gateway object looks like in the cluster?

[yuzisun]

@ZhiminXiang this is the gateway object and we point to this gateway in knative config-istio configmap.
gateway.kubeflow.kubeflow-gateway: istio-ingressgateway.istio-system.svc.cluster.local

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: kubeflow-gateway
  namespace: kubeflow
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP

[JRBANCEL]

I provided some more details here.

[zhanggbj]

I'm seeing a similar issue with Knative 0.12 and Istio 1.4.4.
Restarting Istio ingressgateway and pilot pod helps, but it run into this error after several hours

#7264

[tcnghia]

Knative 0.13 and 0.14 has more lenient prober #6962 .

Can you please try again with 0.14?

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen.Mark the issue as
fresh by adding the comment /remove-lifecycle stale.