Isolate private ksvc using different envoy listener ports

[lionelvillard]

Add support for multiple private ingresses backed by multiple envoy listeners (each with a unique port).

This is an experimental Kourier-only feature guarded by a flag in config-kourier:

# Control the desired level of incoming traffic isolation.
#
# When set to the empty value (default), all incoming traffic flows through
# a shared ingress and listeners.
#
# When set to "port", incoming traffic is isolated by using different
# listener ports.
#
# NOTE: This flag is in an alpha state.
traffic-isolation: ""

The selection of which envoy listener to use is done at the namespace-level. The traffic of all internal services within the same namespace is redirected to the same envoy listener. Here are the annotation for mapping a namespace to a particular envoy listener:

// ListenerPortAnnotationKey is the annotation key for assigning the ingress to a particular
// envoy listener port. Only applicable to internal services.
ListenerPortAnnotationKey = "kourier.knative.dev/listener-port"

The allocation/assignment of a namespace to an envoy listener is an orthogonal concern and not address in this PR. This can be done by manually editing namespace annotations or automatically by a Tenant controller.

Each envoy listener is exposed using a k8s service (e.g kourier-isolation-9000). This PR does not automatically create/delete these services since this concern is also related to tenant management.

Additional context:

• Feature Track
• PoC showing how Network Policies with this PR can be used to isolate tenants.

Changes

• Add a new configuration parameter for enabling port-level internal services isolation
• Generate an envoy configuration containing multiple internal services

/kind enhancement

Fixes #

Release Note

Add a new configuration parameter for enabling port-level internal services isolation. See documentation for more details.

Docs

Remaining tasks are tracked in this issue.

/cc @evankanderson @nak3 @rhuss

[codecov]

Codecov Report

Merging #852 (8791e3f) into main (34458cb) will decrease coverage by 0.82%.
The diff coverage is 62.74%.

@@            Coverage Diff             @@
##             main     #852      +/-   ##
==========================================
- Coverage   81.60%   80.77%   -0.83%     
==========================================
  Files          18       18              
  Lines        1185     1233      +48     
==========================================
+ Hits          967      996      +29     
- Misses        174      190      +16     
- Partials       44       47       +3     

Impacted Files	Coverage Δ
pkg/config/config.go	0.00% <0.00%> (ø)
pkg/generator/ingress_translator.go	82.53% <26.66%> (-3.03%)	⬇️
pkg/generator/caches.go	72.13% <81.25%> (+0.89%)	⬆️
pkg/config/configmap.go	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34458cb...8791e3f. Read the comment docs.

[skonto]

@lionelvillard hi, could you add a link to the ft doc in the description?

[lionelvillard]

@lionelvillard hi, could you add a link to the ft doc in the description?

done.

[lionelvillard]

@skonto can you take a look at this PR? I'm trying to see why the upgrade tests are failing. The log is not very helpful and these tests cannot run on a Mac.

[lionelvillard]

/hold

[lionelvillard]

@skonto

Can we run knative/serving#13094 also with the feature enabled?

Yes. knative/serving#13112

We need to have an e2e test somewhere, could you add one here at least.

I'm looking at this now, it seems easy enough but due to the time constraint, I would rather do this in a separate PR.

IMO, as soon as both knative/serving#13094 and knative/serving#13112 are green, it should be safe to merge this PR.

Also is there a plan for the tenant controller here?

No plans. This is a much bigger topic!

Let's create a list of the remaining tasks pls (tracking issue). In this description of this issue all tasks seem done. Let's add the tenant controller idea and the webhook validation too. Also since we plan to iterate on this let's add the question about label vs annotation there. @lionelvillard could you do this before merging?

Yes I can create an Github issue tracking all remaining tasks

[lionelvillard]

/hold I need to verify the PoC is still working.

[lionelvillard]

/unhold PoC is working.

[lionelvillard]

@evankanderson @skonto @nak3 @rhuss Please can you take another look? Since the last round of reviews I:

• removed the listener label. The Kourier services exposing the envoy listeners are now called kourier-isolation-<listener-port>
• remove "none" in the configmap in favor of the empty string value
• opened various issues to track the remaining tasks

I'm tracking knative/serving#13094 and knative/serving#13112. So far it looks good.

I'm working on e2e tests (in a separate PR). I don't know if I will have time to finish before the 1.6 release (which is tomorrow).

[evankanderson]

A few notes for the next PR, but happy to get this in for 1.6.

/lgtm
/approve

[knative-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: evankanderson, lionelvillard

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [evankanderson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[lionelvillard]

/unhold

[lionelvillard]

@evankanderson can you please TAL? All tests are green (including the Serving ones).

[evankanderson]

/lgtm