E2E testing a KIC built from HEAD by CI. (#869)

fixes #694 

- makes CI build a local KIC image from HEAD
- if on `main` or `next`: pushes to bintray. Can (or, maybe, should) be extended to all pushes
- spins up a local microk8s cluster
- pushes the local KIC to the local image registry
- runs an instance of KIC
- runs a set of tests: applying a bunch of manifests, waiting (open loop - `sleep 6`) for KIC to do its work, asserting on `curl` results
    - There are several possible ways to mitigate the `sleep` but none of them is perfect:
        - watch the `status` field of created resources
        - watch KIC logs for a successful/failed sync
        - watch the Kong Admin API `/config` endpoint

* test(integration): implement harness and basic tests

* test(integration): run on ci

* chore(test): improve test cleanup

* chore(test): switch from microk8s to kind

* chore(makefile): define target `integration-test`

* test(integration): patch instead of sed, disable anonymous reports

* test(e2e): replace kubectl patch with kustomization

* test(e2e): bump kubectl wait timeout to account for slow pulls

* test(e2e): remove unused PROXY_IP variable

* test(e2e): write README

* test(e2e): add shebang to verify.sh files

* test(e2e): gitignore leftover stuff

.github/workflows/build-test.yaml

@@ -0,0 +1,80 @@
+name: Test
+
+on: [push]
+
+jobs:
+ build:
+ runs-on: ubuntu-latest
+ steps:
+ - name: Setup go
+ uses: actions/setup-go@v2
+ with:
+ go-version: '^1.15'
+
+ - name: Cache Go modules
+ uses: actions/cache@v1
+ with:
+ path: ~/go/pkg/mod
+ key: ${{ runner.os }}-build-codegen-${{ hashFiles('**/go.sum') }}
+ restore-keys: |
+ ${{ runner.os }}-go
+
+ - name: 'Check out the repo'
+ uses: actions/checkout@v2
+
+ - name: 'make container'
+ run: |
+ make REGISTRY=local IMGNAME=kong-ingress-controller TAG=ci container
+
+ - run: docker save local/kong-ingress-controller:ci > image.tar
+
+ - uses: actions/upload-artifact@v2
+ with:
+ name: image
+ path: image.tar
+
+ push-public:
+ needs: build
+ runs-on: ubuntu-latest
+ if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/next'
+
+ steps:
+ - uses: actions/download-artifact@v2
+ with:
+ name: image
+
+ - name: 'Arrange: Push the Kong Operator image to the public registry'
+ env:
+ BINTRAY_KEY: ${{ secrets.BINTRAY_KEY }}
+ BINTRAY_USER: ${{ secrets.BINTRAY_USER }}
+ run: |
+ set -e
+
+ REPO="kong-docker-kubernetes-ingress-controller.bintray.io"
+ REF="$(git rev-parse --short HEAD)"
+
+ DOCKER_TAG="$REPO/master:$REF"
+ docker load < image.tar
+ docker tag local/kong-ingress-controller:ci "$DOCKER_TAG"
+ echo "${BINTRAY_KEY}" | docker login -u "${BINTRAY_USER}" ${REPO} --password-stdin
+ docker push "$DOCKER_TAG"
+
+ test-k8s:
+ needs: build
+ runs-on: ubuntu-latest
+ steps:
+ - name: 'Arrange: Check out the repo'
+ uses: actions/checkout@v2
+
+ - uses: actions/download-artifact@v2
+ with:
+ name: image
+
+ - name: 'Arrange: Push the image to the local registry'
+ run: |
+ docker load < image.tar
+ docker tag local/kong-ingress-controller:ci localhost:32000/kong-ingress-controller:ci
+
+ - name: 'Test: Run integration tests'
+ run: |
+ env KIC_IMAGE=local/kong-ingress-controller:ci ./test/integration/test.sh

.gitignore

@@ -37,3 +37,7 @@ kong-ingress-controller
 
 # ignore vendor tree
 vendor
+
+# integration test residuals
+kind
+kubeconfig-test-cluster

CONTRIBUTING.md

@@ -41,7 +41,7 @@ patch](#submitting-a-patch) section.
 
 ## Dependencies
 
-The build uses dependencies are managed by [go modules](https://blog.golang.org/using-go-modules) 
+The build uses dependencies are managed by [go modules](https://blog.golang.org/using-go-modules)
 
 ## Running in dev mode
 
@@ -126,6 +126,8 @@ $ cd $GOPATH/src/github.com/kong/kubernetes-ingress-controller
 $ make test
 ```
 
+To run integration tests, see the [integration test readme](test/integration/README.md).
+
 ## Releasing
 
 Makefile will produce a release binary, as shown above. To publish this

Makefile

@@ -50,3 +50,7 @@ container:
 .PHONY: run
 run:
  ./hack/dev/start.sh ${DB} ${RUN_VERSION}
+
+.PHONY: integration-test
+integration-test: container
+ KIC_IMAGE="${IMAGE}:${TAG}" ./test/integration/test.sh

test/integration/README.md

@@ -0,0 +1,68 @@
+## E2E test
+
+The end-to-end (E2E) integration test performs the following:
+
+1. Start up an ephemeral Kubernetes cluster (KIND)
+1. Install a container registry (localhost:5000)
+1. Push the Kong Ingress Controller image to localhost:5000/kic:local
+1. Run Kong Ingress Controller in the local Kubernetes cluster
+1. Start up example services (echo and httpbin)
+1. Run test cases (`/test/integration/cases/*`):
+ 1. Apply manifests from `.yaml` files present in the test case directory
+  1. Wait for KIC to apply the configuration
+  1. Run ./verify.sh to verify assertions
+  1. Delete manifests created in the apply step above.
+
+### How to run the test
+
+```bash
+make integration-test
+```
+
+This command builds a KIC Docker image and runs the test against that image.
+
+It is possible to run the test against any prebuilt KIC image, skipping the build step:
+
+```bash
+env KIC_IMAGE=some-kic-image:tag ./test/integration/test.sh
+```
+
+### Troubleshooting
+
+If you want to troubleshoot a specific test case, here's how to do it:
+
+#### Run tests with `SKIP_TEARDOWN=yes`
+By passing `SKIP_TEARDOWN=yes` to the test you can inspect the test environment after failure, and run certain test cases manually:
+
+```bash
+make SKIP_TEARDOWN=yes integration-test
+# or
+env KIC_IMAGE=some-kic-image:tag SKIP_TEARDOWN=yes ./test/integration/test.sh
+```
+
+#### Access the test cluster with `kubectl`
+
+After the test invocation command with `SKIP_TEARDOWN=yes` set terminates, KIND will continue running in the cluster. You can access it using `./kubeconfig-test-cluster` as kubeconfig:
+
+```bash
+kubectl --kubeconfig=./kubeconfig-test-cluster get pods --all-namespaces
+```
+
+#### Run a test case manually
+
+You can run a test case manually:
+```bash
+kubectl --kubeconfig=./kubeconfig-test-cluster port-forward -n kong svc/kong-proxy "27080:80" "27443:443"
+
+# in a separate terminal window:
+kubectl --kubeconfig=./kubeconfig-test-cluster apply -f ./test/integration/cases/01-https
+env SUT_HTTP_HOST=127.0.0.1:27080 SUT_HTTPS_HOST=127.0.0.1:27443 ./test/integration/cases/01-https/verify.sh
+kubectl --kubeconfig=./kubeconfig-test-cluster delete -f ./test/integration/cases/01-https
+```
+
+#### Manually tear down the test cluster
+
+At the end of the debugging session, you can tear down the environment like this:
+```bash
+docker rm -f test-cluster-control-plane test-local-registry
+```

test/integration/cases/00-ingress-match-type/ingress.yaml

@@ -0,0 +1,44 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: echo
+ annotations:
+ kubernetes.io/ingress.class: kong
+spec:
+ rules:
+ - http:
+ paths:
+ - backend:
+ service:
+ name: echo
+ port:
+ number: 8080
+ path: /foo
+ pathType: Prefix
+ - http:
+ paths:
+ - backend:
+ service:
+ name: echo
+ port:
+ number: 8080
+ path: /bar1
+ pathType: Exact
+ - http:
+ paths:
+ - backend:
+ service:
+ name: echo
+ port:
+ number: 8080
+ path: /bar2/
+ pathType: Exact
+ - http:
+ paths:
+ - backend:
+ service:
+ name: echo
+ port:
+ number: 8080
+ path: "/baz+/"
+ pathType: ImplementationSpecific

test/integration/cases/00-ingress-match-type/verify.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+set -ex
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/)" == 404 ]
+
+# Match type: Prefix
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/foo)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/foo/)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/fooo)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/fooo/)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/fooo/xxx)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/foo/xxx)" == 200 ]
+
+# Match type: Exact
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar1)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar1/)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar1/xxx)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar2)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar2/)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar2/xxx)" == 404 ]
+
+# Match type: ImplementationSpecific
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/baz)" == 404 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/baz/)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bazzz/)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bazzz/xxx)" == 200 ]

test/integration/cases/01-https/ingress.yaml

@@ -0,0 +1,37 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: echo-http
+ annotations:
+ kubernetes.io/ingress.class: kong
+ konghq.com/protocols: http
+spec:
+ rules:
+ - http:
+ paths:
+ - backend:
+ service:
+ name: echo
+ port:
+ number: 8080
+ path: /foo
+ pathType: Prefix
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: echo-https
+ annotations:
+ kubernetes.io/ingress.class: kong
+ konghq.com/protocols: https
+spec:
+ rules:
+ - http:
+ paths:
+ - backend:
+ service:
+ name: echo
+ port:
+ number: 8080
+ path: /bar
+ pathType: Prefix

test/integration/cases/01-https/verify.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+set -ex
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/foo)" == 200 ]
+[ "$(curl -k -sw '%{http_code}' -o /dev/null https://$SUT_HTTPS_HOST/foo)" == 200 ]
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar)" == 426 ]
+[ "$(curl -k -sw '%{http_code}' -o /dev/null https://$SUT_HTTPS_HOST/bar)" == 200 ]
+

test/integration/cases/02-ingress-apiversions/ingress.yaml

@@ -0,0 +1,32 @@
+apiVersion: networking.k8s.io/v1beta1
+kind: Ingress
+metadata:
+ name: echo-networkingv1beta1
+ annotations:
+ kubernetes.io/ingress.class: kong
+spec:
+ rules:
+ - http:
+ paths:
+ - backend:
+ serviceName: echo
+ servicePort: 8080
+ path: /bar
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: echo-networkingv1
+ annotations:
+ kubernetes.io/ingress.class: kong
+spec:
+ rules:
+ - http:
+ paths:
+ - backend:
+ service:
+ name: echo
+ port:
+ number: 8080
+ path: /baz
+ pathType: Prefix

test/integration/cases/02-ingress-apiversions/verify.sh

@@ -0,0 +1,9 @@
+#!/bin/bash
+set -ex
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/)" == 404 ]
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/bar/)" == 200 ]
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/baz/)" == 200 ]
+
+[ "$(curl -sw '%{http_code}' -o /dev/null http://$SUT_HTTP_HOST/qux/)" == 404 ]