The service and routes created are deleted periodically for no reason #219

deepzz0 · 2018-12-29T10:43:43Z

NOTE: GitHub issues are reserved for bug reports only.
For anything else, please join the conversation
in Kong Nation https://discuss.konghq.com/c/kubernetes.

Summary

I'm testing kong, and I added ingress. But the route will be deleted and rebuilt every once in a while

Kong Ingress controller version
0.2.2

Kong or Kong Enterprise version
0.14.1

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.4", GitCommit:"f49fa022dbe63faafd0da106ef7e05a29721d3f1", GitTreeState:"clean", BuildDate:"2018-12-14T06:59:37Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Environment

Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release): CentOS Linux 7 (Core)
Kernel (e.g. uname -a): Linux master40 3.10.0-957.1.3.el7.x86_64 Status of controller? #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Install tools:

What happened

I created ingress using the following configuration：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  labels:
    app: my-ingress
  name: my-ingress
spec:
  rules:
  - http:
      paths:
      - path: /api/v1/account
        backend:
          serviceName: account
          servicePort: 9000

Initially, I was able to connect and access the interface. After a while, however, route will be deleted and regenerated. I don't understand what's going on here.

I see admin-api logs：

127.0.0.1 - - [29/Dec/2018:10:18:35 +0000] "GET /upstreams?size=1000 HTTP/1.1" 200 693 "-" "Go-http-client/1.1"
127.0.0.1 - - [29/Dec/2018:10:18:35 +0000] "GET /upstreams/7770c54d-466f-4ac5-8d0c-fea696e4c234/targets?size=1000 HTTP/1.1" 200 188 "-" "Go-http-client/1.1"
2018/12/29 10:18:35 [warn] 28#0: *738 [lua] balancer.lua:718: redistributeIndices(): [ringbalancer 2] redistributed indices, size=10000, dropped=10000, assigned=0, left unassigned=10000, client: 127.0.0.1, server: kong_admin, request: "DELETE /upstreams/7770c54d-466f-4ac5-8d0c-fea696e4c234/targets/fd00a1f8-2eee-4fbe-a3c7-14640fd708d4 HTTP/1.1", host: "localhost:8001"
127.0.0.1 - - [29/Dec/2018:10:18:35 +0000] "DELETE /upstreams/7770c54d-466f-4ac5-8d0c-fea696e4c234/targets/fd00a1f8-2eee-4fbe-a3c7-14640fd708d4 HTTP/1.1" 204 0 "-" "Go-http-client/1.1"
2018/12/29 10:18:35 [error] 28#0: *738 [lua] balancer.lua:567: on_target_event(): target delete: upstream not found for 7770c54d-466f-4ac5-8d0c-fea696e4c234, client: 127.0.0.1, server: kong_admin, request: "DELETE /upstreams/7770c54d-466f-4ac5-8d0c-fea696e4c234 HTTP/1.1", host: "localhost:8001"
2018/12/29 10:18:35 [error] 28#0: *738 [lua] balancer.lua:567: on_target_event(): target delete: upstream not found for 7770c54d-466f-4ac5-8d0c-fea696e4c234, client: 127.0.0.1, server: kong_admin, request: "DELETE /upstreams/7770c54d-466f-4ac5-8d0c-fea696e4c234 HTTP/1.1", host: "localhost:8001"
127.0.0.1 - - [29/Dec/2018:10:18:35 +0000] "DELETE /upstreams/7770c54d-466f-4ac5-8d0c-fea696e4c234 HTTP/1.1" 204 0 "-" "Go-http-client/1.1"
127.0.0.1 - - [29/Dec/2018:10:18:35 +0000] "DELETE /services/cf898f68-e584-43c9-83fb-d6b841ada81b HTTP/1.1" 400 175 "-" "Go-http-client/1.1"
2018/12/29 10:18:35 [notice] 28#0: *738 [lua] init.lua:427: delete(): ERROR: update or delete on table "services" violates foreign key constraint "routes_service_id_fkey" on table "routes"
Key (id)=(cf898f68-e584-43c9-83fb-d6b841ada81b) is still referenced from table "routes"., client: 127.0.0.1, server: kong_admin, request: "DELETE /services/cf898f68-e584-43c9-83fb-d6b841ada81b HTTP/1.1", host: "localhost:8001"
127.0.0.1 - - [29/Dec/2018:10:18:35 +0000] "GET /routes?size=1000 HTTP/1.1" 200 313 "-" "Go-http-client/1.1"
127.0.0.1 - - [29/Dec/2018:10:18:35 +0000] "DELETE /routes/94f31a88-c917-4476-9d0d-72a804ce4073 HTTP/1.1" 204 0 "-" "Go-http-client/1.1"
10.244.2.1 - - [29/Dec/2018:10:18:37 +0000] "GET /status HTTP/1.1" 200 210 "-" "kube-probe/1.12"
127.0.0.1 - - [29/Dec/2018:10:18:39 +0000] "GET /certificates?size=1000 HTTP/1.1" 200 24 "-" "Go-http-client/1.1"
127.0.0.1 - - [29/Dec/2018:10:18:39 +0000] "GET /upstreams/default.account.9000 HTTP/1.1" 404 24 "-" "Go-http-client/1.1"
127.0.0.1 - - [29/Dec/2018:10:18:39 +0000] "POST /upstreams HTTP/1.1" 201 672 "-" "Go-http-client/1.1"
127.0.0.1 - - [29/Dec/2018:10:18:39 +0000] "GET /upstreams/default.account.9000 HTTP/1.1" 200 672 "-" "Go-http-client/1.1"
127.0.0.1 - - [29/Dec/2018:10:18:39 +0000] "GET /upstreams/default.account.9000/targets?size=1000 HTTP/1.1" 200 22 "-" "Go-http-client/1.1"
127.0.0.1 - - [29/Dec/2018:10:18:39 +0000] "POST /upstreams/default.account.9000/targets HTTP/1.1" 201 167 "-" "Go-http-client/1.1"

Expected behvaior

How can I not delete.

Steps To Reproduce

The text was updated successfully, but these errors were encountered:

hbagdi · 2019-01-02T06:20:24Z

@deepzz0 Is the state of your k8s service changing or the pods being terminated and recreated?
The ingress controller has deterministic behavior and it should not be deleting and recreating entities.

Could you share your manifests that you've used?

deepzz0 · 2019-01-04T02:10:54Z

@hbagdi nothing，the pod not terminated or recreated.

ingress config:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  labels:
    app: 51ssl-ingress
  name: 51ssl-ingress
spec:
  rules:
  - http:
      paths:
      - path: /api/v1/account
        backend:
          serviceName: account-51ssl
          servicePort: 9000

my service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    plugins.konghq.com: cookies-plugin
  creationTimestamp: null
  labels:
    io.kompose.service: account-51ssl
  name: account-51ssl
spec:
  ports:
  - name: "9000"
    port: 9000
    targetPort: 9000
  selector:
    io.kompose.service: account-51ssl
status:
  loadBalancer: {}

i checked the ingress-controller logs:

I0104 02:06:37.086509       7 kong.go:113] syncing global plugins
W0104 02:06:37.091005       7 kong.go:329] the service defined in the ingress default/51ssl-ingress does not exists
W0104 02:06:37.094592       7 kong.go:743] the service defined in the ingress default/51ssl-ingress does not exists
W0104 02:06:40.408158       7 controller.go:387] service default/account-51ssl does not have any active endpoints
I0104 02:06:40.408219       7 controller.go:128] syncing Ingress configuration...
W0104 02:06:40.410533       7 kong.go:946] the service defined in the ingress default/51ssl-ingress does not exists
I0104 02:06:40.413001       7 kong.go:113] syncing global plugins
W0104 02:06:40.417557       7 kong.go:329] the service defined in the ingress default/51ssl-ingress does not exists
W0104 02:06:40.421931       7 kong.go:743] the service defined in the ingress default/51ssl-ingress does not exists
I0104 02:06:43.741242       7 controller.go:128] syncing Ingress configuration...
I0104 02:06:43.746763       7 kong.go:1023] creating Kong Upstream with name default.account-51ssl.9000
I0104 02:06:43.768298       7 kong.go:241] creating Kong Target 10.244.2.166:9000 for upstream 0xc00045f830
I0104 02:06:43.883172       7 kong.go:113] syncing global plugins
W0104 02:06:43.887304       7 kong.go:335] there is no custom Ingress configuration for rule default/51ssl-ingress
I0104 02:06:43.889024       7 kong.go:401] Creating Kong Service name default.account-51ssl.9000
W0104 02:07:59.978839       7 kong.go:749] there is no custom Ingress configuration for rule default/51ssl-ingress
I0104 02:07:59.980920       7 kong.go:775] creating Kong Route for host _, path /api/v1/account and service 8a29eda3-c190-44de-b6b8-c07b89605452
I0104 02:07:59.997408       7 controller.go:128] syncing Ingress configuration...
I0104 02:08:00.117280       7 kong.go:113] syncing global plugins
W0104 02:08:00.123426       7 kong.go:335] there is no custom Ingress configuration for rule default/51ssl-ingress
W0104 02:08:00.141748       7 kong.go:749] there is no custom Ingress configuration for rule default/51ssl-ingress
I0104 02:08:00.143682       7 kong.go:803] updating Kong Route for host _, path /api/v1/account and service 0xc0002d64e0
W0104 02:08:40.822764       7 controller.go:387] service default/account-51ssl does not have any active endpoints
I0104 02:08:40.822830       7 controller.go:128] syncing Ingress configuration...
W0104 02:08:40.826121       7 kong.go:946] the service defined in the ingress default/51ssl-ingress does not exists
I0104 02:08:40.832751       7 kong.go:113] syncing global plugins
W0104 02:08:40.838337       7 kong.go:329] the service defined in the ingress default/51ssl-ingress does not exists
I0104 02:08:40.849470       7 kong.go:547] deleting Kong Service default.account-51ssl.9000
W0104 02:08:40.908151       7 kong.go:743] the service defined in the ingress default/51ssl-ingress does not exists
I0104 02:08:40.908206       7 kong.go:914] deleting Kong Route 6381f2b6-003c-4577-8e35-ca8f7b32de9a
W0104 02:08:44.156306       7 controller.go:387] service default/account-51ssl does not have any active endpoints
I0104 02:08:44.156375       7 controller.go:128] syncing Ingress configuration...
W0104 02:08:44.158823       7 kong.go:946] the service defined in the ingress default/51ssl-ingress does not exists
I0104 02:08:44.160834       7 kong.go:113] syncing global plugins
W0104 02:08:44.166088       7 kong.go:329] the service defined in the ingress default/51ssl-ingress does not exists
I0104 02:08:44.170898       7 kong.go:547] deleting Kong Service default.account-51ssl.9000
W0104 02:08:44.188827       7 kong.go:743] the service defined in the ingress default/51ssl-ingress does not exists
W0104 02:08:47.489787       7 controller.go:387] service default/account-51ssl does not have any active endpoints
I0104 02:08:47.489850       7 controller.go:128] syncing Ingress configuration...
W0104 02:08:47.492362       7 kong.go:946] the service defined in the ingress default/51ssl-ingress does not exists
I0104 02:08:47.495402       7 kong.go:113] syncing global plugins
W0104 02:08:47.504520       7 kong.go:329] the service defined in the ingress default/51ssl-ingress does not exists
W0104 02:08:47.512475       7 kong.go:743] the service defined in the ingress default/51ssl-ingress does not exists
I0104 02:08:50.823298       7 controller.go:128] syncing Ingress configuration...
I0104 02:08:50.830922       7 kong.go:1023] creating Kong Upstream with name default.account-51ssl.9000
I0104 02:08:50.851478       7 kong.go:241] creating Kong Target 10.244.2.166:9000 for upstream 0xc0009fa960
I0104 02:08:50.968927       7 kong.go:113] syncing global plugins
W0104 02:08:50.973068       7 kong.go:335] there is no custom Ingress configuration for rule default/51ssl-ingress
I0104 02:08:50.975184       7 kong.go:401] Creating Kong Service name default.account-51ssl.9000

deepzz0 · 2019-01-04T09:15:08Z

@hbagdi the controller also print log:

2019/01/04 09:09:45 [error] 42#0: *50307 recv() failed (111: Connection refused), context: ngx.timer, client: 10.244.0.0, server: 0.0.0.0:8000
2019/01/04 09:09:47 [error] 42#0: *50307 recv() failed (111: Connection refused), context: ngx.timer, client: 10.244.0.0, server: 0.0.0.0:8000
2019/01/04 09:09:47 [error] 42#0: *50307 recv() failed (111: Connection refused), context: ngx.timer, client: 10.244.0.0, server: 0.0.0.0:8000
2019/01/04 09:21:21 [warn] 45#0: *53537 [lua] reports.lua:49: log(): [reports] could not set peer name for UDP socket: [toip() name lookup failed]: dns lookup pool exceeded retries (1): failed to receive reply from UDP server 10.96.0.10:53: connection refused, context: ngx.timer

Sorry, I don't use k8s very long, Can you give me some solutions. Best wishes.

hbagdi · 2019-03-08T01:42:07Z

Could you please verify the behavior with Kong Ingress Controller 0.3.0 and Kong 1.0?

deepzz0 · 2019-03-21T07:52:36Z

I get the same problem. Am i using a custom plugin?
At present. I run kong as a service in k8s.

hbagdi · 2019-03-21T18:13:07Z

I get the same problem. Am i using a custom plugin?
At present. I run kong as a service in k8s.

@deepzz0 I'm not sure if I understand you correctly, could you please elaborate?

bgarcial · 2019-04-10T14:33:58Z

@deepzz0 and @hbagdi
I have a similar problem, my kong-ingress-controller are not deleting the routes and services itself, but it is in a permanent CrashLoopBackOff status

I have documented here in this issue.
#267

GuillaumeSmaha · 2019-04-10T14:50:42Z

@bgarcial I had the same problem. I also saw one time it was OOM killed, try with more memory.
Adding more memory solves my CrashLoopBackOff status

bgarcial · 2019-04-23T11:34:05Z

@GuillaumeSmaha Really? I have debugged the different things and exactly I can't find the reasons to the CrashLoopBackOff
Did you mean add more memory to kong-ingress-controller deployment resource?
Something like that?
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/

GuillaumeSmaha · 2019-04-23T17:25:24Z

@bgarcial Before updating I used 256Mo for kong (not kong-ingress-controller).
In the helm chart values.yaml, the example is 128Mi https:/helm/charts/blob/master/stable/kong/values.yaml#L96-L102 (this container limitation, not namespace limitation).
I saw only one time the OOM, but I also had CrashLoopBackOff without explicit reason.
Since I update to 1Gi, it works fine.
Currently, my kong-ingress-controller doesn't have resource limitation.

hbagdi · 2019-04-24T18:51:14Z

The core logic for syncing resources to Kong has been completely redone, please use Kong Ingress Controller 0.4.0 to avoid hitting this issue.

hbagdi added the work in progress Work In Progress label Apr 6, 2019

hbagdi removed the work in progress Work In Progress label Apr 24, 2019

hbagdi closed this as completed Apr 24, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The service and routes created are deleted periodically for no reason #219

The service and routes created are deleted periodically for no reason #219

deepzz0 commented Dec 29, 2018

hbagdi commented Jan 2, 2019

deepzz0 commented Jan 4, 2019

deepzz0 commented Jan 4, 2019 •

edited

Loading

hbagdi commented Mar 8, 2019

deepzz0 commented Mar 21, 2019

hbagdi commented Mar 21, 2019

bgarcial commented Apr 10, 2019

GuillaumeSmaha commented Apr 10, 2019 •

edited

Loading

bgarcial commented Apr 23, 2019

GuillaumeSmaha commented Apr 23, 2019

hbagdi commented Apr 24, 2019

The service and routes created are deleted periodically for no reason #219

The service and routes created are deleted periodically for no reason #219

Comments

deepzz0 commented Dec 29, 2018

Summary

Kubernetes version

Environment

What happened

Expected behvaior

Steps To Reproduce

hbagdi commented Jan 2, 2019

deepzz0 commented Jan 4, 2019

deepzz0 commented Jan 4, 2019 • edited Loading

hbagdi commented Mar 8, 2019

deepzz0 commented Mar 21, 2019

hbagdi commented Mar 21, 2019

bgarcial commented Apr 10, 2019

GuillaumeSmaha commented Apr 10, 2019 • edited Loading

bgarcial commented Apr 23, 2019

GuillaumeSmaha commented Apr 23, 2019

hbagdi commented Apr 24, 2019

deepzz0 commented Jan 4, 2019 •

edited

Loading

GuillaumeSmaha commented Apr 10, 2019 •

edited

Loading