Wrong active connections in status endpoint?

[manuelmazzuola]

Hi,
currently using Kong Enterprise 3.4.3.10.

I'm trying to understand why Kong status endpoint says that i have more than 200 active connections but if I lsof (lsof -nP -i TCP -s TCP:ESTABLISHED) all active TCP connections in the proxy container it reports that i only have 5 connections, all from the prometheus server.

Anyone has encountered this mismatch in numbers?

Thanks

[chobits]

The server: connections_active value of Kong status page comes from stub status module of underlying nginx core, see its explanation: https://nginx.org/en/docs/http/ngx_http_stub_status_module.html#data

Active connections
The current number of active client connections including Waiting connections.

lsof with -s TCP:ESTABLISHED will show only TCP connections that are actively established at the system level, excluding other stages (e.g., connections waiting to be closed or in other states).

Could you try this to see all the connection in kong $ lsof -p <kong/nginx worker pid>

[chobits]

I have checked the Nginx source code, and there are only four places where stats_active is incremented when the associated connections are created:

nginx $ grep ngx_stat_active -R src/ | grep ', 1'
src/http/v3/ngx_http_v3_request.c:    (void) ngx_atomic_fetch_add(ngx_stat_active, 1);
src/event/ngx_event_accept.c:        (void) ngx_atomic_fetch_add(ngx_stat_active, 1);
src/event/ngx_event_udp.c:        (void) ngx_atomic_fetch_add(ngx_stat_active, 1);
src/event/quic/ngx_event_quic_udp.c:        (void) ngx_atomic_fetch_add(ngx_stat_active, 1);

I suspect that:

1. Kong's active value may not be accurately reflecting ngx_stat_active. You can use GDB to check this:

$ gdb -p <nginx worker pid>
(gdb) p *ngx_stat_active

2. There may be connections used by Nginx that exceed the actual number of fds. You can use this GDB script: nginx.gdb.

# Download nginx.gdb

$ gdb -p <nginx worker pid>
(gdb) source nginx.gdb  # Ignore any error reports
(gdb) dumpic 10000      # See which connections are used by Nginx

[manuelmazzuola]

*ngx_stat_active

The ngx_stat_active value is correct, kong reflects its value.

However if i use the ss command to list all the sockets I have only a few of ESTAB sockets, and kong reports 130 active connections:

kong@kong-kong-67d574447c-6xdqr:/$  ss -p | grep nginx
u_str ESTAB 0      0                                                 * 150975731                     * 150975732 users:(("nginx",pid=2480,fd=15),("nginx",pid=1,fd=15))
u_str ESTAB 0      0                                                 * 150975732                     * 150975731 users:(("nginx",pid=2441,fd=16),("nginx",pid=1,fd=16))
u_str ESTAB 0      0                                                 * 151110663                     * 151110662 users:(("nginx",pid=2480,fd=14),("nginx",pid=1,fd=14))
u_str ESTAB 0      0                                                 * 151110662                     * 151110663 users:(("nginx",pid=2441,fd=3),("nginx",pid=1,fd=3))
u_str ESTAB 0      0                                                 * 151108827                     * 151108831 users:(("nginx",pid=2480,fd=16))
u_str ESTAB 0      0                   /kong_prefix/worker_events.sock 151108825                     * 151108811 users:(("nginx",pid=2480,fd=3))
u_str ESTAB 0      0                   /kong_prefix/worker_events.sock 151108831                     * 151108827 users:(("nginx",pid=2480,fd=22))
u_str ESTAB 0      0                                                 * 151108811                     * 151108825 users:(("nginx",pid=2441,fd=15))
tcp   ESTAB 0      0                                         127.0.0.1:8001                  127.0.0.1:57482     users:(("nginx",pid=2480,fd=29))
tcp   ESTAB 0      0                                      10.18.30.143:8443               10.18.31.190:59029     users:(("nginx",pid=2480,fd=32))
tcp   ESTAB 0      0                                      10.18.30.143:8443                10.18.34.39:32937     users:(("nginx",pid=2441,fd=23))
tcp   ESTAB 0      0                                         127.0.0.1:8001                  127.0.0.1:57470     users:(("nginx",pid=2480,fd=31))
tcp   ESTAB 0      0                                      10.18.30.143:8443               10.18.31.190:48652     users:(("nginx",pid=2441,fd=21))
tcp   ESTAB 0      0                                      10.18.30.143:8443                10.18.28.44:39541     users:(("nginx",pid=2441,fd=22))
tcp   ESTAB 0      0                                      10.18.30.143:33254              10.18.21.143:5432      users:(("nginx",pid=2480,fd=21))
tcp   ESTAB 0      0                                      10.18.30.143:8100               10.18.30.255:38124     users:(("nginx",pid=2480,fd=28))
tcp   ESTAB 0      0                                      10.18.30.143:8100                10.18.32.33:37892     users:(("nginx",pid=2480,fd=27))
tcp   ESTAB 0      0                                      10.18.30.143:33246              10.18.21.143:5432      users:(("nginx",pid=2441,fd=14))

[manuelmazzuola]

and ss -s gives

Total: 62
TCP:   348 (estab 26, closed 302, orphaned 0, timewait 263)

Transport Total     IP        IPv6
RAW       0         0         0
UDP       0         0         0
TCP       46        38        8
INET      46        38        8
FRAG      0         0         0

[manuelmazzuola]

@chobits do you know if there is a way to dump the current nginx active connections? like getting URL and the HTTP method of all the current active connections?

[chobits]

1. See my nginx.gdb , there's a function to help dump the HTTP-like data (like http request) for a single connection.
   https://gist.github.com/chobits/b2db8d1aca8ebcc94637d91e28850957#file-nginx-gdb-L195

 (gdb) dumpic_wr <connecitons number>

This way is to dump the static info in current nginx.

2. Another way you could see how nginx process all the things inside it, which includes connection and http request information is to enable nginx's debug log. But you need to build kong with --with-debug option for nginx/openresty.
   nginx debug log: https://nginx.org/en/docs/debugging_log.html