Install Signoz on linux, but clickhouse is unhealty.

[hrjin]

In what situation are you experiencing subpar performance?

I perform this link. https://signoz.io/docs/install/docker/#install-signoz-using-the-install-script.
But clickhouse is unhealthy like attached screenshot.

How to reproduce

1. git clone -b main https:/SigNoz/signoz.git && cd signoz/deploy/
2. ./install.sh
3. But clickhouse is unhealthy.

Your Environment

• [ O] Linux
• Mac
• Windows

Please provide details of OS version etc.

Additional context

Thank you for your performance issue report – we want SigNoz to be blazing fast!

[welcome]

Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.

[prashant-shahi]

Issue not reproducible.

Screenshot:

---

Could you share more information like which linux and architecture?
Share output of uname -a and docker version.

[kamilkosinski]

Hi @prashant-shahi,
I have the same problem and I think signoz stopped working after docker upgrade (to 26.x.x version).
I ended up trying to do a fresh install.

uname -a

Linux dobrymechanik-tools 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

docker version

Client: Docker Engine - Community
 Version:           26.0.2
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        3c863ff
 Built:             Thu Apr 18 16:27:07 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.2
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.9
  Git commit:       7cef0d9
  Built:            Thu Apr 18 16:27:07 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker compose -f docker/clickhouse-setup/docker-compose.yaml logs -n 20 clickhouse

signoz-clickhouse  | 2024.04.23 07:41:19.108331 [ 1 ] {} <Information> loadMetadata: Start synchronous loading of databases
signoz-clickhouse  | 2024.04.23 07:41:19.108351 [ 1 ] {} <Information> loadMetadata: Start synchronous startup of databases
signoz-clickhouse  | 2024.04.23 07:41:19.120049 [ 1 ] {} <Information> UserDefinedSQLObjectsLoaderFromDisk: Loading user defined objects from /var/lib/clickhouse/user_defined/
signoz-clickhouse  | 2024.04.23 07:41:19.120210 [ 1 ] {} <Information> Application: Tasks stats provider: procfs
signoz-clickhouse  | 2024.04.23 07:41:19.120237 [ 1 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_priority' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
signoz-clickhouse  | 2024.04.23 07:41:19.120596 [ 1 ] {} <Warning> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.1.2.5 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
signoz-clickhouse  | 2024.04.23 07:41:19.120824 [ 1 ] {} <Warning> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.1.2.5 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
signoz-clickhouse  | 2024.04.23 07:41:19.124334 [ 1 ] {} <Warning> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.1.2.5 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
signoz-clickhouse  | 2024.04.23 07:41:19.124580 [ 1 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.1.2.5 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
signoz-clickhouse  | 2024.04.23 07:41:19.278045 [ 1 ] {} <Information> CertificateReloader: One of paths is empty. Cannot apply new configuration for certificates. Fill all paths and try again.
signoz-clickhouse  | 2024.04.23 07:41:19.284445 [ 1 ] {} <Information> Application: Listening for http://0.0.0.0:8123
signoz-clickhouse  | 2024.04.23 07:41:19.284569 [ 1 ] {} <Information> Application: Listening for native protocol (tcp): 0.0.0.0:9000
signoz-clickhouse  | 2024.04.23 07:41:19.284743 [ 1 ] {} <Information> Application: Listening for MySQL compatibility protocol: 0.0.0.0:9004
signoz-clickhouse  | 2024.04.23 07:41:19.284907 [ 1 ] {} <Information> Application: Listening for PostgreSQL compatibility protocol: 0.0.0.0:9005
signoz-clickhouse  | 2024.04.23 07:41:19.305611 [ 722 ] {} <Information> ZooKeeperClient: Connected to ZooKeeper at 172.25.0.2:2181 with session_id 72083802339016704
signoz-clickhouse  | 2024.04.23 07:41:19.314863 [ 722 ] {} <Information> ZooKeeperClient: Keeper feature flag FILTERED_LIST: disabled
signoz-clickhouse  | 2024.04.23 07:41:19.314950 [ 722 ] {} <Information> ZooKeeperClient: Keeper feature flag MULTI_READ: disabled
signoz-clickhouse  | 2024.04.23 07:41:19.315010 [ 722 ] {} <Information> ZooKeeperClient: Keeper feature flag CHECK_NOT_EXISTS: disabled
signoz-clickhouse  | 2024.04.23 07:41:19.315040 [ 722 ] {} <Information> ZooKeeperClient: Keeper feature flag CREATE_IF_NOT_EXISTS: disabled
signoz-clickhouse  | 2024.04.23 07:41:19.315163 [ 1 ] {} <Information> Application: Ready for connections.

[hrjin]

Hello! @prashant-shahi
My docker version is 26.x.x too.
So I performed downgrade docker version, and success.

[hrjin]

Hello, @kamilkosinski
Your comment is good. Thank you.

[TheShubhendra]

Hello everyone , I was facing the same issue. Now after a long debugging session I got the cause.

By default, the clickhouse-server listens on 0.0.0.0:8123 but the health checks are running to ping on localhost:8123/ping.
So the quick fix is change the "localhost:8123/ping" in docker-compose.yaml to "0.0.0.0:8123/ping"

[paulohmorais]

Hey!

Same here, please fix it, this is important with impact on new users of signoz. The installation of signoz with docker are broken,

I validated and the problem is with the docker version 26.x.x as the community mentioned on this issue.

I made a downgrade on my ubuntu 22.04 with

sudo apt-get install docker-ce=5:25.0.5-1~ubuntu.22.04~jammy docker-ce-cli=5:25.0.5-1~ubuntu.22.04~jammy docker-ce-rootless-extras=5:25.0.5-1~ubuntu.22.04~jammy

And the installation with docker back to work.

Maybe the suggestion of @TheShubhendra can fix it quickly.

[ykshev]

Same problem, I wanted to try it but couldn't get up on Ububntu 22.

Update:

Hello everyone , I was facing the same issue. Now after a long debugging session I got the cause.

By default, the clickhouse-server listens on 0.0.0.0:8123 but the health checks are running to ping on localhost:8123/ping. So the quick fix is change the "localhost:8123/ping" in docker-compose.yaml to "0.0.0.0:8123/ping"

This actually works

[prashant-shahi]

The fix is part of the latest release v0.44.0.

@TheShubhendra Kudos for sharing the quickfix with us. 👍