Elastic-Agent standalone fails to start on Linux if /tmp is a tmpfs

[belimawr]

On a Linux VM (Archlinux) whenever I try to start a standalone Elastic-Agent (built from a recent main) I get the following error:

Exiting: could not start the HTTP server for the API: listen unix /tmp/elastic-agent/7uyxrneW50ZMk_CZbrE0Hjg3idmd7AzZ.sock: bind: no such file or directory

Full logs:

{"log.level":"error","@timestamp":"2024-04-26T12:48:31.175Z","message":"Exiting: could not start the HTTP server for the API: listen unix /tmp/elastic-agent/7uyxrneW50ZMk_CZbrE0Hjg3idmd7AzZ.sock: bind: no such file or directory","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":1346,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-04-26T12:48:31.175Z","message":"Exiting: could not start the HTTP server for the API: listen unix /tmp/elastic-agent/7uyxrneW50ZMk_CZbrE0Hjg3idmd7AzZ.sock: bind: no such file or directory","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0"}

It just stays in this error loop with Filebeat never starting. It likely affects other Beats as well, but on my tests only Filebeat was used. Here are a whole set of logs until Filebeat is restarted

{"log.level":"info","@timestamp":"2024-04-26T13:03:08.396Z","message":"Home path: [/home/vagrant/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/data/elastic-agent-af619c/components] Config path: [/home/vagrant/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/data/elastic-agent-af619c/components] Data path: [/home/vagrant/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/data/elastic-agent-af619c/run/filestream-default] Logs path: [/home/vagrant/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64/data/elastic-agent-af619c/components/logs]","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":817,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-26T13:03:08.396Z","message":"Beat ID: b2a8b285-1f2e-4074-b295-9ec5926b8e91","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.origin":{"file.line":825,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-26T13:03:08.398Z","message":"Output reload is enabled, the beat will restart as needed on change of output config","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"centralmgmt","log.origin":{"file.line":198,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.NewV2AgentManagerWithClient"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-26T13:03:08.398Z","message":"Set gc percentage to: 100","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":876,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-26T13:03:08.405Z","message":"running under elastic-agent, per-beat lockfiles disabled","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":442,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-26T13:03:08.406Z","message":"filebeat stopped.","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":457,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-04-26T13:03:08.406Z","message":"Exiting: could not start the HTTP server for the API: listen unix /tmp/elastic-agent/7uyxrneW50ZMk_CZbrE0Hjg3idmd7AzZ.sock: bind: no such file or directory","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.origin":{"file.line":1346,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.handleError"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-04-26T13:03:08.406Z","message":"Exiting: could not start the HTTP server for the API: listen unix /tmp/elastic-agent/7uyxrneW50ZMk_CZbrE0Hjg3idmd7AzZ.sock: bind: no such file or directory","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-26T13:03:08.409Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":606},"message":"Component state changed filestream-default (STARTING->STOPPED): Suppressing FAILED state due to restart for '39778' exited with code '1'","log":{"source":"elastic-agent"},"component":{"id":"filestream-default","state":"STOPPED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-26T13:03:08.409Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":624},"message":"Unit state changed filestream-default (STARTING->STOPPED): Suppressing FAILED state due to restart for '39778' exited with code '1'","log":{"source":"elastic-agent"},"component":{"id":"filestream-default","state":"STOPPED"},"unit":{"id":"filestream-default","type":"output","state":"STOPPED","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-04-26T13:03:08.409Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":624},"message":"Unit state changed filestream-default-your-input-id (STARTING->STOPPED): Suppressing FAILED state due to restart for '39778' exited with code '1'","log":{"source":"elastic-agent"},"component":{"id":"filestream-default","state":"STOPPED"},"unit":{"id":"filestream-default-your-input-id","type":"input","state":"STOPPED","old_state":"STARTING"},"ecs.version":"1.6.0"}                                                                                

Creating the folder /tmp/elastic-agent solves the problem.

Details about my /tmp:

vagrant@🌐 archlinux ~/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64 % mount |grep "/tmp"             
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,nr_inodes=1048576,inode64)

vagrant@🌐 archlinux ~/elastic-agent-8.15.0-SNAPSHOT-linux-x86_64 % ll / |grep tmp
drwxrwxrwt  10 root    root     320 Apr 26 13:02 tmp

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Is the agent installed when this happens? Is it installed as root?

Can your user succeed with mkdir /tmp/elastic-agent and touch /tmp/elastic-agent/test-file.txt while the agent is failing to do the same?

[belimawr]

Is the agent installed when this happens? Is it installed as root?

No, it's a standalone Elastic-Agent, not installed.

Can your user succeed with mkdir /tmp/elastic-agent and touch /tmp/elastic-agent/test-file.txt while the agent is failing to do the same?

Yes, my user can succeed running mkdir -p /tmp/elastic-agent and that fixes the Elastic-Agent.

[belimawr]

The way I'm running it:

• Get the Linux package
• Extract it
• Edit the elastic-agent.yml to a working configuration. I noticed this crash when testing Log raw events to a separate log file #4549
• Run the Elastic-Agent standalone, without installing: ./elastic-agent
• Watch the logs: tail -F data/elastic-agent-*/logs/*.ndjson

[cmacknz]

If you run it as root and/or install it does this get fixed?

[belimawr]

Running as root does not change anything.

Installing creates other problems. I'll have to carefully reproduce/report later/next week. But the gist is: install succeeds, the Elastic-Agent does not start and I cannot uninstall it.