Log to /var/log/telegraf/telegraf.log on systems using systemd

[danielnelson]

Feature Request

Proposal:

On systems using sysvinit, logs are set to write to /var/log/telegraf/telegraf.log. For systems using systemd the logs instead end up in /var/log/messages, the /var/log/telegraf directory remains empty.

Current behavior:

• systemd systems log to /var/log/messages

Desired behavior:

• systemd systems log to /var/log/telegraf/telegraf.log

[arkady-emelyanov]

Well,

  ## Specify the log file name. The empty string means to log to stderr.
  logfile = "/var/log/telegraf/telegraf.log"

Line above in telegraf.conf solve all problems with systemd based distros. no telegraf related entries in /var/log/messages.

So, just to clarify, what is the problem exactly?

set logfile to be an empty string and adjust systemd unit like this:

StandardError=/var/log/telegraf/telegraf.log

or, force logfile to be /var/log/telegraf/telegraf.log.

isn't sufficient?
Or I just miss something?

[danielnelson]

We should probably set StandardError, I'll do that. It looks like centos7 and debian still use logrotate so we are good there. I don't want to set logfile in the telegraf.conf yet as there are some issues with it in sysvinit #4101.

[phemmer]

Re-iteration, and elaboration on #4184 (comment)

systemd systems log to /var/log/messages

This is not inherently true. By default systemd does not log to any (plain text) files at all.
The statement is true on some systems which have a logging daemon (such as rsyslog or syslog-ng) that reads from the journal and writes plain text files (EL7 comes this way). But you can uninstall these log daemons and the journal continues to work just fine.

I would say that if the user wants these files created on a systemd based system, it is up to that user to do so. There are 3 ways they can do it.

1. Update the telegraf config as @arkady-emelyanov mentioned.
2. Modify the config of that logging daemon (syslog-ng or rsyslog) to match the telegraf process and log to those files.
3. Use a systemd drop-in to set the StandardOutput/StandardError parameters.

I would also vote that it is up to the OS package maintainers to make this decision. Then the package defaults can be tailored to the expectations of that distribution. The native vendor-provided unit file should provide an out of the box configuration idiomatic to the service management system.

[danielnelson]

Thank you @phemmer this is really good advice. Since this is not systemd/journald related we shouldn't make any changes to its configuration.

I would also vote that it is up to the OS package maintainers to make this decision.

Of course for the official packages this is me :). I think what seems best is to add a rsyslog config to /etc/rsyslog.d/, this seems to be supported out of the box by RHEL, Debian, and SUSE which are the main platforms we are targeting. Ideally, we would also use syslog for sysvinit so that the logfile is configured in one place and to avoid some existing issues in the initscripts, this will likely involve a revamp of the current logging system.

Certainly we should make sure that maintainers for Gentoo, FreeBSD, and other platforms can easily create a well behaving package.

[mhf-ir]

On ubuntu 18.04 also with:

[agent]
  interval = "30s"
  collection_jitter = "20s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  flush_interval = "45s"
  flush_jitter = "15s"
  precision = ""
  debug = false
  quiet = true
  logfile = "/dev/null"
  omit_hostname = false

Logs will apear on syslog

tail -n 1 /var/log/syslog
May 25 21:15:57 ns1 telegraf[481]: cpu,cpu=cpu0,host=serv1,usage_user=0,usage_nice=0,usage_iowait=2.846219201359371,usage_irq=0,usage_guest=0,usage_guest_nice=0,usage_system=0.2124044180118932,usage_idle=96.94137638062861,usage_softirq=0,usage_steal=0 1558818909000000000

How could we completly disable telegraf logs to syslog?

[danielnelson]

@mhf-ir It looks like you probably have the file output enabled, normally Telegraf does not log this type of data. If you disable the file output it should be very quiet with that configuration.

[mhf-ir]

@mhf-ir It looks like you probably have the file output enabled, normally Telegraf does not log this type of data. If you disable the file output it should be very quiet with that configuration.

Sorry my bad, you're right. 👍

[vladak]

I hit the same problem - forgot outputs.file section (to stdout) in one of the files in the /etc/telegraf/telegraf.d/ directory and it applied also to inputs from /etc/telegraf/telegraf.conf.

[mahish512]

my "messages" file under /var/log/ directory is filling with telegraf messages. i have enabled below configs. so any help will be appreciated.

debug = false
quiet = true
logfile = "/logs/telegraf/telegraf.log"

[cellcoresystems]

@mahish512 @vladak @mhf-ir you can change the telegraf.service unit.

It can be stopped by adding the following to prevent telegraf.service stdout writing to /var/log/messages.
And you will receive stderr.
source: silent-a-systemd-service

# /usr/lib/systemd/system/telegraf.service
[Service]
StandardOutput=null
StandardError=journal

# don't forget, sometimes the unit file is not reloaded ;)
systemctl daemon-reload           
systemctl restart telegraf.service

It even works if you have outputs.file configured for example.

[[outputs.file]]
  files = ["stdout","/tmp/metrics.out"]
  rotation_max_size = "50MB"
  rotation_max_archives = 4
  data_format = "influx"

Then you can use the agent settings on top.

[agent]
  debug = false
  quiet = true
  logtarget = "file"
  logfile = "/var/log/telegraf/telegraf.log"

---

@danielnelson I think there is no need to change anything of the default as there are all combinations possible.

Furthermore systemd from version 236+ allows StandardOutput=file:/var/log/telegraf/telegraf.log see here

Hope it helps.

[yosiasz]

none of these settings in the config are working for me on latest telegraf on Ubuntu

[srebhan]

As there is a workaround I'm closing this issue. @yosiasz if your issue still exists, please open a new issue with more details on your configuration, expected behavior etc.

Thanks @cellcoresystems for providing your solution!