Retrieve AWS EC2 tags using instance metadata

[kirillmandrygin]

Use Case

I want to add EC2 tags using metadata and I don't want to use the instance role to do this.

AWS Documentation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/work-with-tags-in-IMDS.html

Expected behavior

The AWS EC2 Metadata Processor plugin should support retrieving tags from IMDS

Actual behavior

The aws_ec2 processor plugin doesn't support this

Additional info

No response

[srebhan]

For my understanding, do you want the processor to use the instance-metadata endpoint instead of accessing the actual compute instance?

[kirillmandrygin]

@srebhan currently processors.aws_ec2 supports two options imds_tags (from instance metadata) and ec2_tags (from ec2 through instance profile with ec2:DescribeTags policy). And I think the second option is not required because we have option to retrieve tags from instance metadata, but we can leave it for backward compatibility

[srebhan]

Sorry I'm not exactly sure what you are requesting here? To me it sounds like you want to use the imds_tags part of the processor but as you opened an issue I'm clearly missing something here. :-)

[kirillmandrygin]

I'd like to get instance tags via imds_tags without using ec2_tags and ec2:DescribeTags policy. Have you opened the link I provided in the initial request?

[srebhan]

That's exactly what should happen if you only specify the imds_tags without using the ec2_tags... What am I missing here?

And yes I've read the document you linked. ;-)

[kirillmandrygin]

you can not get instance tags from imds_tags, for now you can get instance tags only from ec2_tags

[kirillmandrygin]

for now imds_tags retrieves tags from this url: /latest/dynamic/instance-identity/document/

telegraf/plugins/processors/aws_ec2/ec2.go

Line 146 in f693df8

iido, err := r.imdsClient.GetInstanceIdentityDocument(

https:/aws/aws-sdk-go-v2/blob/main/feature/ec2/imds/api_op_GetInstanceIdentityDocument.go

But I want to get also EC2 instance tags from this path: /latest/meta-data/tags/instance

https:/aws/aws-sdk-go-v2/blob/main/feature/ec2/imds/api_op_GetMetadata.go

[srebhan]

I see. Sorry for not getting this earlier! Are you willing to contribute a pull-request?

[kirillmandrygin]

Probably I didn't explain what I needed in the initial request well and that's what confused you. I apologise and it's great that we've come to an understanding

I'm not sure of my golang skills, but I'll try to add this feature myself next week ;)

[srebhan]

@kirillmandrygin please test the binary in PR #15795, available as soon as CI finished the tests, and let me know if this is what you had in mind. You should use the new metadata_tags setting for your use-case...

[kirillmandrygin]

@srebhan it seems this is not working for me, I can see imds_tags but there is no metadata_tags

telegraf.conf:

[global_tags]
[agent]
  interval = "60s"
  collection_jitter = "0s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = false
[[outputs.prometheus_client]]
  listen = ":9281"
  path = "/metrics"
  expiration_interval = "120s"
  collectors_exclude = ["gocollector", "process"]
  string_as_label = true
[[inputs.cpu]]
  percpu = false
  totalcpu = true
  collect_cpu_time = false
  report_active = false
[[processors.aws_ec2]]
  imds_tags = [ "accountId", "instanceId", "instanceType", "privateIp", "availabilityZone" ]
  metadata_tags = [ "Name", "Stage" ]
[[processors.template]]
  tag = "instance"
  template = '{{ .Tag "Name" }}-{{ .Tag "privateIp" }}'

telegraf version:

# /usr/bin/telegraf --version
Telegraf 1.32.0-6e912139 (git: pull/15795@6e912139)

tags are available through IMDS:

# curl -H "X-aws-ec2-metadata-token: $TOKEN" ${AWS_API_URL}/latest/meta-data/tags/instance/Name
test-kmandrygin

and there is no any error in debug log:

# /usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d --debug
2024-08-30T07:15:58Z I! Loading config: /etc/telegraf/telegraf.conf
2024-08-30T07:15:58Z I! Starting Telegraf 1.32.0-6e912139 brought to you by InfluxData the makers of InfluxDB
2024-08-30T07:15:58Z I! Available plugins: 235 inputs, 9 aggregators, 32 processors, 26 parsers, 62 outputs, 6 secret-stores
2024-08-30T07:15:58Z I! Loaded inputs: cpu
2024-08-30T07:15:58Z I! Loaded aggregators:
2024-08-30T07:15:58Z I! Loaded processors: aws_ec2 template
2024-08-30T07:15:58Z I! Loaded secretstores:
2024-08-30T07:15:58Z I! Loaded outputs: prometheus_client
2024-08-30T07:15:58Z I! Tags enabled: host=**********
2024-08-30T07:15:58Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"***********", Flush Interval:10s
2024-08-30T07:15:58Z D! [agent] Initializing plugins
2024-08-30T07:15:58Z D! [processors.aws_ec2] Initializing AWS EC2 Processor
2024-08-30T07:15:58Z D! [processors.aws_ec2] Initializing AWS EC2 Processor
2024-08-30T07:15:58Z D! [agent] Connecting outputs
2024-08-30T07:15:58Z D! [agent] Attempting connection to [outputs.prometheus_client]
2024-08-30T07:15:58Z I! [outputs.prometheus_client] Listening on http://[::]:9281/metrics
2024-08-30T07:15:58Z D! [agent] Successfully connected to outputs.prometheus_client
2024-08-30T07:15:58Z D! [processors.aws_ec2] cache: size=1000
2024-08-30T07:15:58Z D! [agent] Starting service inputs
2024-08-30T07:16:08Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:16:18Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:16:28Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:16:38Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:16:48Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:16:58Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2024-08-30T07:17:08Z D! [outputs.prometheus_client] Wrote batch of 1 metrics in 87.327µs
2024-08-30T07:17:08Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics

[srebhan]

Could you please use the latest version of the binary in the PR and set

[[processors.aws_ec2]]
  log_level = "trace"
  imds_tags = [ "accountId", "instanceId", "instanceType", "privateIp", "availabilityZone" ]
  metadata_tags = [ "Name", "Stage" ]

as this should print out all received metadata key-value pairs (I hope)...

Could you please also provide the output of

# curl -H "X-aws-ec2-metadata-token: $TOKEN" ${AWS_API_URL}/latest/meta-data

[kirillmandrygin]

@srebhan already provided trace here: #15795 (comment)

2024-08-30T11:34:00Z T! [processors.aws_ec2] received metadata: {values:map[{}:{Results:[{Err:<nil> Retryable:false Retried:false ResponseMetadata:{values:map[]}}]}]}

and please take a look at the output of the meta-data URL:

# curl -H "X-aws-ec2-metadata-token: $TOKEN" ${AWS_API_URL}/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
identity-credentials/
instance-action
instance-id
instance-life-cycle
instance-type
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-keys/
reservation-id
security-groups
services/
system
tags/

full path to a tag will be /latest/meta-data/tags/instance/<TAG_NAME>

[kirillmandrygin]

Hi @srebhan ! It is working good but I have two issues:

1. Error when I run telegraf with --test:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x740411b]

goroutine 109 [running]:
github.com/influxdata/telegraf/plugins/processors/aws_ec2.(*AwsEc2Processor).Stop(0xc00252e9c0?)
	/go/src/github.com/influxdata/telegraf/plugins/processors/aws_ec2/ec2.go:166 +0x3b
github.com/influxdata/telegraf/models.(*RunningProcessor).Stop(...)
	/go/src/github.com/influxdata/telegraf/models/running_processor.go:115
github.com/influxdata/telegraf/agent.(*Agent).runProcessors.func1(0xc001fec0c0)
	/go/src/github.com/influxdata/telegraf/agent/agent.go:666 +0x172
created by github.com/influxdata/telegraf/agent.(*Agent).runProcessors in goroutine 14
	/go/src/github.com/influxdata/telegraf/agent/agent.go:656 +0x3e

2. Metadata tag added with slash (for example: test_metric{/Name="test-instance"}), is this expected behaviour?