[exporter-helper] [resource_to_label_conversion] add helper function and expose exporter settings #2060

hossain-rayhan · 2020-11-04T18:41:03Z

Description:
We need a common helper function which can convert resource attributes to metric labels and be utilized by every exporter. In the linked issue, @bogdandrutu suggested two possible ways- a consumer or a helper function. In this PR, we have both of the consumer and helper function ready. However, I was only able to utilize the helper function for end to end test modifying the logging exporter. Every exporter can call this helper function before exporting the metrics to destination.

For the consumer setup, I cannot find a place to plug in this in between a processor and an exporter. I think I need more guidance on how we can make it a common utility for all exporters. Thanks in advance.

Link to tracking Issue:
Issue 1359

Testing:
End_To_End Test: Tested manually with awsecscontainermetrics receiver and logging exporter.
Unit test added.

codecov · 2020-11-04T18:49:01Z

Codecov Report

Merging #2060 (fee5a75) into master (c6b8f28) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2060      +/-   ##
==========================================
+ Coverage   92.10%   92.11%   +0.01%     
==========================================
  Files         278      279       +1     
  Lines       16978    17038      +60     
==========================================
+ Hits        15637    15695      +58     
- Misses        922      923       +1     
- Partials      419      420       +1

Impacted Files	Coverage Δ
exporter/exporterhelper/common.go	`100.00% <100.00%> (ø)`
exporter/exporterhelper/metricshelper.go	`86.84% <100.00%> (+0.73%)`	⬆️
exporter/exporterhelper/resource_to_label.go	`100.00% <100.00%> (ø)`
translator/internaldata/resource_to_oc.go	`91.54% <0.00%> (-2.82%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c6b8f28...fee5a75. Read the comment docs.

hossain-rayhan · 2020-11-05T17:56:34Z

Hi @bogdandrutu can you share your views here regarding the next steps. As discussed yesterdays Collector SIG meeting, the core function is working fine. I need guidance regarding making it a common utility for all exporters.

exporter/exporterhelper/resource_to_label.go

hossain-rayhan · 2020-11-06T18:18:28Z

Update
[1] Now the helper function and the consumer clone the incoming pdata.Metrics and modify the copy.
[2] ResourceToLabel setting option is exposed from exporterhelper so that all the exporters can utilize it.

TODO
Now we have access to ResourceToLabelSettings.Enabled value (true or false- which user can configure in OpenTelemetry config file). Need to write some kind of mechanism to use our ConvertResourceToLabels () helper function if this option is enabled.

Help Needed
From the exporterhelper, I cannot see a way to get access to pdata.Metrics to modify (the copy) and pass it to the next consumer. Maybe I am missing something. Working on it. If I cannot figure it out, I would highly appreciate any help/guidance here @bogdandrutu .

Thanks.

exporter/exporterhelper/resource_to_label.go

exporter/exporterhelper/common.go

bogdandrutu · 2020-11-06T19:28:36Z

One option (needs some refactoring) you can do it in metricshelper.ConsumeMetrics:

type metricsExporter struct {
	*baseExporter
	consumer.MetricsConsumer
}

// Delete the current ConsumeMetrics from metricsExporter

// NewMetricsExporter creates an MetricsExporter that records observability metrics and wraps every request with a Span.
func NewMetricsExporter(
	cfg configmodels.Exporter,
	logger *zap.Logger,
	pushMetricsData PushMetricsData,
	options ...ExporterOption,
) (component.MetricsExporter, error) {

        .......

	rc := &requestConsumer{
		sender: be.sender,
		pusher: pushMetricsData,
	}

        // Can plugin the resource consumer here if enabled.

	cc := &contextConsumer{
		name: be.cfg.Name(),
		next: rc,
	}

	return &metricsExporter{
		baseExporter:    be,
		MetricsConsumer: cc,
	}, nil
}


type contextConsumer struct {
	name string
	next consumer.MetricsConsumer
}

func (cc *contextConsumer) ConsumeMetrics(ctx context.Context, md pdata.Metrics) error {
	return cc.next.ConsumeMetrics(obsreport.ExporterContext(ctx, cc.name), md)
}

type requestConsumer struct {
	sender requestSender
	pusher PushMetricsData
}

func (rc *requestConsumer) ConsumeMetrics(ctx context.Context, md pdata.Metrics) error {
	req := newMetricsRequest(ctx, md, rc.pusher)
	_, err := rc.sender.send(req)
	return err
}

bogdandrutu

Your option also works, I think requires less work, so even better. We can cleanup later the consumer chain as I proposed.

exporter/exporterhelper/metricshelper.go

hossain-rayhan · 2020-11-06T19:33:41Z

Thanks. I will send another PR with updated README and your few minor suggestions.

bogdandrutu · 2020-11-06T20:01:05Z

Not sure I understand, are you going to close this PR and send a new one?

hossain-rayhan · 2020-11-06T20:03:07Z

Not sure I understand, are you going to close this PR and send a new one?

Sorry, my bad. I will send an update (commit).

bogdandrutu

Please rebase

Signed-off-by: Rayhan Hossain <[email protected]>

hossain-rayhan · 2020-11-06T21:32:08Z

Hi @bogdandrutu, will appreciate a final look from you.
I rebased the code. Still, the contrib-repo test is failing. I don't think it is happening because any of my changes.

Signed-off-by: Rayhan Hossain <[email protected]>

bogdandrutu

Almost LGTM

bogdandrutu · 2020-11-07T00:02:27Z

exporter/exporterhelper/resource_to_label.go

+ if dataPoint.IsNil() {
+ continue
+ }
+ newLabelMap.ForEach(func(k, v string) {


These 3 lines are code duplicate, can have a simple func.

Yea, they look similar but are of different types. From my understanding, for creating a different function we need to introduce some kind of interface/wrapping over these types. I think maybe it's OK to have this simple and straight forward solution rather than making it more complex.
I also found some similar patterns in our existing codebase. your call- would appreciate a second thought.

Are they? You have two StringMap.

Never mind. I got it wrong earlier- I thought about about line 97-100. Thanks, it makes the code nicer. Pushed an update.

exporter/exporterhelper/resource_to_label.go

bogdandrutu · 2020-11-07T00:05:11Z

exporter/exporterhelper/resource_to_label.go

+)
+
+// ResourceToLabelSettings defines configuration for converting resource attributes to labels
+type ResourceToLabelSettings struct {


ResourceToTelemetey? We want to extend in the future to do resource to span attributes and log attributes.

Same changes applies to With func.

Signed-off-by: Rayhan Hossain <[email protected]>

…RTER_OTLP_TRACES_INSECURE (open-telemetry#2060)

hossain-rayhan requested a review from a team November 4, 2020 18:41

hossain-rayhan mentioned this pull request Nov 6, 2020

[metricstransformprocessor] add new operation- convert_resource_attributes_to_metric_labels open-telemetry/opentelemetry-collector-contrib#1337

Closed

bogdandrutu reviewed Nov 6, 2020

View reviewed changes

exporter/exporterhelper/resource_to_label.go Outdated Show resolved Hide resolved

exporter/exporterhelper/resource_to_label.go Outdated Show resolved Hide resolved

tigrannajaryan assigned bogdandrutu Nov 6, 2020

bogdandrutu reviewed Nov 6, 2020

View reviewed changes

exporter/exporterhelper/metricshelper.go Outdated Show resolved Hide resolved

bogdandrutu reviewed Nov 6, 2020

View reviewed changes

hossain-rayhan added 7 commits November 6, 2020 13:06

add helper function to convert resource attributes to labels

6a9e308

Signed-off-by: Rayhan Hossain <[email protected]>

add unit tests to improve coverage

40f4ba6

Signed-off-by: Rayhan Hossain <[email protected]>

get a copy of incoming metrics and modify it

7c91cb6

Signed-off-by: Rayhan Hossain <[email protected]>

update unit tests to make sure incoming pdata.Metrics are unchanged

3a4be62

Signed-off-by: Rayhan Hossain <[email protected]>

expose Resource_To_Label seeting option for all exporters

47bf714

Signed-off-by: Rayhan Hossain <[email protected]>

use the helper function directly instead of seeting up a consumer

2841c88

update README and address PR feedbacks

96df6b7

Signed-off-by: Rayhan Hossain <[email protected]>

hossain-rayhan force-pushed the resource-to-label branch from 14c92d9 to 96df6b7 Compare November 6, 2020 21:26

hossain-rayhan changed the title ~~[In Progress] add helper function to convert resource attributes to labels~~ [exporter-helper] add helper function to convert resource attributes to labels Nov 6, 2020

hossain-rayhan changed the title ~~[exporter-helper] add helper function to convert resource attributes to labels~~ [exporter-helper] [resource_to_label_conversion] add helper function and expose exporter settings Nov 6, 2020

add unit tests to improve coverage

2897b69

Signed-off-by: Rayhan Hossain <[email protected]>

bogdandrutu reviewed Nov 7, 2020

View reviewed changes

hossain-rayhan added 2 commits November 6, 2020 18:42

PR feedback: rename ResourceToLabel to ResourceToTelemetry

3353e5f

Signed-off-by: Rayhan Hossain <[email protected]>

refactoring: extract common code and create a simple func

fee5a75

Signed-off-by: Rayhan Hossain <[email protected]>

bogdandrutu approved these changes Nov 9, 2020

View reviewed changes

bogdandrutu merged commit 5f60c72 into open-telemetry:master Nov 9, 2020

serena7981 mentioned this pull request Feb 11, 2021

Support the resource_to_telemetry_conversion config option in OpenCensus exporter #2465

Closed

Troels51 pushed a commit to Troels51/opentelemetry-collector that referenced this pull request Jul 5, 2024

[EXPORTER] GRPC endpoint scheme should take precedence over OTEL_EXPO…

211d9c9

…RTER_OTLP_TRACES_INSECURE (open-telemetry#2060)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter-helper] [resource_to_label_conversion] add helper function and expose exporter settings #2060

[exporter-helper] [resource_to_label_conversion] add helper function and expose exporter settings #2060

hossain-rayhan commented Nov 4, 2020

codecov bot commented Nov 4, 2020 •

edited

Loading

hossain-rayhan commented Nov 5, 2020

hossain-rayhan commented Nov 6, 2020 •

edited

Loading

bogdandrutu commented Nov 6, 2020

bogdandrutu left a comment

hossain-rayhan commented Nov 6, 2020

bogdandrutu commented Nov 6, 2020

hossain-rayhan commented Nov 6, 2020

bogdandrutu left a comment

hossain-rayhan commented Nov 6, 2020

bogdandrutu left a comment

bogdandrutu Nov 7, 2020

hossain-rayhan Nov 7, 2020

bogdandrutu Nov 7, 2020

hossain-rayhan Nov 7, 2020

bogdandrutu Nov 7, 2020

bogdandrutu Nov 7, 2020

hossain-rayhan Nov 7, 2020

[exporter-helper] [resource_to_label_conversion] add helper function and expose exporter settings #2060

[exporter-helper] [resource_to_label_conversion] add helper function and expose exporter settings #2060

Conversation

hossain-rayhan commented Nov 4, 2020

codecov bot commented Nov 4, 2020 • edited Loading

Codecov Report

hossain-rayhan commented Nov 5, 2020

hossain-rayhan commented Nov 6, 2020 • edited Loading

bogdandrutu commented Nov 6, 2020

bogdandrutu left a comment

Choose a reason for hiding this comment

hossain-rayhan commented Nov 6, 2020

bogdandrutu commented Nov 6, 2020

hossain-rayhan commented Nov 6, 2020

bogdandrutu left a comment

Choose a reason for hiding this comment

hossain-rayhan commented Nov 6, 2020

bogdandrutu left a comment

Choose a reason for hiding this comment

bogdandrutu Nov 7, 2020

Choose a reason for hiding this comment

hossain-rayhan Nov 7, 2020

Choose a reason for hiding this comment

bogdandrutu Nov 7, 2020

Choose a reason for hiding this comment

hossain-rayhan Nov 7, 2020

Choose a reason for hiding this comment

bogdandrutu Nov 7, 2020

Choose a reason for hiding this comment

bogdandrutu Nov 7, 2020

Choose a reason for hiding this comment

hossain-rayhan Nov 7, 2020

Choose a reason for hiding this comment

codecov bot commented Nov 4, 2020 •

edited

Loading

hossain-rayhan commented Nov 6, 2020 •

edited

Loading