Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trace distributed work related to consuming SQS messages #4730

Merged
merged 8 commits into from
Feb 23, 2023

Conversation

mcculls
Copy link
Contributor

@mcculls mcculls commented Feb 14, 2023

Builds on top of #4739

What Does This Do

This PR enables more accurate distributed tracing of work related to consuming SQS messages.

The resulting traces (and switches) are modelled on existing messaging instrumentations such as Kafka and JMS.

It also avoids generating consuming spans when SQS.receiveMessage returns no messages.

Additional Notes

Example SQS distributed trace (default configuration)

The Java tracer will track work done while consuming SQS messages and attempt to connect that work back to the trace that produced those messages. The producing and consuming spans, as well as any work done consuming messages, will have a service name of sqs.

dd_sqs_legacy_tracing_enabled_true

Example SQS distributed trace with time-in-queue span

When legacy SQS tracing is disabled the Java Tracer will add a "time-in-queue" span representing the time the message spent on the queue between being produced and consumed. The "time-in-queue" span will have a service name of "sqs".

The producing and consuming spans, plus any work done related to that message, will now use the application's service name. Only the "time-in-queue" span will have a service name of "sqs".

-Ddd.sqs.legacy.tracing.enabled=false

DD_SQS_LEGACY_TRACING_ENABLED=false

dd_sqs_legacy_tracing_enabled_false

Example trace without any special SQS handling

If you want to restore the behaviour in 0.88.0 and earlier releases where AWS-SDK calls were modelled as simple HTTP client calls, and included the underlying HTTP span, then you can re-enable legacy AWS-SDK tracing with the following option.

-Ddd.aws-sdk.legacy.tracing.enabled=true

DD_AWS_SDK_LEGACY_TRACING_ENABLED=true

dd_aws_sdk_legacy_tracing_enabled=true

Example disconnected SQS trace

Tracking and including work related to SQS messages in the the original trace may lead to some very big traces.

Use the following setting on selected processes to break very long distributed traces into more manageable chunks.

-Ddd.sqs.propagation.enabled=false

DD_SQS_PROPAGATION_ENABLED=false

dd_sqs_propagation_enabled_false

Example disconnected SQS trace with time-in-queue span

You can also turn off propagation at the AWS-SDK level, which removes generation of X-Amzn-Trace-Id headers for all AWS-SDK calls, not just SQS.

Note you can still enable "time-in-queue" spans even when the X-Amzn-Trace-Id header is not available.

-Ddd.sqs.legacy.tracing.enabled=false -Ddd.aws-sdk.propagation.enabled=false

DD_SQS_LEGACY_TRACING_ENABLED=false; DD_AWS_SDK_PROPAGATION_ENABLED=false

dd_aws_sdk_propagation_enabled_false

@mcculls mcculls added the tag: do not merge Do not merge changes label Feb 14, 2023
@mcculls mcculls force-pushed the mcculls/instrument-sqs-receive-as-messaging branch 2 times, most recently from ac7d0eb to 7af72c3 Compare February 14, 2023 14:53
@mcculls mcculls changed the base branch from master to mcculls/simplify-aws-sdk-scope-activation February 14, 2023 14:59
@mcculls mcculls force-pushed the mcculls/simplify-aws-sdk-scope-activation branch from c117766 to f0a71fc Compare February 14, 2023 16:46
@mcculls mcculls force-pushed the mcculls/instrument-sqs-receive-as-messaging branch from 7af72c3 to 4b37543 Compare February 14, 2023 19:04
@mcculls mcculls force-pushed the mcculls/simplify-aws-sdk-scope-activation branch from f0a71fc to 544a628 Compare February 15, 2023 13:15
@mcculls mcculls force-pushed the mcculls/instrument-sqs-receive-as-messaging branch 8 times, most recently from 2cdee18 to 97a0248 Compare February 17, 2023 02:12
@mcculls mcculls changed the title [WIP] instrument SQS receive as messaging Instrument SQS receive as messaging [3 of 3] Feb 17, 2023
@mcculls mcculls added inst: others All other instrumentations tag: breaking change Breaking changes inst: aws sdk AWS SDK instrumentation and removed tag: do not merge Do not merge changes labels Feb 17, 2023
@mcculls mcculls marked this pull request as ready for review February 17, 2023 16:10
@mcculls mcculls requested a review from a team as a code owner February 17, 2023 16:10
@mcculls mcculls changed the title Instrument SQS receive as messaging [3 of 3] Trace distributed work related to consuming SQS messages [3 of 3] Feb 17, 2023
@mcculls mcculls force-pushed the mcculls/instrument-sqs-receive-as-messaging branch 4 times, most recently from f2bbe38 to 82882ac Compare February 17, 2023 19:42
Copy link
Contributor

@ygree ygree left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@mcculls mcculls force-pushed the mcculls/simplify-aws-sdk-scope-activation branch from 544a628 to 1b33ea5 Compare February 23, 2023 00:40
Base automatically changed from mcculls/simplify-aws-sdk-scope-activation to master February 23, 2023 01:21
@mcculls mcculls force-pushed the mcculls/instrument-sqs-receive-as-messaging branch from 82882ac to fe101e8 Compare February 23, 2023 01:25
@mcculls mcculls changed the title Trace distributed work related to consuming SQS messages [3 of 3] Trace distributed work related to consuming SQS messages Feb 23, 2023
@mcculls mcculls merged commit 9d11824 into master Feb 23, 2023
@mcculls mcculls deleted the mcculls/instrument-sqs-receive-as-messaging branch February 23, 2023 02:23
@github-actions github-actions bot added this to the 1.9.0 milestone Feb 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
inst: aws sdk AWS SDK instrumentation inst: others All other instrumentations tag: breaking change Breaking changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants