Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(specs): Add specification for output buffer persistence strategy #14928

Merged
merged 3 commits into from
Mar 15, 2024

Conversation

DStrand1
Copy link
Member

@DStrand1 DStrand1 commented Mar 4, 2024

Summary

Add specification for Telegraf's behavior on persisting output buffer metrics to disk

Checklist

  • No AI generated code was used in this PR

Related issues

Related to #802 and #14805

@telegraf-tiger telegraf-tiger bot added the docs Issues related to Telegraf documentation and configuration descriptions label Mar 4, 2024
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @DStrand1 for the nice write-up!

Please change the filename to tsd-005 as we do still have two old, not-yet-public specs to do (on my list). This way we can keep chronological order.

Can we please also specify what happens in case of errors for disk write, e.g. permissions denied or disk full?

docs/specs/tsd-003-output-buffer-strategy.md Outdated Show resolved Hide resolved
docs/specs/tsd-003-output-buffer-strategy.md Outdated Show resolved Hide resolved
Copy link
Contributor

@powersj powersj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for putting this together! Some questions from side as well.

docs/specs/tsd-003-output-buffer-strategy.md Outdated Show resolved Hide resolved
docs/specs/tsd-003-output-buffer-strategy.md Outdated Show resolved Hide resolved
docs/specs/tsd-003-output-buffer-strategy.md Outdated Show resolved Hide resolved
docs/specs/tsd-003-output-buffer-strategy.md Outdated Show resolved Hide resolved
## Is/Is-not
- Is a way to increase the durability of metrics and reduce the potential
for metrics to be dropped due to a full in-memory buffer
- Is not a way to guarantee data safety in the event of a crash or system failure
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excuse me if I am imposing on an internal review, saw the ping over at #802

The write-through feature seems to be able to support improved data safety in cases of crash or system failures as far as I have read about it above. Is the emphasis here on "guarantee"? E.g in that no guarantees provided, but a design goal of the feature is to improve data safety in these cases too?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal of this spec is not to ensure data safety, but to not drop metrics when the memory buffer fills. Inherently this will improve data safety, especially with write-through, though this is not the goal of this spec and as such cannot be completely guaranteed

Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@srebhan srebhan removed their assignment Mar 14, 2024
@srebhan srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Mar 14, 2024
Copy link
Contributor

@powersj powersj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@powersj powersj merged commit 9d7de5e into influxdata:master Mar 15, 2024
13 of 15 checks passed
@github-actions github-actions bot added this to the v1.30.1 milestone Mar 15, 2024
powersj pushed a commit that referenced this pull request Apr 1, 2024
@DStrand1 DStrand1 deleted the buffer-strategy-spec branch October 11, 2024 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Issues related to Telegraf documentation and configuration descriptions ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. specification
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants