Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bugfix: Tenant index writer does not update age #1146

Merged
merged 2 commits into from
Dec 2, 2021

Conversation

joe-elliott
Copy link
Member

@joe-elliott joe-elliott commented Dec 1, 2021

What this PR does:
There is a bug where the tenant index age will not be updated correctly for the tenant index writer. To reproduce:

  • Scale to a bunch of compactors
  • Allow a poll cycle to complete.
  • Scale to 1 compactor. This compactor is now the tenant index writer for all tenants and so it will no longer update tempodb_blocklist_tenant_index_age_seconds for any tenants. All of the tenants it had previously been polling (and not writing) will now grow stale.
  • Watch the metrics for all tenants show stale even though it's not true.

To fix just force roll the compactors and they should reset their metrics correctly.

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Joe Elliott <[email protected]>
@joe-elliott joe-elliott changed the title Bugfix: Tenant index writer resets age Bugfix: Tenant index writer does not update age Dec 1, 2021
@joe-elliott joe-elliott merged commit 6d9200d into grafana:main Dec 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants