-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prometheus out of order sample from remote write #1622
Labels
bug
Something isn't working
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
Which version of the demo you are using? 1.10.0
Symptom
Prometheus logs indicate out-of-order sample errors and sometimes restarts.
What is the expected behavior?
No log message after
ts=2024-06-22T13:59:41.447Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..."
and Prometheus doesn't restartWhat is the actual behavior?
After some period of time (a few minutes) out of order sample logs are seen and in some cases I've experienced Prometheus restarts. For example:
While the
cartservice
/ .NET appears to be the most common, this is seen across other services/languages as well.Reproduce
Download version 1.10.0 and run
make start
. Once started, tail the Prometheus containers logs (tail -f <ID>
) and wait.Additional Context
Consider enabling out-of-order sample support available in Prometheus 2.39.x: prometheus/prometheus#11075
The text was updated successfully, but these errors were encountered: