Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

Error notifications despite the resource being successfully reconciled #3480

Closed
Diaoul opened this issue May 5, 2021 · 2 comments
Closed
Labels
flux2 Resolution suggested - already fixed in Flux v2

Comments

@Diaoul
Copy link

Diaoul commented May 5, 2021

Describe the bug

Flux sends out error level notifications despite the resource being successfully reconciled.
This is the discord notification I received, in that order:

helmrelease/jellyfin.media
Helm upgrade has started
revision
7.3.2
helmrelease/jellyfin.media
Helm upgrade succeeded
revision
7.3.2
helmrelease/jellyfin.media
reconciliation failed: Operation cannot be fulfilled on helmreleases.helm.toolkit.fluxcd.io "jellyfin": the object has been modified; please apply your changes to the latest version and try again
revision
7.3.2

And when I checked later:

$ flux get helmrelease -n media jellyfin
NAME    	READY	MESSAGE                         	REVISION	SUSPENDED
jellyfin	True 	Release reconciliation succeeded	7.3.2   	False 

All of this happened in a 2 minutes time window between the start of the reconciliation and the error notification.

To Reproduce

Hard to tell. No manual intervention was made besides updating the docker image in the values of the chart on the gitops repository, all those resources are managed by flux. Last time jellyfin was reconciled it worked fine. A week ago grafana reconciliation had the same error but not after so it does not seem to be related to a helm chart in particular.
My guess is that there is a conflict because flux tries to run two reconciliations at the same time of the same resource.

Expected behavior

Error notifications sent when reconciliation actually fails, maybe for a longer period of time? At least make this maybe a warning level on first occurrence.
I am not sure what should be done, but throwing an error seems wrong.

Logs

N/A

@Diaoul Diaoul added blocked-needs-validation Issue is waiting to be validated before we can proceed bug labels May 5, 2021
@kingdonb
Copy link
Member

kingdonb commented May 5, 2021

This repo is for Flux v1, which is in maintenance mode.

I think you are looking for https:/fluxcd/notification-controller or perhaps Flux v2 discussions, Q&A: https:/fluxcd/flux2/discussions/categories/q-a

I'm not sure there is enough to open an issue (instructions to reproduce are not especially clear) but you may get some clarity from one of the maintainers by asking about this in the Q&A section. I think you have understood the issue, it looks like what you expected: this notification means that a reconciliation was triggered while one was already in progress, which maybe is only an important error if it happens more than a few times in a row.

I suspect this is an error class notification, among info severity of notifications. The info notifications can be pretty noisy. You can tune them out by selecting error, but in this case it wouldn't have helped, since it was an error that contributed to the noise floor here, it would be even worse without the info context showing everything really did come out alright.

https://fluxcd.io/docs/components/notification/alert/

You can configure the alert with an ExclusionList, if this message is not ever important then you can always exclude it. Maybe that's enough to resolve this for you? I'm not sure what else to suggest, flux alerts are only based on the parameters that you configure. Hope this information is helpful!

@kingdonb kingdonb closed this as completed May 5, 2021
@kingdonb kingdonb added flux2 Resolution suggested - already fixed in Flux v2 and removed blocked-needs-validation Issue is waiting to be validated before we can proceed bug labels May 5, 2021
@Diaoul
Copy link
Author

Diaoul commented May 5, 2021

Thanks for your detailed answer, I know I can just make a custom rule to remove this error from even showing up but it seems wrong that this is an error as everything went well in the end. I thought I'd just let the maintainer know of this behavior and if no fix is possible I will ignore it.
I've opened the issue on the correct repo, sorry for the noise!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
flux2 Resolution suggested - already fixed in Flux v2
Projects
None yet
Development

No branches or pull requests

2 participants