DeriveFromObject slow when using ObservedObjectCollection

[haarchri]

What happened?

have an issue with ObservedObjectCollection in provider-kubernetes - When I start using them, all other objects with spec.readiness.policy set to DeriveFromObject become extremely slow.
It takes up to 15 minutes to get the status updated, whereas without ObservedObjectCollection, it takes around 5 minutes. The ObservedObjectCollection is not used in combination with these other objects - they are independent of each other

NAME                                                                  KIND                      PROVIDERCONFIG                               SYNCED   READY   AGE
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-8hsn8   Provider                  ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp     True     False   5m23s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-9ntqr   Provider                  8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp     True     False   5m23s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-ftw5w   Provider                  ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp     True     False   5m23s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-gmsj2   Provider                  8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp     True     False   5m23s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-jsdxk   Provider                  8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp     True     False   5m23s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-tdjjf   Provider                  ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp     True     False   5m24s

NAME                                                                                    KIND   PROVIDERCONFIG                             SYNCED   READY   AGE
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-6bxtc          ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp   True     True    5m24s
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-8gcc2          ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp   True     True    5m23s
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-btwcs          8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp   True     True    5m24s
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-bxrvn          8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp   True     True    5m24s
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-qlqff          ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp   True     True    5m23s
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-wgbjq          8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp   True     True    5m23s

Also, the observed objects are not created - it takes around 15-25 minutes.
Interestingly, when I switch spec.readiness.policy to default, all resources are created immediately.
So, something is blocking when using ObservedObjectCollection in combination with objects that have DeriveFromObject set for spec.readiness.policy

NAME                                                                                    KIND   PROVIDERCONFIG                             SYNCED   READY   AGE
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-6klh7-c167b6f   ProviderRevision          8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp     True     True    2s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-8hsn8           Provider                  ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp     True     True    7m13s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-9ntqr           Provider                  8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp     True     True    7m13s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-ftw5w           Provider                  ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp     True     True    7m13s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-gmsj2           Provider                  8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp     True     True    7m13s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-h6gbk-9e50f04   ProviderRevision          8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp     True     True    1s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-jsdxk           Provider                  8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp     True     True    7m13s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-jv77w-64835f1   ProviderRevision          ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp     True     True    2s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-tdjjf           Provider                  ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp     True     True    7m14s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-tvh2s-9e50f04   ProviderRevision          ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp     True     True    2s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-vdvzs-64835f1   ProviderRevision          8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp     True     True    3s
object.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-xj22n-c167b6f   ProviderRevision          ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp     True     True    1s

NAME                                                                                    KIND   PROVIDERCONFIG                             SYNCED   READY   AGE
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-6bxtc          ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp   True     True    7m34s
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-8gcc2          ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp   True     True    7m33s
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-btwcs          8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp   True     True    7m34s
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-bxrvn          8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp   True     True    7m34s
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-qlqff          ea1e5ffc-8798-42ac-bad1-f69a127fa151-ctp   True     True    7m33s
observedobjectcollection.kubernetes.crossplane.io/upbound-upbound-aws-us-east-1-wgbjq          8d24ce56-40df-4db9-ae7d-96c823d2f564-ctp   True     True    7m33s

How can we reproduce it?

What environment did it happen in?

Crossplane version: