Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read timed out issue with CosmosDB #37079

Open
maxiedaniels opened this issue Aug 28, 2024 · 5 comments
Open

Read timed out issue with CosmosDB #37079

maxiedaniels opened this issue Aug 28, 2024 · 5 comments
Assignees
Labels
Client This issue points to a problem in the data-plane of the library. Cosmos customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.

Comments

@maxiedaniels
Copy link

  • Package Name: azure-cosmos
  • Package Version: 4.5.1
  • Operating System: Mac 13.6.8
  • Python Version: 3.9.15

I'm using FastAPI, and have setup cosmos in my app.state:

app.state.cosmos_client = CosmosClient(
        cosmosdb_url,
        credential=cosmosdb_key,
        retry_total=3,
        retry_backoff_max=10,
        retry_on_status_codes=[429, 500, 503],
        retry_backoff_factor=1.5,
    )

I'm using a dev clone of my CosmosDB database, so it's only myself hitting the database right now.

Multiple times in the last twenty minutes, I've gotten an error:
ServiceResponseError, and all it says is read timed out, and says read timeout default is 300.

This occurs when doing:
users_container.query_items(query=query, parameters=parameters, enable_cross_partition_query=True))

I've checked on Azure and don't see any errors reported, or anything strange. My RU usage is hitting a max of 8%.

Funny enough, I remember putting in the retry values in the CosmosClient instantiation because of this exact issue. I also have Sentry running on our production server, and I can see this error comes up a decent amount.

Any idea why this would be happening?

@github-actions github-actions bot added Client This issue points to a problem in the data-plane of the library. Cosmos customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team. labels Aug 28, 2024
Copy link

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @AbhinavTrips @bambriz @Pilchie @pjohari-ms @simorenoh.

@kristapratico
Copy link
Member

@maxiedaniels thanks for your issue, the team will take a look and get back to you as soon as possible.

@maxiedaniels
Copy link
Author

Checking in on this. It's still happening, very infrequently but I've had customers with issues and when testing myself locally, it occasionally comes up.

@simorenoh
Copy link
Member

simorenoh commented Oct 10, 2024

Hi @maxiedaniels, sorry about the delay getting back to you. Could you please enable cosmos diagnostics logging following this guide here (https:/Azure/azure-sdk-for-python/tree/main/sdk/cosmos/azure-cosmos#logging-diagnostics) and share some of those with us as this happens? ServiceResponseErrors are rare and generally have to do with network conditions more so than anything SDK related, but with some diagnostics we may be able to find more about this.

@maxiedaniels
Copy link
Author

I will but it happens intermittently so it may be a minute. Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Client This issue points to a problem in the data-plane of the library. Cosmos customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention Workflow: This issue is responsible by Azure service team.
Projects
Status: No status
Development

No branches or pull requests

3 participants