Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resolve slow http response parse time ( potentially enable gzip feature) #99

Open
rkdud007 opened this issue Jun 26, 2024 · 0 comments
Open
Labels
optimization Improvements of performance or optimization P-middle Middle priority, for near future

Comments

@rkdud007
Copy link
Member

rkdud007 commented Jun 26, 2024

Note: http response size is more than 10MB

Fortunately seems solution : https://users.rust-lang.org/t/using-json-with-response-data-in-reqwest-is-slow/95346/19 --- one questionable thing is, compare to this case we need to parse all of the response anyway. not sure how much improvement it will give

With latest version of indexer, we can interact much more blocks as response, and this works well in curl.

However with exact query ( fetching like 2k ~16k ) indexer wrapper of hdp doesn't have same performance.

I left log, basically fetching time it self is super similar from curl however, parsing part takes alot of time ...

get_headers_proof took 3021 ms
start parsing response
json response took 21787 ms
2024-06-26T14:29:37.982150Z  INFO hdp_provider::evm::provider: Time taken (Headers Proofs Fetch): 26.856025709s

so, this single line takes like 20s !!

 let body: Value = response.json().await.map_err(IndexerError::ReqwestError)?;
@rkdud007 rkdud007 added P-middle Middle priority, for near future optimization Improvements of performance or optimization labels Jun 26, 2024
@rkdud007 rkdud007 added this to the hdp v0.3.0 milestone Jun 26, 2024
@rkdud007 rkdud007 changed the title Investigate parsing http response super slow with indexer enable gzip feature to resolve slow http response parse time Jun 26, 2024
@rkdud007 rkdud007 changed the title enable gzip feature to resolve slow http response parse time resolve slow http response parse time ( potentially enable gzip feature) Jun 26, 2024
@rkdud007 rkdud007 removed this from the hdp v0.3.0 milestone Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
optimization Improvements of performance or optimization P-middle Middle priority, for near future
Projects
None yet
Development

No branches or pull requests

1 participant