try-runtime-cli: improve ci stability

[liamaharon]

• Updates HttpClientBuilder config to fix some stability issues with the CI and increase the max possible batch size
• Limits parallelism to 4 threads. Performance improvement beyond 4 is negligible, and can lead to some weird stability issues with the HttpClient ("dispatch dropped without returning error" when using Client from tests hyperium/hyper#2112)
  • Ideally we would refactor the code to remove all threading and rely on tokio futures for request parallelism (or use just two threads: one for inserting keys and one dedicated for http requests using tokio futures). However limiting the number of threads seems sufficient for fixing the CI issue for now. Larger refactor will come later, probably when we implement lazy-download.
• CI tested against this branch here: [DNM] test try-runtime ci stability polkadot#7133
  • westend still failing, not due to download but due to an on-runtime-upgrade try-runtime hook actually failing: the on_chain version is equal or more than the current one. I guess we need to update its Migrations struct? If so, I'm happy to handle that in a seperate PR.

Closes #14013 and #14006

[niklasad1]

Looks good to me

It would be safer to create a new HttpClient in each thread spawned in the "remote externalities" than to limit the number of threads, fine for now

[liamaharon]

Looks good to me

It would be safer to create a new HttpClient in each thread spawned in the "remote externalities" than to limit the number of threads, fine for now

Yup I was considering this but it seems like it would require a larger refactor. We'll be doing a large refactor later anyway for lazy-download, and CI seems happy with this configuration, so decided not to do it here.

[ggwpez]

Thanks!