support max_concurrency
in upload_blob
and download_blob
operations
#420
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
max_concurrency=None
kwarg in async fs methods that use the azure SDKupload_blob
anddownload_blob
(which azure then uses to parallelize async chunk uploads/downloads)AzureBlobFileSystem.max_concurrency
attribute which is used whenever method-levelmax_concurrency
is not setfsspec.asyn._get_batch_size()
max_concurrency
defaults to 1 for the individual file upload/download.batch_size=...
is used to parallelize uploads/downloads at the file level (in something likefs._get()
orfs._put()
), and no additional parallelization is done for chunks within each file.batch_size
andmax_concurrency
. i.e.fs.get(path, batch_size=4, max_concurrency=2)
would download up to 4 files at a time, and up to 2 chunks at a time within each file, giving an overall concurrency of up to 8 async download coroutines being run in the loop at a time.Closes #268 (and supercedes the changes in the
concurrent_io
branch)This PR is incompatible with #329 (but there was discussion in that PR regarding changing the name of the parameter used to something other than
max_concurrency
since it conflicts with the the azure SDK parameter)