Added Python 3.5 “async for” compatibility

[flying-sheep]

Instead of

while True:
    msg = await dataqueue.read()
    ...

do:

async for msg in dataqueue:
    ...

TODO: how to handle EofStream?

[asvetlov]

EofStream should be converted to StopAsyncIterator.
Tests should be added.
Documentation -- updated.
What's about iterating over chunks also (.read(), .readexactly() and .readeany() should be supported too I believe).

[flying-sheep]

OK! but i don’t get the last point.

what do you mean by “supported”? we can only add one __aiter__ and one __anext__ function here.

we could of course add explicit asnc generator functions, e.g. aiter_any, which would crate an async iterator that loops data as available instead of lines.

readexactly is parameterized, so it makes no sense to make __anext__ defer to it. read and readany are basically the same when called unparameterized, so it makes no sense to distinguish.

so:

1. what should __anext__ do for the stream readers? readany or readline?
2. how to support the other? aiter_lines()/aiter_any()/aiter_exactly(n) methods? which of those?

[asvetlov]

Well, .readexactly() doesn't make sense.
My initial intention was adding a way by both iterating over fixed size chunks and any present data.

Something like

async for chunk in resp.content.iter_chunked(1024):
    ...

async for slice in resp.content.iter_any():
    ...

Does it make sense?

Default iteration for StreamReader should be line-mode, sure -- like iteration over opened text file.

[flying-sheep]

OK, then we’re on the same page 😄

we now only need a way to indicate in the docs that those methods are only available on python 3.5. is there a sphinx directive for that?

maybe we should also version-gate them?

class StreamReader(...):

    def __init__(self, ...):
        ...

    if python_version > (3, 5):
        def iter_any(self):
            ...

[asvetlov]

Yes, lease use check like https:/KeepSafe/aiohttp/blob/master/aiohttp/client.py#L447

I believe just versionadded directive with text like "available for Python 3.5+ only" should be enough.

We dropped Py3.3, I'm converting the doc to use async/await syntax so Py3.5 is preferable version for next release.

[flying-sheep]

OK. final thing for now: do i only need to document the iter_* functions? because __aiter__ is documented in the stdlib already.

[asvetlov]

Magic methods should not be documented itself but I would see usage example in class section, e.g. http://aiohttp.readthedocs.org/en/latest/client_reference.html#aiohttp.ClientResponse

[flying-sheep]

OK!

[flying-sheep]

hmm, i have this AsyncStreamReaderMixin which is used by StreamReader and EmptyStreamReader, but that results in the iter_chunked and iter_any methods not appearing in the docs.

can i force autodoc to treat the methods of AsyncStreamReaderMixin as if they were in StreamReader?

/edit: got it!

[asvetlov]

I believe we should document StreamReader manually, without autodoc.
Eventually we will drop autodoc at all.

[flying-sheep]

maybe after this is done?

[asvetlov]

ok, you can use autodoc for now if you want.

[flying-sheep]

done. this got more complex than i thought due to readexactly leaving this in a state where 0 bytes are left but EOF isn’t actually hit.

tests pass, flake8 also (why the hell does it complain about “unused variable _”? every other tool recognizes that as placeholder)

[asvetlov]

Just drop the file.
test_py35 is not a package.

[flying-sheep]

i explained that in the file:

This file allows files in here to be called the same as other test files.

if there is e.g. /tests/test_streams and /tests/test_py35/test_streams, then /tests/test_py35/__init__.py has to exist due to the module cache. try it if you don’t believe me 😄

[asvetlov]

this got more complex than i thought due to readexactly leaving this in a state where 0 bytes are left but EOF isn’t actually hit.

Hmm. Looks like a bug. I think .readexcaltly should raise EofStream after .feed_eof() call.
Anyway .iter_chunked should use .read not .readexactly I believe.

[flying-sheep]

done!

[flying-sheep]

@asvetlov good to merge?

[flying-sheep]

@asvetlov ping

[asvetlov]

Thanks!

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.