Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python stdin buffering issue #18

Closed
punit-dato opened this issue Oct 29, 2015 · 10 comments
Closed

python stdin buffering issue #18

punit-dato opened this issue Oct 29, 2015 · 10 comments

Comments

@punit-dato
Copy link

Hey - I'm trying to the send/message functionality on python-shell to exchange data between a long-running python script and node, but I'm seeing that the messages don't get handled till I end the input stream. For example, the following sample code does not invoke the 'message' callback (or 'close' or 'error):

var pyshell = new PythonShell('python/helloworld.py', options);

pyshell.send({'hello': 'world'});

pyshell.on('message', function(message) {
  console.log("message - ", message);
}).on('close', function (result) {
  console.log("close - ", result);
}).on('error', function (err) {
  console.log("error")
});

The python is the same as your echo script.

import sys, json

# simple JSON echo script
for line in sys.stdin:
  print json.dumps(json.loads(line))

Now, if I close the connection after the above code, then I get all the messages:

pyshell.end(function (err) {
  if (err) throw err;
  console.log('finished');
});

This won't work for me as my python script maintains state and I don't want to re-run the script every time. I'd rather have a long running child process I could pass messages back and forth from. This seems like a bug. Any suggestions?

@extrabacon
Copy link
Owner

Stdout is being buffered by default, which is good for most apps. If your Node app does not receive messages, it means not enough data has been written.

You can explicitly flush the buffer to force the data to reach your Node app immediately.

See: http://stackoverflow.com/questions/10019456/usage-of-sys-stdout-flush-method

@punit-dato
Copy link
Author

Thanks for the quick response. Unfortunately even after calling flush after every print in my python code this still isn't working.

import sys, json

# simple JSON echo script
for line in sys.stdin:
  print json.dumps(json.loads(line))
  sys.stdout.flush()

I also tried the -u option, and the Unbuffered wrapper (which just does a flush as well) as per suggestions here: http://stackoverflow.com/questions/107705/disable-output-buffering

None of these seem to work. Any other thoughts?

@extrabacon
Copy link
Owner

Both stdin and stdout are buffered. This code writes to stdout what comes from stdin, so now you may want to flush on both sides.

@punit-dato
Copy link
Author

I can't find anything on flushing stdin. thoughts?

@extrabacon
Copy link
Owner

True, writable streams in node cannot be flushed that simply. Maybe this would help: http://stackoverflow.com/questions/18849112/stream-child-process-output-in-flowing-mode

Let me know if you can make it work.

@punit-dato
Copy link
Author

I spent way too long on this. Apparently "for line in stdin:" blocks until it sees an EOF in Python 2 (I haven't verified on Python 3).

The following code seems to work for me and will also terminate itself when it sees the EOF character. Also included is a flush after every print.

line=' '
while line:
    line = sys.stdin.readline()
    print line
    sys.stdout.flush()

@punit-dato punit-dato changed the title python-shell 'send' issue python stdin buffering issue Oct 30, 2015
@extrabacon
Copy link
Owner

I see, thank you for sharing your experience.

@JonathanLehner
Copy link

this should be way more prominent in the documentation. I spent 4 hours wondering why the send is only received after end() and if this is by design

@JonathanLehner
Copy link

however line='' didn't work for me.
https://stackoverflow.com/questions/31696444/iterating-over-standard-in-blocks-until-eof-is-read should be
-u Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode. Note that there is internal buffering in xread‐ lines(), readlines() and file-object iterators ("for line in sys.stdin") which is not influenced by this option. To work around this, you will want to use "sys.stdin.readline()" inside a "while 1:" loop.

@Almenon
Copy link
Collaborator

Almenon commented Jul 15, 2018

@JonathanLehner what do you mean by

however line='' didn't work for me

-u is already mentioned in the documentation I believe. You can submit a pull request if you want to change it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants