Processes Die due to ValueError: I/O operation on closed file leads to incorrect status

[jsrawan-mobo]

In one of circusd workers we see:

ValueError: I/O operation on closed file
2015-10-15 00:56:15 tornado.application[58561] [ERROR] Future exception was never retrieved: Traceback (most recent call last):
  File "/opt/interana/third_party/lib/python2.7/site-packages/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/opt/interana/third_party/lib/python2.7/site-packages/circus/watcher.py", line 535, in manage_processes
    yield self.spawn_processes()
  File "/opt/interana/third_party/lib/python2.7/site-packages/tornado/gen.py", line 807, in run
    value = future.result()
  File "/opt/interana/third_party/lib/python2.7/site-packages/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "/opt/interana/third_party/lib/python2.7/site-packages/tornado/gen.py", line 212, in wrapper
    yielded = next(result)
  File "/opt/interana/third_party/lib/python2.7/site-packages/circus/watcher.py", line 592, in spawn_processes
    res = self.spawn_process()
  File "/opt/interana/third_party/lib/python2.7/site-packages/circus/watcher.py", line 627, in spawn_process
    self.stream_redirector.start()
  File "/opt/interana/third_party/lib/python2.7/site-packages/circus/stream/redirector.py", line 59, in start
    count += self._start_one(name, process, pipe)
  File "/opt/interana/third_party/lib/python2.7/site-packages/circus/stream/redirector.py", line 48, in _start_one
    fd = pipe.fileno()

As a result the status are incorrect, and do not match the stats (which are missing for processes that have died)

$ /opt/interana/third_party/bin/circusctl status
backstop_0: active
backstop_1: active
backstop_2: active
backstop_3: active
backstop_4: active
$ /opt/interana/third_party/bin/circusctl stats
backstop_4:
backstop_2:
backstop_3:
backstop_0:
57574: 57574  bash interana 0 2M 9M 0.0 0.0 0:00.20
   57577  perl interana 0 38M 87M 0.0 0.1 0:00.65
backstop_1:

When i restart one of the dead processes, stats shows it alive, along with ps

$ /opt/interana/third_party/bin/circusctl restart backstop_4
ok
$ /opt/interana/third_party/bin/circusctl stats
backstop_4:
59973: 59973  bash interana 0 2M 9M 0.0 0.0 0:00.0
   59974  perl interana 0 38M 87M 0.0 0.1 0:00.58
backstop_2:
backstop_3:
backstop_0:
58782: 58782  N/A interana 0 0B 0B 0.0 0.0 0:00.10
backstop_1:

Config file here

[watcher:backstop_0]
cmd = /home/ubuntu/pipeline-backstop/ubuntu-WUS-001
args = 0
numprocesses = 1
copy_env = True
priority = 1
stop_children = True
respawn = True
stop_signal = SIGKILL
uid = ubuntu
gid = ubuntu
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/ubuntu/backstop-0.log
stdout_stream.refresh_time = 0.3
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/ubuntu/backstop-0.log
stderr_stream.refresh_time = 0.3

[watcher:backstop_1]
cmd = /home/[watcher:backstop_0]
cmd = /home/ubuntu/pipeline-backstop/ubuntu-WUS-001
args = 0
numprocesses = 1
copy_env = True
priority = 1
stop_children = True
respawn = True
stop_signal = SIGKILL
uid = ubuntu
gid = ubuntu
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/ubuntu/backstop-0.log
stdout_stream.refresh_time = 0.3
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/ubuntu/backstop-0.log
stderr_stream.refresh_time = 0.3

[watcher:backstop_1]
cmd = /home/ubuntu/pipeline-backstop/ubuntu-WUS-001
args = 1
numprocesses = 1
copy_env = True
priority = 1
stop_children = True
respawn = True
stop_signal = SIGKILL
uid = ubuntu
gid = ubuntu
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/ubuntu/backstop-1.log
stdout_stream.refresh_time = 0.3
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/ubuntu/backstop-1.log
stderr_stream.refresh_time = 0.3

[watcher:backstop_2]
cmd = /home/ubuntu/pipeline-backstop/ubuntu-WUS-001
args = 2
numprocesses = 1
copy_env = True
priority = 1
stop_children = True
respawn = True
stop_signal = SIGKILL
uid = ubuntu
gid = ubuntu
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/ubuntu/backstop-2.log
stdout_stream.refresh_time = 0.3
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/ubuntu/backstop-2.log
stderr_stream.refresh_time = 0.3

[watcher:backstop_3]
cmd = /home/ubuntu/pipeline-backstop/ubuntu-WUS-001
args = 3
numprocesses = 1
copy_env = True
priority = 1
stop_children = True
respawn = True
stop_signal = SIGKILL
uid = ubuntu
gid = ubuntu
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/ubuntu/backstop-3.log
stdout_stream.refresh_time = 0.3
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/ubuntu/backstop-3.log
stderr_stream.refresh_time = 0.3

[watcher:backstop_4]
cmd = /home/ubuntu/pipeline-backstop/ubuntu-WUS-001
args = 4
numprocesses = 1
copy_env = True
priority = 1
stop_children = True
respawn = True
stop_signal = SIGKILL
uid = ubuntu
gid = ubuntu
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/ubuntu/backstop-4.log
stdout_stream.refresh_time = 0.3
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/ubuntu/backstop-4.log
stderr_stream.refresh_time = 0.3
ubuntu/pipeline-backstop/ubuntu-WUS-001
args = 1
numprocesses = 1
copy_env = True
priority = 1
stop_children = True
respawn = True
stop_signal = SIGKILL
uid = ubuntu
gid = ubuntu
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/ubuntu/backstop-1.log
stdout_stream.refresh_time = 0.3
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/ubuntu/backstop-1.log
stderr_stream.refresh_time = 0.3

[watcher:backstop_2]
cmd = /home/ubuntu/pipeline-backstop/ubuntu-WUS-001
args = 2
numprocesses = 1
copy_env = True
priority = 1
stop_children = True
respawn = True
stop_signal = SIGKILL
uid = ubuntu
gid = ubuntu
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/ubuntu/backstop-2.log
stdout_stream.refresh_time = 0.3
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/ubuntu/backstop-2.log
stderr_stream.refresh_time = 0.3

[watcher:backstop_3]
cmd = /home/ubuntu/pipeline-backstop/ubuntu-WUS-001
args = 3
numprocesses = 1
copy_env = True
priority = 1
stop_children = True
respawn = True
stop_signal = SIGKILL
uid = ubuntu
gid = ubuntu
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/ubuntu/backstop-3.log
stdout_stream.refresh_time = 0.3
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/ubuntu/backstop-3.log
stderr_stream.refresh_time = 0.3

[watcher:backstop_4]
cmd = /home/ubuntu/pipeline-backstop/ubuntu-WUS-001
args = 4
numprocesses = 1
copy_env = True
priority = 1
stop_children = True
respawn = True
stop_signal = SIGKILL
uid = ubuntu
gid = ubuntu
stdout_stream.class = FileStream
stdout_stream.filename = /var/log/ubuntu/backstop-4.log
stdout_stream.refresh_time = 0.3
stderr_stream.class = FileStream
stderr_stream.filename = /var/log/ubuntu/backstop-4.log
stderr_stream.refresh_time = 0.3

[k4nar]

I'm closing this in favor of #979 which got more traction.