zebra (stream_set_getp) assert seen in topotest

[louberger]

seen on TopoTest Master arm8
https://ci1.netdef.org/browse/FRR-TOPOPR-TMARM8-256

error	11-Jul-2018 07:09:30	r4: zebra crashed. Core file found - Backtrace follows:
error	11-Jul-2018 07:09:30	[New LWP 10324]
error	11-Jul-2018 07:09:30	[New LWP 10312]
error	11-Jul-2018 07:09:30	[Thread debugging using libthread_db enabled]
error	11-Jul-2018 07:09:30	Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
error	11-Jul-2018 07:09:30	Core was generated by `/usr/lib/frr/zebra'.
error	11-Jul-2018 07:09:30	Program terminated with signal SIGABRT, Aborted.
error	11-Jul-2018 07:09:30	#0  0x0000ffffacf32528 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
error	11-Jul-2018 07:09:30	[Current thread is 1 (Thread 0xffffacb28090 (LWP 10324))]
error	11-Jul-2018 07:09:30	#0  0x0000ffffacf32528 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
error	11-Jul-2018 07:09:30	#1  0x0000ffffacf339e0 in __GI_abort () at abort.c:89
error	11-Jul-2018 07:09:30	#2  0x0000ffffad0cbc08 in _zlog_assert_failed (assertion=assertion@entry=0xffffad114db8 "(((s)->getp) <= (s)->endp)", file=file@entry=0xffffad114ce8 "lib/stream.c", line=line@entry=217, function=function@entry=0xffffad114988 <__func__.12690> "stream_set_getp") at lib/log.c:710
error	11-Jul-2018 07:09:30	#3  0x0000ffffad0e4fc4 in stream_set_getp (s=0xaaab1660db30, pos=pos@entry=0) at lib/stream.c:217
error	11-Jul-2018 07:09:30	#4  0x0000aaaadc5e2188 in zserv_read (thread=<optimized out>) at zebra/zserv.c:400
error	11-Jul-2018 07:09:30	#5  0x0000ffffad0ede38 in thread_call (thread=thread@entry=0xffffacb277c0) at lib/thread.c:1576
error	11-Jul-2018 07:09:30	#6  0x0000ffffad0bf228 in fpt_run (arg=0xaaab166186c0) at lib/frr_pthread.c:293
error	11-Jul-2018 07:09:30	#7  0x0000ffffad04efc4 in start_thread (arg=0xffffad0bf130 <fpt_run>) at pthread_create.c:335
error	11-Jul-2018 07:09:30	#8  0x0000ffffacfc72e0 in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:89
error	11-Jul-2018 07:09:30	2018-07-11 14:09:30,362 ERROR: assert failed at "test_ospf6_topo1/test_shutdown_check_stderr": 

[louberger]

full log plan-34865825-TMARM8-256.log

[qlyoung]

Root cause is a data race caused by deinitialization procedures in SIGINT handler made incorrect by introduction of pthreads

WARNING: ThreadSanitizer: data race (pid=22461)
  Read of size 8 at 0x7b0800010020 by main thread:
    #0 buffer_reset /home/vagrant/frr/lib/buffer.c:127:17 (libfrr.so.0+0x248e2)
    #1 buffer_free /home/vagrant/frr/lib/buffer.c:90:2 (libfrr.so.0+0x24888)
=>  #2 zserv_client_free /home/vagrant/frr/zebra/zserv.c:590:3 (zebra+0x592988)
    #3 list_delete_all_node /home/vagrant/frr/lib/linklist.c:216:4 (libfrr.so.0+0x81f30)
    #4 sigint /home/vagrant/frr/zebra/main.c:142:2 (zebra+0x4eb706)
    #5 quagga_sigevent_process /home/vagrant/frr/lib/sigevent.c:103:6 (libfrr.so.0+0xcd6db)
    #6 thread_fetch /home/vagrant/frr/lib/thread.c:1401:4 (libfrr.so.0+0xfa773)
    #7 frr_run /home/vagrant/frr/lib/libfrr.c:913:9 (libfrr.so.0+0x7f30a)
    #8 main /home/vagrant/frr/zebra/main.c:395:2 (zebra+0x4ebe5a)

  Previous write of size 8 at 0x7b0800010020 by thread T1:
    #0 buffer_flush_available /home/vagrant/frr/lib/buffer.c:471:17 (libfrr.so.0+0x25c75)
    #1 buffer_flush_all /home/vagrant/frr/lib/buffer.c:241:16 (libfrr.so.0+0x256be)
    #2 zserv_write /home/vagrant/frr/zebra/zserv.c:243:10 (zebra+0x5938de)
    #3 thread_call /home/vagrant/frr/lib/thread.c:1576:2 (libfrr.so.0+0xfc456)
    #4 fpt_run /home/vagrant/frr/lib/frr_pthread.c:293:4 (libfrr.so.0+0x5d7b0)

  Location is heap block of size 24 at 0x7b0800010020 allocated by main thread:
    #0 calloc <null> (zebra+0x457c4c)
    #1 qcalloc /home/vagrant/frr/lib/memory.c:72:27 (libfrr.so.0+0x8e306)
    #2 buffer_new /home/vagrant/frr/lib/buffer.c:70:6 (libfrr.so.0+0x24762)
    #3 zserv_client_create /home/vagrant/frr/zebra/zserv.c:659:15 (zebra+0x5920a2)
    #4 zserv_accept /home/vagrant/frr/zebra/zserv.c:731:2 (zebra+0x593f25)
    #5 thread_call /home/vagrant/frr/lib/thread.c:1576:2 (libfrr.so.0+0xfc456)
    #6 frr_run /home/vagrant/frr/lib/libfrr.c:914:3 (libfrr.so.0+0x7f320)
    #7 main /home/vagrant/frr/zebra/main.c:395:2 (zebra+0x4ebe5a)

  Thread T1 (tid=22537, running) created by main thread at:
    #0 pthread_create <null> (zebra+0x437026)
    #1 frr_pthread_run /home/vagrant/frr/lib/frr_pthread.c:174:8 (libfrr.so.0+0x5e2f3)
    #2 zserv_client_create /home/vagrant/frr/zebra/zserv.c:698:2 (zebra+0x5924a1)
    #3 zserv_accept /home/vagrant/frr/zebra/zserv.c:731:2 (zebra+0x593f25)
    #4 thread_call /home/vagrant/frr/lib/thread.c:1576:2 (libfrr.so.0+0xfc456)
    #5 frr_run /home/vagrant/frr/lib/libfrr.c:914:3 (libfrr.so.0+0x7f320)
    #6 main /home/vagrant/frr/zebra/main.c:395:2 (zebra+0x4ebe5a)