qpidd unable to bind any port

[michaelinipop]

Hi Guys,

I figure you guys are dealing with similar bugs that I am experiencing. I was wanting to use QPIDD on the Linux bash build for a Microsoft conference talk I am doing. When I am starting the message broker for my application I get a nasty error stating that I am unable to bind my application to a port. I suspect it is an issues with the Accept socket function implementation.

*error code and source file
2016-08-27 16:02:22 critical Unexpected error: Can't bind to port [::]:60181: Address already in use (qpid/sys/posix/Socket.cpp:206)

https://apache.googlesource.com/qpid/+/asyncstore/cpp/src/qpid/sys/posix/BSDSocket.cpp

• Your Windows build number
  14393
• Steps / commands required to reproduce the error

Turned off firewall and ran the cmd in administrator mode.

bash
sudo apt-get install qpidd
sudo qpidd --port=0 or just sudo qpidd

• Strace of the failing command
  https://gist.github.com/michaelinipop/9907a6d8f47a2ce74ceb5becae3e14ba
• Required packages and commands to install
  qpidd

Thanks guys,

michael

[rodrymbo]

Just to check: run netstat -an | more from CMD and confirm that nothing on the Windows side is listening on that port. (Each port on the machine can only be bound once.)

Adding that to your report would increase its completeness.

[michaelinipop]

@rodrymbo thank you for the suggestion.

I did run netstat to confirm there is no interference.

before posting this bug submit I did try different ports. When you run -p0 the program tries to find a available port.

when I run sudo qpidd it uses the default amqp standard port 5672 if you look at a snippet from the strace the socket is created and also closed on exit of the error message . Therefore there is no chance the port being used by another process.

futex(0x7f486f4813c0, FUTEX_WAKE_PRIVATE, 2147483647) = 0 socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 9 connect(9, {sa_family=AF_INET6, sin6_port=htons(5672), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0 getsockname(9, {sa_family=AF_INET6, sin6_port=htons(56159), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 connect(9, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0 connect(9, {sa_family=AF_INET, sin_port=htons(5672), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 getsockname(9, {sa_family=AF_INET6, sin6_port=htons(56159), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 close(9)

[therealkenc]

Looks same as #674.

[sunilmut]

@michaelinipop - Thanks for reporting the issue and trying out WSL! @therealkenc is correct in pointing out that the issue is with the lack of proper support for IPV6_V6ONLY in WSL (see the trace snippets below). The problem is that WSL allows the app to set the IPV6_V6ONLY socket option to be set on a socket, but when it is not currently able to interact with the network layer to fully honor the option. In that case, when a socket sets this option and tries to bind to a IPV6 address on a given port, it also tries to reserve the same port for IPV4 traffic and runs into collisions as below. This is something we are aware of and are actively working with the networking team to resolve the issue at our end. Meanwhile, if you can force the app to only work with IPV4 address (or use different addresses for IPV4 and IPV6), you may be able to work around the issue. We apologize for the inconvenience, but hopefully the real fix is coming soon.

`
bind(9, {sa_family=AF_INET, sin_port=htons(5672), sin_addr=inet_addr("0.0.0.0")}, 16) = 0

setsockopt(10, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0

bind(10, {sa_family=AF_INET6, sin6_port=htons(5672), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EADDRINUSE (Address already in use)
`

[sunilmut]

With the help of the Windows networking team, we were able to fully implement the IPV6_V6ONLY flag. With that change the Address already in use issue is gone. But, unfortunately, the qpidd server is still not able to start. It fails in the futex - FUTEX_CMP_REQUEUE_PRIVATE flag, which is currently not implemented. I have opened a task to track this. The fix for IPV6_V6ONLY is still in our dev branch, but should soon hit the release branch.

[michaelinipop]

@sunilmut Thank you so much for following up on this. I can easily imagine how technical it is to support almost every edge case within the posix command set. Nevertheless, I look forward to seeing this working.

[therealkenc]

FUTEX_CMP_REQUEUE_PRIVATE (#1006 and friends) is rearing it's head more now that IPC over sockets is working much better. I wonder (out loud without a lot of basis) whether it can be stubbed in WSL somehow as a first step, rather than trying to get the semantics correct all at once. In most/all the OSS code I've come across, the software doesn't really care if there is a thundering herd of threads/processes waking up only to go right back to sleep. Because in reality there is no herd; there's a few cows standing around. The real problem is that glibc pthreads thinks the kernel functionality is there, so it gets used.

Anyway, just a thought. I haven't looked at the glibc pthread implementation to see if it is possible to just lie. The musl pthread implementation doesn't even use it AFAICT.

[sunilmut]

Just FYI that the IPV6_V6ONLY issue is fixed in 14936 (see release notes)

[benhillis]

I've added support for FUTEX_CMP_REQUEUE. The change will be in an Insider build soon.

[therealkenc]

The fixinbound tag sailed here back in August, so this is fixedinfallcreatorsupdate by fiat. If there are still problems with the qpidd (there might) please open a new issue following the template.