[Jetson Linux API Reference] Slightly different behavior: OE4T <-> L4T

[Aufschlauer]

Hi all,

First of all I'm not quite sure whether this is really a bug, so I start a discussion first...

Describe
When I run one of the Jetson Linux API Reference samples I get a different behavior on OE4T and L4T. I noticed this first on hardknott (32.5.1) and I just verified on dunfell (32.6.1) as well.

This is what it look like on dunfell (32.6.1):

$ cd /opt/tegra-mmapi/bin/
$ ./jpeg_decode num_files 1 ../data/Picture/nvidia-logo.jpg /tmp/nvidia-logo.yuv
Failed to query video capabilities: Inappropriate ioctl for device
Image Resolution - 1920 x 1080
libv4l2_nvvidconv (0):(802) (INFO) : Allocating (1) OUTPUT PLANE BUFFERS Layout=0
libv4l2_nvvidconv (0):(818) (INFO) : Allocating (1) CAPTURE PLANE BUFFERS Layout=0
Segmentation fault

Surprisingly one can find the decoded JPEG in /tmp/nvidia-logo.yuv!

On L4T it behaves different:

$ ./jpeg_decode num_files 1 ../../data/Picture/nvidia-logo.jpg ~jetson/nvidia-logo.yuv
Opening in BLOCKING MODE
Image Resolution - 1920 x 1080
libv4l2_nvvidconv (0):(802) (INFO) : Allocating (1) OUTPUT PLANE BUFFERS Layout=0
libv4l2_nvvidconv (0):(818) (INFO) : Allocating (1) CAPTURE PLANE BUFFERS Layout=0
App run was successful

Not mentioning that the decoded JPEG can be found in ~jetson/nvidia-logo.yuv.

I tried to get closer by running the with strace and it turns out that "Inappropriate ioctl for device" came from /dev/nvhost-nvdec:

openat(AT_FDCWD, "/dev/nvhost-nvdec", O_RDWR) = 13
...
ioctl(13, VIDIOC_QUERYCAP, 0x7fd9f31530) = -1 ENOTTY (Inappropriate ioctl for device)

So I wonder why in one case the IOCTL cmd runs fine and on the other case it is an "Inappropriate ioctl for device"

To Reproduce
Just follow the steps above.

Additional context
I searched the Wiki for something regarding the samples, but I failed...

[madisongh]

The "inappropriate ioctl" message can be ignored. One of the libv4l2 plugins is trying to query the decoder device, but the driver doesn't support that ioctl. I'm not sure why you don't see that on stock L4T, since I think the same plugin is installed there.

As for the segfault, it looks like it's happening in the shutdown/cleanup code, so it's after the conversion has been completed, which is why you see the output file.

The mmapi samples recipe doesn't get a whole lot of attention, and I don't often get a chance to exercise the samples. I often have problems with running them with each new release, partly due to toolchain differences, partly due to the C++ classes in the samples having some iffy coding, and partly due to our use of more (and possibly different) built-from-source bits. The makefiles also require a bunch of patches to get them to build in our environment, and there could be something wrong with those, too.

I'd welcome any help with getting them in better shape.

[madisongh]

I spent some time looking at this, and the segfault issue is a bit complicated.

It behaves differently in stock L4T because Debian (and thus Ubuntu) have a patch for glibc's NPTL implementation that has been around for over 10 years but has never made it upstream: link. This patch adds an extra check for a null thread ID before dereferencing it as a pointer. Stock glibc does not have that extra check.

Now, as part of the destructor of an NvV4l2Element object, NvV4l2ElementPlane::deinitPlane() is called on the output and capture planes. That method in turn calls the setStreamStatus() method, which sends a STREAMOFF v4l2_ioctl on the file descriptor. That apparently triggers the libv4l2_nvvidconv plugin to clean up a thread it has created. It calls pthread_join and apparently clears the thread ID for the thread it created.

The problem appears to be that a single v4l2 fd is shared between the element and both element plane objects. When the first STREAMOFF is performed, for the the thread ID gets cleared; when the second STREAMOFF happens for the other plane, the plugin again calls pthread_join, this time with the null thread ID. And boom, you get the segfault on our builds, but it works OK on Ubuntu.

I'm not sure whether it's a bug in the MMAPI or in the V4L2 plugin, though.

[hacker1024]

The libv4l2_nvvidconv plugin is no longer included in Jetpack, but I am observing a very similar issue still when attempting to encode video on NixOS with FFmpeg. Someone else reported it at Keylost/jetson-ffmpeg#4, but resolved their issue by presumably using the patched glibc.

I'm not sure what code is running now to cause the same clearing of the thread ID. The JPEG encoding sample actually works just fine. In fact, removing the two NVIDIA plugins (libv4l2_nvvideocodec.so and libv4l2_nvcuvidvideocodec.so) entirely does not prevent the error.

That apparently triggers the libv4l2_nvvidconv plugin to clean up a thread it has created. It calls pthread_join and apparently clears the thread ID for the thread it created.

How did you discover that libv4l2_nvvidconv was the culprit here?

Failed to query video capabilities: Inappropriate ioctl for device
libv4l2: error getting capabilities: Inappropriate ioctl for device

Thread 1 "ffmpeg" received signal SIGSEGV, Segmentation fault.
0x0000fffff54f6e4c in __pthread_clockjoin_ex () from /nix/store/i69jll18a0an6z14dq71b49xdl8kcqhk-glibc-2.39-5/lib/libc.so.6

(gdb) bt
#0  0x0000fffff54f6e4c in __pthread_clockjoin_ex () from /nix/store/i69jll18a0an6z14dq71b49xdl8kcqhk-glibc-2.39-5/lib/libc.so.6
#1  0x0000fffff43050ac in NvV4l2ElementPlane::waitForDQThread(unsigned int) ()
   from /nix/store/vbf8pgm4iwyqiic8706r5z2mpjf8b2jz-nvmpi-0-unstable-2023-06-26/lib/libnvmpi.so.1
#2  0x0000fffff43059cc in NvV4l2ElementPlane::deinitPlane() ()
   from /nix/store/vbf8pgm4iwyqiic8706r5z2mpjf8b2jz-nvmpi-0-unstable-2023-06-26/lib/libnvmpi.so.1
#3  0x0000fffff42f226c in NvV4l2Element::~NvV4l2Element() () from /nix/store/vbf8pgm4iwyqiic8706r5z2mpjf8b2jz-nvmpi-0-unstable-2023-06-26/lib/libnvmpi.so.1
#4  0x0000fffff4313bc0 in NvVideoEncoder::createVideoEncoder(char const*, int) ()
   from /nix/store/vbf8pgm4iwyqiic8706r5z2mpjf8b2jz-nvmpi-0-unstable-2023-06-26/lib/libnvmpi.so.1
#5  0x0000fffff42ebed0 in nvmpi_create_encoder () from /nix/store/vbf8pgm4iwyqiic8706r5z2mpjf8b2jz-nvmpi-0-unstable-2023-06-26/lib/libnvmpi.so.1
#6  0x0000fffff697db84 in nvmpi_encode_init (avctx=0x5e14c0) at libavcodec/nvmpi_enc.c:187
#7  0x0000fffff6a61e5c in avcodec_open2 (avctx=avctx@entry=0x5e14c0, codec=codec@entry=0xfffff74b1750 <ff_h264_nvmpi_encoder>, 
    options=options@entry=0x58f5a0) at libavcodec/avcodec.c:337
#8  0x0000000000412708 in enc_open (ost=ost@entry=0x58f490, frame=frame@entry=0x58ed50) at fftools/ffmpeg_enc.c:311
#9  0x0000000000413ed4 in enc_frame (ost=ost@entry=0x58f490, frame=frame@entry=0x58ed50) at fftools/ffmpeg_enc.c:855
#10 0x0000000000415fec in fg_output_frame (ofp=ofp@entry=0x58d1c0, frame=frame@entry=0x58ef40) at fftools/ffmpeg_filter.c:2125
#11 0x0000000000417f54 in fg_output_step (flush=0, ofp=0x58d1c0) at fftools/ffmpeg_filter.c:2220
#12 reap_filters (fg=0x5a4300, flush=flush@entry=0) at fftools/ffmpeg_filter.c:2239
#13 0x000000000040a304 in transcode_step (ost=<optimized out>) at fftools/ffmpeg.c:1155
#14 transcode (err_rate_exceeded=<synthetic pointer>) at fftools/ffmpeg.c:1204
#15 main (argc=<optimized out>, argv=<optimized out>) at fftools/ffmpeg.c:1330

[madisongh]

How did you discover that libv4l2_nvvidconv was the culprit here?

IIRC, I just spent a lot of time with gdb. I may have had to resort to doing some disassembly along the way, since the V4L2 plugin was provided in binary form.

We also apply a patch to the multimedia API sample classes. From the above backtrace, it looks like you may need that patch.

[hacker1024]

Thanks, that patch was it!

[madisongh]

I've opened #785 to track this, and I think I have a workaround we can use. I've also got some updates pending to how we build libv4l2 to address discrepancies between the stock upstream version and NVIDIA's version.

[madisongh]

The workaround has been merged into all of the currently-maintained branches, and their associated demo distro branches have also been updated to pick it up. There was one additional issue in the NvEglRenderer class that I've added a patch for in most of the demo-distro branches (hasn't yet made it to master). With these changes, I was able to get successful runs of all of the sample programs that I could - I didn't have my USB camera handy, so I couldn't test the ones that depend on having that hardware.

[madisongh]

Just to follow up... I've moved the recipes for building the argus and MMAPI samples into meta-tegra for the L4T R32.6.1-based branches (master and dunfell), and have added a test script to the corresponding branches in the demo distro to run through all the MMAPI samples (except the few multi-camera/multi-stream ones). This was a good exercise, as I uncovered a few more issues that required some additional patches to the samples, and a change to the way we were packaging the cuda-nvrtc libraries.

[Aufschlauer]

I just verified that my initial issue is solved. :)

Thank you for your prompt answer and fix!