Feature request: convert to image

[Tecol87]

Can you please add functions to convert the stream to a image from the image crate?
Or is there already a crate/examples which does this easily?

[raymanfx]

So stream.next(), e.g. for the memory-mapped IO transport (see: https:/raymanfx/libv4l-rs/blob/master/examples/stream_capture_mmap.rs#L35) gives you a buffer which is essentially a slice of bytes: &[u8].

It is important that you configure the v4l2 device to use a format that is supported by image-rs, e.g. RGB888. Most webcams will not support that, but only provide YUYV (aka Yuv422). I saw your comment on the image-rs repo and unfortunately it's still not something they support (AFAIK). You can use ffimage v0.10.0 from crates.io to convert a slice of YUYV bytes to a slice of RGB bytes (see: https:/raymanfx/ffimage/blob/next/ffimage-yuv/benches/convert.rs#L29).

You should be able to then use ImageBuffer::from_raw() from image-rs (see: https://docs.rs/image/latest/image/struct.ImageBuffer.html#method.from_raw) and pass it those bytes.

Let me know if this helps; and sorry for the late reply!

[sjbeskur]

Great question, I would really like to use this library but I too am struggling with the capture buffer conversion to an image format I can work with. The ffimage example is not clear to me .. any further guidance would be greatly appreciated.

[sjbeskur]

I think I have figured it out:

use image::{ImageBuffer};
use v4l::buffer::Type;
use v4l::io::traits::CaptureStream;
use v4l::prelude::*;
use v4l::video::Capture;    
use ffimage_yuv::yuv422::Yuv422;
use ffimage_yuv::yuv::Yuv;
use ffimage::color::Rgb;
use ffimage::iter::{BytesExt, ColorConvertExt, PixelsExt};
    
// in your main function (or whatever).

let mut dev = Device::new(0).expect("Failed to open device");
let format = dev.format()?;
let (buf, meta) = stream.next().unwrap();

// unwinding the expression below in comments:

let rgb: Vec<u8>  = buf.iter()
    .copied()                           // Vec<u8> 
    .pixels::<Yuv422<u8, 0, 2, 1, 3>>() // Vec<Yuv422<u8, 0, 2, 1, 3>> // after pixels
    .colorconvert::<[Yuv<u8>; 2]>()     // Vec<[Yuv<u8>;2]>  // after colorconvert::<[Yuv<u8>; 2]>()
    .flatten()                          // Vec<Yuv<u8>> // after flatten   
    .colorconvert::<Rgb<u8>>()          // Vec<Rgb<u8>> // after colorconvert::<Rgb<u8>>
    .bytes()                            // Vec<[u8;3]> // after bytes;
    .flatten()
    .collect();

// https:/raymanfx/libv4l-rs/issues/74
// NOTE: dont confuse image::Rgb with ffimage::Rgb

let img: ImageBuffer<image::Rgb<u8>, Vec<u8>> = ImageBuffer::from_raw(
    format.width, 
    format.height, 
    rgb,
)?;

img.save("test.png");

[raymanfx]

Yep, that looks about correct (given that the frames you get from the camera are indeed in the YUYV order).

[sjbeskur]

@raymanfx many thanks. This works for one of my cameras but my other camerahas a format that is Y16 (grayscale)? Seems it also supports Yu12 (YUV4:2:0) and NV12 (Y/CbCr 4:2:0) will your library work with these formats? Sorry for the dumb questions but this rather new to me.

[raymanfx]

Well this library is format agnostic. If you are asking whether ffimage will be able to convert those formats into packed RGB pixels, the answer is probably no. It should not be hard to add the conversion routines for those formats though. The existing YUV <-> RGB code should give you an idea of how to convert the image data.

There's even existing code handling YUV420p: https:/raymanfx/ffimage/blob/master/ffimage-yuv/src/yuv420.rs. So if you want RGB pixels, you'll need to implement the YU12 (planar) -> YUV444 (packed) conversion step and the complete pipeline would probably look like this: YU12 -> YUV444 -> RGB.

[sjbeskur]

Brilliant thank you @raymanfx . I was able to get it working wit the following snippet (replacing the critical section in the code above)

        let rgb: Vec<u8> = Yuv420p::pack(&buf, 640, 512)
            .into_iter()
            .colorconvert::<Rgb<u8>>()
            .bytes()
            .flatten()
            .collect();

The images look Ok, but this is a grayscale (IR camera) and I suspect this is not efficient. I'll study your suggestion and see what I can figure out.