Green Screen Replacement


Our most recent project is green screen replacement.
The screen needs to be treated as green regardless of dark green or light green, and regardless of the brightness of the lighting, e.g. indoors or outdoors.
I used the HSV color space. A narrow threshhold was set on Hue component, with a wider threshold on Saturation, for what kind of green, and Value, for the lighting.
Here is my intuitive way of viewing HSV, which is not accurate, but worked in this project:
We make color by mixing paint with water, and shining light on it. Hue is the color of the paint, Saturation is how much paint to mix, Value is how much light to shine.

GreenDemo

Advertisements

Pixel format: RGB YUV and Bayer


The three have been confusing to me due to terminology. To be more accurate, I use rgb24 yuv420p and bayer_rggb8 instead, which can be found in the output of “ffmpeg -pix_fmts”

  1. rgb24: one byte for each RGB component, and therefore three bytes per pixel.
  2. yuv420p: one byte luma for each pixel, and two bytes chrominance is shared by four pixels. therefore, one and half bytes for each pixel.
  3. bayer_rggb8: only one byte per pixel. In another word, only one of the RGB color is saved for each pixel.

Their strength:

  1. rgb24: the most straightforward.
  2. yuv420p: 1. save memory space  2. in different color space of YCbCr, therefore gives a different perspective of the colors
  3. bayer_rggb8: save memory space, and it’s in the same color space as rgb24.

Camera sensors use bayer_rggb8 instead of rgb24 in order to lower the cost. For each pixel, it would need three sensors to capture all three RGB components.

HLS audio video decoupled: two audio ts streams share one video ts stream


Audio and Video can be specified separately. In another word, audio and video can be in separate .ts files, and therefore the same video can be shared by different audio, most commonly, for different languages. This feature was added in HLS version 4.
(VLC223 doesn’t support this; it has to be audio video in the same .ts file)
In this sample stream, http://riowing.net/local/SepAV/SepAV.htm, played by JW Player, the audio can shifted between English and Chinese.

Pitfall: it happened that some video works on Windows, but not on Android. Most likely it’s caused by the .ts files, instead of the .m3u8 playlists.

Reference: https://www.gpac-licensing.com/2014/12/01/apple-hls-comparing-versions/
Thanks to Jimmy for pointing this out.

SepAV

Headless Ubuntu Server Installation with GeniCam cameras


To test GeniCam cameras, I installed headless Linux, Ubuntu Server 16.04.4 LTS, on a laptop, without graphics, to mimic our AWS servers, none of which have GUI. The whole process takes about one hour, excluding research time, of course. Some key notes are listed below.

  1. Download: https://www.ubuntu.com/download/server  over 800M, only half size of the Ubuntu desktop version.
  2. Bootable USB: created from the .iSO above, by rufus-2.18p.exe
  3. Boot from USB: Boot in UEFI mode. Booting from CD needs legacy mode.
  4. Boot manager: GRUB2 version 2.02
  5. Partitions: sda1: UEFI boot partition, sda5: Linux OS boot volume, I set it to 10G. The default AWS Root Device is only 8G
  6. Wireless: different from Desktop Network Manager. Some key commands are: (wlp6s0 is wireless interface name)
    1. sudo ifconfig wlp6s0 up
    2. sudo iwlist wlp6s0 scan
    3. sudo iwconfig wlp6s0 essid “RwAtt” key mypassword
    4. sudo dhclient -v wlp6s0
  7. GeniCam Camera driver installation(Smartek as example):
    1. install libnl-route-3-200
    2. build kernel module from source, which ends up as /lib/modules/4.4.0-116-generic/kernel/drivers/video/u3vdrv.ko
    3. add configuration to /etc/ld.so.conf.d
  8. Pitfalls: I always get boot failure with message “no bootable device found” whenever trying to if=/dev/sda5, or changing boot mode to Legacy then back to UEFI. I think it’s BIOS bug. Fix: tell BIOS the GRUB binary location by typing in \EFI\ubuntu\grubx64.efi in boot options in BIOS