Video Codec Approaches

I have worked with two video compression standards for IP cameras: H.264 and MJPEG.

H.265 is not widely supported. We don’t use  WMV.

H.264 is by far the most popular.

Why transcoding: to overlay scoreboard info and commercials in the streamed sports game.

 SDK: Mainconcept 9.1.0 (MC), in both pure software and hardware accelerated.

 bufstream_tt is the key class from MC

The difference: DXVA2 is used for HW acceleration, DirectX Video Acceleration from Microsoft

Steps SW:

Setup: pass frame_tt to bufstream_tt to receive decoded frame

Retrieval after decoding: bufstream_tt.auxinfo(GET_PIC)

Steps HW:

Setup: IDirect3DDeviceManager9 -> dxva2_config -> bufstream_tt

Retrieval after decoding:

bufstream_tt.auxinfo(HWACC_GET_PIC) -> dxva_surface_t -> LockRect -> pBits

To check if DXVA2 is supported: IDirectXVideoDecoderService

MJPEG takes more space, but easy to decode.

SDK: IJG.lib from Independent JPEG Group


              JPEG image buffer -> jpeg_decompress_struct with JCS_YCbCr -> jpeg_read_scanlines -> fill YUV buffer

Other Image process such as muxing, overlay, scaling

Why: to transform from to interlaced to progressive, Picture in Picture, Video side by side




Video Rendering Approaches

1. DirectShow: old technology which started in the late 90s.

It can get the job done by constructing a graph with existing filters.
More flexibility can be achieved by customizing the rendering filters.
I used in 2010 to playback school buses surveillance video.

2. Direct3D:

This is like making a rendering filter but outside of DirectShow’s frame work, video frame as texture on a poly.
Great flexibility can be achieved, but Direct3D coding may take more time to get up to speed.
Input can be raw video video frame such as YUV420. The YUV gets copied to the texture, by the route below.
YUV -> CreateOffscreenPlainSurface.IDirect3DSurface9.D3DLOCKED_RECT.pBits -> IDirect3DTexture9.IDirect3DSurface9
I did this in 2014 in my project to render video from IP cameras.

3. GDI

for rendering GUI in the old days, not for video as it’s too slow.

How to keep a process running after logout with bash without help from disown and nohup

Necessary and Sufficient Conditions based on my test: (AWS Ubuntu EC2)
1. redirect input and output
2. put it to background by & sign

By doing the two above, with neither disown nor nohup, my process keeps running even after logout from ssh.
However, I am not very clear why.
Example: for(( i = 0; i < 5; i++)); do sleep 2; echo $i; done t.txt 2>&1 &

HLS segmentation

Two ways to do this: server pull or video source push
Push: server runs nginx-rtmp-module, video comes in through RTMP
Pro: all video source devices can stay behind NAT in case no control on router.
Pull: server runs ffmpeg to pull video directly from cameras through RTSP.
Pro: no server needed on video source site, since server directly talks to cameras.
ffmpeg is far more feature rich than nginx-rtmp-module
rtmp-module doesn’t even have the option to organize segment into sub folders
(think hundrdeds of thousands of .ts files).

Extra credits to ffmpeg:
Some software stays stable by keeping the feature set small.
However, not only does Ffmpeg stays stable, but it has so many features and does every single one so well.