[FFmpeg-user] Stitching two overlapping fisheye videos with maskedmerge

Michael Koch astroelectronic at t-online.de
Tue Mar 3 11:33:10 EET 2020


I'd like to stitch two fisheye videos together to an equirectangular 
video. The input videos have larger than 180° field of view, so that 
there is an overlap region and I want to use the maskedmerge filter for 
merging the videos.
My first (successful) test is with png images:

First create the mask file for maskedmerge:
ffmpeg -f lavfi -i nullsrc=size=640x640 -vf 
"format=gray8,geq='clip(128-128/10*(180-191.5/(640/2)*hypot(X-640/2,Y-640/2)),0,255)',v360=fisheye:e:ih_fov=191.5:iv_fov=191.5,format=rgb24" 
-frames 1 -y mergemap.png
The pixel values in this image are 0 for one fisheye region, 255 for the 
other fisheye region and a 0-255 ramp in the overlap region.

Now create two fisheye inputs (one is red and the other is green) and 
transform them to equirectangular:
ffmpeg -f lavfi -i color=red:size=640x640 -lavfi 
"drawbox=w=1:h=1:color=black,v360=input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5" 
-frames 1 -y eq1.png
ffmpeg -f lavfi -i color=green:size=640x640 -lavfi 
"drawbox=w=1:h=1:color=black,v360=input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5:yaw=180" 
-frames 1 -y eq2.png

Now merge the two images with maskedmerge:
ffmpeg -i eq1.png -i eq2.png -i mergemap.png -lavfi "maskedmerge" 
-frames 1 -y out.png

Up to this point, everything did work as expected. The output image 
looks correct. As the next step, I want to do the same thing with 
videos. I replace the file extensions by *.mp4, I replace "-frames 1" by 
"-t 5", and I add "-loop 1" for the third input file (which is the same 
as before):
ffmpeg -f lavfi -i color=red:size=640x640 -lavfi 
"drawbox=w=1:h=1:color=black,v360=input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5" 
-t 5 -y eq1.mp4
ffmpeg -f lavfi -i color=green:size=640x640 -lavfi 
"drawbox=w=1:h=1:color=black,v360=input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5:yaw=180" 
-t 5 -y eq2.mp4
ffmpeg -i eq1.mp4 -i eq2.mp4 -loop 1 -i mergemap.png -lavfi 
"maskedmerge" -t 5 -y out.mp4

The intermediate videos eq1.mp4 and eq2.mp4 look correct, but the output 
video is wrong.
I did already try many format conversions, but I haven't yet figured out 
what's going wrong here.
Below is the console output for the last command line.

Michael


C:\Users\mKoch\Desktop>c:\ffmpeg\ffmpeg -i eq1.mp4 -i eq2.mp4 -loop 1 -i 
mergema
p.png -lavfi "maskedmerge" -t 5 -y out.mp4
ffmpeg version git-2020-03-01-1281399 Copyright (c) 2000-2020 the FFmpeg 
develop
ers
   built with gcc 9.2.1 (GCC) 20200122
   configuration: --enable-gpl --enable-version3 --enable-sdl2 
--enable-fontconfi
g --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d 
--enable-libb
luray --enable-libfreetype --enable-libmp3lame 
--enable-libopencore-amrnb --enab
le-libopencore-amrwb --enable-libopenjpeg --enable-libopus 
--enable-libshine --e
nable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame 
--enable
-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 
--enable-libx265 -
-enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib 
--enable-gmp --enab
le-libvidstab --enable-libvorbis --enable-libvo-amrwbenc 
--enable-libmysofa --en
able-libspeex --enable-libxvid --enable-libaom --enable-libmfx 
--enable-ffnvcode
c --enable-cuda-llvm --enable-cuvid --enable-d3d11va --enable-nvenc 
--enable-nvd
ec --enable-dxva2 --enable-avisynth --enable-libopenmpt --enable-amf
   libavutil      56. 42.100 / 56. 42.100
   libavcodec     58. 73.102 / 58. 73.102
   libavformat    58. 39.101 / 58. 39.101
   libavdevice    58.  9.103 / 58.  9.103
   libavfilter     7. 77.100 /  7. 77.100
   libswscale      5.  6.100 /  5.  6.100
   libswresample   3.  6.100 /  3.  6.100
   libpostproc    55.  6.100 / 55.  6.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'eq1.mp4':
   Metadata:
     major_brand     : isom
     minor_version   : 512
     compatible_brands: isomiso2avc1mp41
     encoder         : Lavf58.39.101
   Duration: 00:00:05.00, start: 0.000000, bitrate: 22 kb/s
     Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 
1280x640
[SAR 1:1 DAR 2:1], 19 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
     Metadata:
       handler_name    : VideoHandler
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'eq2.mp4':
   Metadata:
     major_brand     : isom
     minor_version   : 512
     compatible_brands: isomiso2avc1mp41
     encoder         : Lavf58.39.101
   Duration: 00:00:05.00, start: 0.000000, bitrate: 21 kb/s
     Stream #1:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 
1280x640
[SAR 1:1 DAR 2:1], 17 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
     Metadata:
       handler_name    : VideoHandler
Input #2, png_pipe, from 'mergemap.png':
   Duration: N/A, bitrate: N/A
     Stream #2:0: Video: png, rgb24(pc), 1280x640 [SAR 1:1 DAR 2:1], 25 
fps, 25 t
br, 25 tbn, 25 tbc
Stream mapping:
   Stream #0:0 (h264) -> maskedmerge:base
   Stream #1:0 (h264) -> maskedmerge:overlay
   Stream #2:0 (png) -> maskedmerge:mask
   maskedmerge -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
[libx264 @ 00000000029ac840] using SAR=1/1
[libx264 @ 00000000029ac840] using cpu capabilities: MMX2 SSE2Fast SSSE3 
SSE4.2
AVX FMA3 BMI2 AVX2
[libx264 @ 00000000029ac840] profile High, level 3.1, 4:2:0, 8-bit
[libx264 @ 00000000029ac840] 264 - core 159 - H.264/MPEG-4 AVC codec - 
Copyleft
2003-2019 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 
deblock=1
:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 
me_rang
e=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 
chroma_qp_
offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 
interla
ced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 
b_adapt=1 b_bias
=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 
scenecut=40
intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 
qpmin=0 qpma
x=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'out.mp4':
   Metadata:
     major_brand     : isom
     minor_version   : 512
     compatible_brands: isomiso2avc1mp41
     encoder         : Lavf58.39.101
     Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 
1280x640 [S
AR 1:1 DAR 2:1], q=-1--1, 25 fps, 12800 tbn, 25 tbc (default)
     Metadata:
       encoder         : Lavc58.73.102 libx264
     Side data:
       cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=   47 fps=0.0 q=0.0 size=       0kB time=00:00:00.00 bitrate=N/A 
speed=
frame=   96 fps= 93 q=28.0 size=       0kB time=00:00:01.72 bitrate=   
0.2kbits/
frame=  125 fps= 88 q=-1.0 Lsize=      15kB time=00:00:04.88 bitrate=  
24.7kbits
/s speed=3.44x
video:12kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB 
muxing ov
erhead: 18.545912%
[libx264 @ 00000000029ac840] frame I:1     Avg QP:17.92  size:  6000
[libx264 @ 00000000029ac840] frame P:31    Avg QP:16.01  size:    71
[libx264 @ 00000000029ac840] frame B:93    Avg QP:12.67  size:    41
[libx264 @ 00000000029ac840] consecutive B-frames:  0.8%  0.0%  0.0% 99.2%
[libx264 @ 00000000029ac840] mb I  I16..4:  9.3% 89.0%  1.7%
[libx264 @ 00000000029ac840] mb P  I16..4:  0.0%  0.0%  0.0% P16..4:  
0.1%  0.3
%  0.0%  0.0%  0.0%    skip:99.5%
[libx264 @ 00000000029ac840] mb B  I16..4:  0.0%  0.0%  0.0% B16..8:  
0.1%  0.0
%  0.0%  direct: 0.0%  skip:99.9%  L0:84.8% L1:15.2% BI: 0.0%
[libx264 @ 00000000029ac840] 8x8 transform intra:88.9% inter:100.0%
[libx264 @ 00000000029ac840] coded y,uvDC,uvAC intra: 1.8% 7.4% 6.6% 
inter: 0.0%
  0.0% 0.0%
[libx264 @ 00000000029ac840] i16 v,h,dc,p: 56% 17% 28%  0%
[libx264 @ 00000000029ac840] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 68%  3% 28%  
0%  0%
  0%  0%  0%  0%
[libx264 @ 00000000029ac840] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 42% 26% 19%  
3%  1%
  3%  5%  2%  2%
[libx264 @ 00000000029ac840] i8c dc,h,v,p: 88%  6%  6%  1%
[libx264 @ 00000000029ac840] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 00000000029ac840] ref P L0: 99.2%  0.2%  0.4%  0.2%
[libx264 @ 00000000029ac840] ref B L0: 19.8% 80.2%
[libx264 @ 00000000029ac840] kb/s:19.25



More information about the ffmpeg-user mailing list