[FFmpeg-user] hstack with one video offset in time (and keep audio synced)?

[Steven Kan]

> On Mar 6, 2021, at 11:22 AM, Steven Kan <steven at kan.org> wrote:
> 
>> On Mar 5, 2021, at 2:00 PM, Michael Koch <astroelectronic at t-online.de> wrote:
>> 
>> Am 05.03.2021 um 20:33 schrieb Steven Kan:
>>>>>> I’d like to assemble these videos, side-by-side, but synced in time, which
>>>>>> means the TrailDown video needs to start 50 seconds after the TrailUp
>>>>>> video. 
>> 
>> try this command line:
>> 
>> ffmpeg -i input1.mp4 -i input2.mp4 -filter_complex "[0]tpad=start_duration=50[a];[a][1]hstack” out.mp4
> 
> Thank you! That worked perfectly, and now that I understand the syntax, this works as well (to pad the end of the second track by 15 sec:
> 
> ffmpeg -i Input1.mp4 -i Input2 -filter_complex "[0]tpad=start_duration=50[a];[1]tpad=stop_duration=15[b];[a][b]hstack” Out.mp4

More on this! 

tpad=start_duration works to delay the start of one video, but now I need to fix the audio sync. For example this command:

ffmpeg -i TrailDown.mp4 -i TrailUp.mp4 -filter_complex "[0]tpad=start_duration=2.5[a];[a][1]hstack" -vcodec libx264 Coyote2Up.mp4

results in this video:

https://www.youtube.com/watch?v=_PDPONEU3YA#t=35s

Only the left half (TrailDown) camera has a microphone, and it’s apparent that the audio sync is off by the same 2.5 seconds that I’ve delayed its video. You can hear the female coyote (with the stumpy tail) scratching the ground 2.5 seconds before she actually does it.

What flag should I add to also delay its audio? Thanks!


ffmpeg -i /Users/steven/Downloads/Record/DownLoad/TrailDown.mp4 -i /Users/steven/Downloads/Record/DownLoad/TrailUp.mp4 -filter_complex "[0]tpad=start_duration=2.5[a];[a][1]hstack" -vcodec libx264 Coyote2Up.mp4
ffmpeg version N-100466-g29cef1bcd6-tessus  https://evermeet.cx/ffmpeg/  Copyright (c) 2000-2020 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
  libavutil      56. 62.100 / 56. 62.100
  libavcodec     58.115.102 / 58.115.102
  libavformat    58. 65.100 / 58. 65.100
  libavdevice    58. 11.103 / 58. 11.103
  libavfilter     7. 94.100 /  7. 94.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/steven/Downloads/Record/DownLoad/TrailDown.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    creation_time   : 2021-03-17T17:23:18.000000Z
  Duration: 00:00:46.15, start: 0.000000, bitrate: 6339 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 2592x1944, 6321 kb/s, 20 fps, 20 tbr, 1k tbn, 2k tbc (default)
    Metadata:
      creation_time   : 2021-03-17T17:23:18.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 8000 Hz, mono, fltp, 15 kb/s (default)
    Metadata:
      creation_time   : 2021-03-17T17:23:18.000000Z
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/steven/Downloads/Record/DownLoad/TrailUp.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    creation_time   : 2021-03-17T17:22:48.000000Z
    encoder         : Lavf58.45.100
  Duration: 00:00:50.20, start: 0.000000, bitrate: 6292 kb/s
    Stream #1:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 2592x1944, 6291 kb/s, 20 fps, 20 tbr, 16k tbn, 32k tbc (default)
    Metadata:
      creation_time   : 2021-03-17T17:22:48.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
File 'Coyote2Up.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 (h264) -> tpad (graph 0)
  Stream #1:0 (h264) -> hstack:input1 (graph 0)
  hstack (graph 0) -> Stream #0:0 (libx264)
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[aac @ 0x7fe8b501b600] Too many bits 8832.000000 > 6144 per frame requested, clamping to max
[libx264 @ 0x7fe8b5011c00] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x7fe8b5011c00] profile High, level 6.0, 4:2:0, 8-bit
[libx264 @ 0x7fe8b5011c00] 264 - core 161 r3027 4121277 - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=20 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'Coyote2Up.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.65.100
    Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(progressive), 5184x1944, q=2-31, 20 fps, 10240 tbn (default)
    Metadata:
      encoder         : Lavc58.115.102 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 8000 Hz, mono, fltp, 48 kb/s (default)
    Metadata:
      creation_time   : 2021-03-17T17:23:18.000000Z
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc58.115.102 aac
frame= 1004 fps=7.0 q=-1.0 Lsize=   77280kB time=00:00:50.05 bitrate=12649.0kbits/s speed=0.351x    
video:77094kB audio:167kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.025656%
[libx264 @ 0x7fe8b5011c00] frame I:5     Avg QP:17.47  size:1543698
[libx264 @ 0x7fe8b5011c00] frame P:468   Avg QP:19.13  size:140654
[libx264 @ 0x7fe8b5011c00] frame B:531   Avg QP:25.71  size: 10168
[libx264 @ 0x7fe8b5011c00] consecutive B-frames: 18.5% 27.5% 16.1% 37.8%
[libx264 @ 0x7fe8b5011c00] mb I  I16..4:  3.3% 34.7% 62.0%
[libx264 @ 0x7fe8b5011c00] mb P  I16..4:  0.1%  0.5%  0.5%  P16..4: 28.5%  4.7%  5.1%  0.0%  0.0%    skip:60.5%
[libx264 @ 0x7fe8b5011c00] mb B  I16..4:  0.0%  0.1%  0.0%  B16..8: 25.7%  0.5%  0.1%  direct: 0.1%  skip:73.4%  L0:33.7% L1:65.8% BI: 0.5%
[libx264 @ 0x7fe8b5011c00] 8x8 transform intra:40.7% inter:25.7%
[libx264 @ 0x7fe8b5011c00] coded y,uvDC,uvAC intra: 87.6% 0.0% 0.0% inter: 7.8% 0.0% 0.0%
[libx264 @ 0x7fe8b5011c00] i16 v,h,dc,p: 22% 15% 36% 27%
[libx264 @ 0x7fe8b5011c00] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 13% 12% 29%  6%  8%  8%  8%  8%  8%
[libx264 @ 0x7fe8b5011c00] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 15% 12%  7% 10% 10%  9%  9%  9%
[libx264 @ 0x7fe8b5011c00] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 0x7fe8b5011c00] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fe8b5011c00] ref P L0: 86.0%  5.4%  7.5%  1.1%
[libx264 @ 0x7fe8b5011c00] ref B L0: 92.9%  6.8%  0.2%
[libx264 @ 0x7fe8b5011c00] ref B L1: 96.6%  3.4%
[libx264 @ 0x7fe8b5011c00] kb/s:12580.63
[aac @ 0x7fe8b501b600] Qavg: 65230.441