[FFmpeg-trac] #8983(undetermined:new): Video Sync Support

FFmpeg trac at avcodec.org
Thu Nov 12 16:46:16 EET 2020


#8983: Video Sync Support
-------------------------------------+-------------------------------------
             Reporter:  villmer      |                     Type:
                                     |  enhancement
               Status:  new          |                 Priority:  normal
            Component:               |                  Version:
  undetermined                       |  unspecified
             Keywords:               |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 '''Synchronously''' acquiring two camera streams (such as V4L2) is
 '''critical''' for both recording (and live streaming of) stereoscopic
 video and, in our case, 360° video (using two back-to-back cameras). Both
 360° video and stereo video utilize ffmpeg framepack filter to place the
 two streams side-by-side.

 Unfortunately, the synchronous acquisition of two V4L2 video streams is
 currently not possible with ffmpeg. While two streams can be requested,
 there is '''always''' a delay of at least ~1second. Despite using various
 filters, such as itsoffset, setpts etc, we have been unable to synchronize
 two streams. As the ffmpeg team has already taken steps implementing
 various 360° video support, we would like to see dual-stream
 synchronization capabilities added to ffmpeg. In this way, ffmpeg can be
 directly used to '''create''' 360° video - instead of just performing
 post-production effects to it.

 Our current ffmpeg pipeline to capture two V4L2 streams and one alsa audio
 stream looks like this:

 ffmpeg -y \
 -f v4l2 -input_format h264 -video_size 1080x1080 -framerate 20
 -thread_queue_size 32k  -i /dev/video0 \
 -f v4l2 -input_format h264 -video_size 1080x1080 -framerate 20
 -thread_queue_size 32k  -i /dev/video1 \
 -f alsa -ac 2 -acodec pcm_s32le -sample_rate 11025 -thread_queue_size 32k
 -async 1 -itsoffset 0.3 -i hw:1,0 \
 -filter_complex "[0:v][1:v]framepack=sbs" \
 -f mp4 \
 -c:v libx264 \
 -b:v 2M \
 -b:a 128k \
 -preset ultrafast \
 -pix_fmt yuv420p \
 video.mp4

 The code above successfully records two V4L2 videos into a side-by-side
 formation, synchronously merges one alsa  audio stream (via -itsoffset
 0.3) and exports everything out to an mp4 file on-the-fly!

 The '''ONLY''' (unsolvable) problem is the ~1 second lag between videos.
 We are, quite literally, one second from a workable solution. However,
 removing this issue has not been successful despite many days of effort.

 On the Raspberry Pi (Compute Module 4) we can use the Raspivid application
 to capture frame-accurate, synchronous video from two cameras @ 30fps by
 simply adding the '''-3d sbs''' tag. We believe ffmpeg  oils potentially
 use similar techniques / code to synchronize the videos with a modified
 V4L2 driver. We have a new post on the Raspberry Pi forums describing
 this:
 https://www.raspberrypi.org/forums/viewtopic.php?f=61&t=290197&p=1754295#p1754295

 We are attaching a file showing the results of the code above. As you see,
 FFMPEG successfully places two (circular) videos side-by-side. The ONLY
 issue is the ~1second time lag.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/8983>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker


More information about the FFmpeg-trac mailing list