[FFmpeg-trac] #8983(undetermined:new): Video Sync Support
FFmpeg
trac at avcodec.org
Thu Nov 12 16:46:16 EET 2020
#8983: Video Sync Support
-------------------------------------+-------------------------------------
Reporter: villmer | Type:
| enhancement
Status: new | Priority: normal
Component: | Version:
undetermined | unspecified
Keywords: | Blocked By:
Blocking: | Reproduced by developer: 0
Analyzed by developer: 0 |
-------------------------------------+-------------------------------------
'''Synchronously''' acquiring two camera streams (such as V4L2) is
'''critical''' for both recording (and live streaming of) stereoscopic
video and, in our case, 360° video (using two back-to-back cameras). Both
360° video and stereo video utilize ffmpeg framepack filter to place the
two streams side-by-side.
Unfortunately, the synchronous acquisition of two V4L2 video streams is
currently not possible with ffmpeg. While two streams can be requested,
there is '''always''' a delay of at least ~1second. Despite using various
filters, such as itsoffset, setpts etc, we have been unable to synchronize
two streams. As the ffmpeg team has already taken steps implementing
various 360° video support, we would like to see dual-stream
synchronization capabilities added to ffmpeg. In this way, ffmpeg can be
directly used to '''create''' 360° video - instead of just performing
post-production effects to it.
Our current ffmpeg pipeline to capture two V4L2 streams and one alsa audio
stream looks like this:
ffmpeg -y \
-f v4l2 -input_format h264 -video_size 1080x1080 -framerate 20
-thread_queue_size 32k -i /dev/video0 \
-f v4l2 -input_format h264 -video_size 1080x1080 -framerate 20
-thread_queue_size 32k -i /dev/video1 \
-f alsa -ac 2 -acodec pcm_s32le -sample_rate 11025 -thread_queue_size 32k
-async 1 -itsoffset 0.3 -i hw:1,0 \
-filter_complex "[0:v][1:v]framepack=sbs" \
-f mp4 \
-c:v libx264 \
-b:v 2M \
-b:a 128k \
-preset ultrafast \
-pix_fmt yuv420p \
video.mp4
The code above successfully records two V4L2 videos into a side-by-side
formation, synchronously merges one alsa audio stream (via -itsoffset
0.3) and exports everything out to an mp4 file on-the-fly!
The '''ONLY''' (unsolvable) problem is the ~1 second lag between videos.
We are, quite literally, one second from a workable solution. However,
removing this issue has not been successful despite many days of effort.
On the Raspberry Pi (Compute Module 4) we can use the Raspivid application
to capture frame-accurate, synchronous video from two cameras @ 30fps by
simply adding the '''-3d sbs''' tag. We believe ffmpeg oils potentially
use similar techniques / code to synchronize the videos with a modified
V4L2 driver. We have a new post on the Raspberry Pi forums describing
this:
https://www.raspberrypi.org/forums/viewtopic.php?f=61&t=290197&p=1754295#p1754295
We are attaching a file showing the results of the code above. As you see,
FFMPEG successfully places two (circular) videos side-by-side. The ONLY
issue is the ~1second time lag.
--
Ticket URL: <https://trac.ffmpeg.org/ticket/8983>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker
More information about the FFmpeg-trac
mailing list