[FFmpeg-devel] Mixture of homographies video stabilization

[Matthew Lai]

I just realized: OpenCV has functions to do all the homography hard work
already, so I'll probably use that since OpenCV is already a dependency.

On Fri, Aug 3, 2018 at 1:29 PM Matthew Lai <matthewlai at google.com> wrote:

> Hi Michael,
>
> The operations needed are multiplications, SVD, inversion, scaling, and
> additions, as far as I can tell. Convolutions would probably also be
> useful. They are certainly possible to implement directly, but it's
> unlikely that any naive implementation will be within an order of magnitude
> in performance to an optimized BLAS library. That may or may not matter - I
> don't have a good idea of where the performance bottlenecks will be yet.
> The matrices are 4x4.
>
> Motion estimation is one way, but most state of the art implementations
> don't use it because it's too noisy (motion from low contrast regions).
> Most use something like a corner detector to find sharp feature points, and
> track them (this can potentially be implemented using motion estimation,
> but only around those points).
>
> Matthew
>
> On Fri, Aug 3, 2018 at 2:05 AM Michael Niedermayer <michael at niedermayer.cc>
> wrote:
>
>> On Thu, Aug 02, 2018 at 05:24:08PM +0100, Matthew Lai wrote:
>> > Ah ok thanks! I'm surprised no one has need a linear algebra library. I
>> > guess there's OpenCV and people use it to do the heavy lifting?
>> >
>> > Will look into the API more.
>>
>> alot ot linear algebra we needed has been implemented directly.
>> Some of it is shared and available from libavutil, like lls.*
>>
>> Can you elaboarte what exact linear algebra operations are needed?
>> also what amounts of data (matrix types/sizes) and speed requirements
>> this has
>>
>> Also IIUC (please correct me if iam wrong) this uses motion estimation
>> at one step. IIRC ronald and others) want motion estimation to be
>> factored and shared and not duplicated. The current motion estimation is
>> in libavcodec/motion_est*.
>>
>> Thanks
>>
>> >
>> > Thanks
>> > matthew
>> >
>> > On Thu, Aug 2, 2018 at 3:31 PM Paul B Mahol <onemda at gmail.com> wrote:
>> >
>> > > On 8/2/18, Matthew Lai <matthewlai-at-google.com at ffmpeg.org> wrote:
>> > > > Hello!
>> > > >
>> > > > I want to write a more advanced video stabilizer for libavfilter
>> (*),
>> > > > implementing the algorithm described here -
>> > > >
>> > >
>> https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37744.pdf
>> > > > The focus of the paper is rolling shutter removal, but it builds on
>> top
>> > > of
>> > > > another algorithm that does full frame stabilization, and the new
>> > > algorithm
>> > > > does that as well.
>> > > >
>> > > > This is the algorithm used in YouTube's stabilizing filter, and is
>> state
>> > > of
>> > > > the art. Adobe calls it Warp Stabilizer (it's the same thing as far
>> as I
>> > > > can tell from public information anyways).
>> > > >
>> > > > 3 questions:
>> > > > 1. Is there a linear algebra library already in use? I didn't see
>> > > anything
>> > > > in configure, but would be surprised if none of the existing
>> filters work
>> > > > with matrices?
>> > >
>> > > There is no such library here used. There are indeed video/audio
>> > > filters that work with matrices.
>> > >
>> > > > 2. Is there anything to watch out for re. a high frame delay (say a
>> few
>> > > > hundred frames)? Looking at the API, I don't see a callback to
>> flush out
>> > > > remaining frames when input frames are finished? Is doing it in two
>> > > passes
>> > > > the only option?
>> > >
>> > > It is handled internally, there are two internal APIs, activate one
>> and
>> > > legacy.
>> > > With legacy you can flush frames when you receive last frame from
>> input.
>> > > With newer, activate API, its similar.
>> > >
>> > > > 3. doc/writing_filters.txt says only slice threading is available.
>> That's
>> > > > not really possible with this filter, but frame threading is. Can I
>> just
>> > > > buffer frames internally (which I need to do anyways to smooth out
>> > > motion),
>> > > > and do my own threading?
>> > >
>> > > You could do it.
>> > >
>> > > >
>> > > > * vid.stab is good for what it does, but it only does rotation and
>> > > > translation, and doesn't handle zoom, perspective distortion, or
>> rolling
>> > > > shutter. This means it's limited when it comes to things like scuba
>> > > diving
>> > > > videos, where the camera is filming the seabed at a small distance
>> and at
>> > > > an angle.
>> > > >
>> > > > Thanks!
>> > > > Matthew
>> > > > _______________________________________________
>> > > > ffmpeg-devel mailing list
>> > > > ffmpeg-devel at ffmpeg.org
>> > > > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> > > >
>> > > _______________________________________________
>> > > ffmpeg-devel mailing list
>> > > ffmpeg-devel at ffmpeg.org
>> > > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> > >
>> > _______________________________________________
>> > ffmpeg-devel mailing list
>> > ffmpeg-devel at ffmpeg.org
>> > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> --
>> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>>
>> It is dangerous to be right in matters on which the established
>> authorities
>> are wrong. -- Voltaire
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>