[FFmpeg-devel] Mixture of homographies video stabilization

[Michael Niedermayer]

On Thu, Aug 02, 2018 at 05:24:08PM +0100, Matthew Lai wrote:
> Ah ok thanks! I'm surprised no one has need a linear algebra library. I
> guess there's OpenCV and people use it to do the heavy lifting?
> 
> Will look into the API more.

alot ot linear algebra we needed has been implemented directly.
Some of it is shared and available from libavutil, like lls.*

Can you elaboarte what exact linear algebra operations are needed?
also what amounts of data (matrix types/sizes) and speed requirements this has

Also IIUC (please correct me if iam wrong) this uses motion estimation
at one step. IIRC ronald and others) want motion estimation to be
factored and shared and not duplicated. The current motion estimation is
in libavcodec/motion_est*. 

Thanks

> 
> Thanks
> matthew
> 
> On Thu, Aug 2, 2018 at 3:31 PM Paul B Mahol <onemda at gmail.com> wrote:
> 
> > On 8/2/18, Matthew Lai <matthewlai-at-google.com at ffmpeg.org> wrote:
> > > Hello!
> > >
> > > I want to write a more advanced video stabilizer for libavfilter (*),
> > > implementing the algorithm described here -
> > >
> > https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37744.pdf
> > > The focus of the paper is rolling shutter removal, but it builds on top
> > of
> > > another algorithm that does full frame stabilization, and the new
> > algorithm
> > > does that as well.
> > >
> > > This is the algorithm used in YouTube's stabilizing filter, and is state
> > of
> > > the art. Adobe calls it Warp Stabilizer (it's the same thing as far as I
> > > can tell from public information anyways).
> > >
> > > 3 questions:
> > > 1. Is there a linear algebra library already in use? I didn't see
> > anything
> > > in configure, but would be surprised if none of the existing filters work
> > > with matrices?
> >
> > There is no such library here used. There are indeed video/audio
> > filters that work with matrices.
> >
> > > 2. Is there anything to watch out for re. a high frame delay (say a few
> > > hundred frames)? Looking at the API, I don't see a callback to flush out
> > > remaining frames when input frames are finished? Is doing it in two
> > passes
> > > the only option?
> >
> > It is handled internally, there are two internal APIs, activate one and
> > legacy.
> > With legacy you can flush frames when you receive last frame from input.
> > With newer, activate API, its similar.
> >
> > > 3. doc/writing_filters.txt says only slice threading is available. That's
> > > not really possible with this filter, but frame threading is. Can I just
> > > buffer frames internally (which I need to do anyways to smooth out
> > motion),
> > > and do my own threading?
> >
> > You could do it.
> >
> > >
> > > * vid.stab is good for what it does, but it only does rotation and
> > > translation, and doesn't handle zoom, perspective distortion, or rolling
> > > shutter. This means it's limited when it comes to things like scuba
> > diving
> > > videos, where the camera is filming the seabed at a small distance and at
> > > an angle.
> > >
> > > Thanks!
> > > Matthew
> > > _______________________________________________
> > > ffmpeg-devel mailing list
> > > ffmpeg-devel at ffmpeg.org
> > > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

It is dangerous to be right in matters on which the established authorities
are wrong. -- Voltaire
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180803/8ff2ba3e/attachment.sig>