Understanding motion vectors from ff_print_debug_info()
Hey All- A while back Paolo suggested that people looking to extract motion vectors in ffmpeg should look at the method ff_print_debug_info() in libavcodec/mpegvideo.c. This advice turned out to be very useful in my case. Thx Paolo. However, I have a couple of questions about the constants being used ff_print_debug_info() that would really help noobs like myself really understand what is going on. Please consider posting a reply if you are familiar with this part of ffmpeg. # It looks like you have to divide by 2 (or a higher power of 2) the values in the motion_val array to obtain the true motion vectors. Why is this so? For example ff_print_debug_info() processes the motion_val vectors in the following manner: motion_val[direction][xy][0]>>shift where shift = 1 + s->quarter_sample. The 'quarter_sample' variable makes sense. It's accounting for some sort of mode where the video is sub-sampled by 2 in the X and Y axis. But I don't understand why the '1' is added to 'shift' in effect dividing all motion vectors by 2. # For computing the 'mv_stride' variable avcodec.h suggests the following formula: mv_stride= (mb_width << mv_sample_log2) + 1 While I don't understand why that '1' is added mv_stride. What worries me is that the addition of that '1' is conditional and I don't understand when not to add that '1'. For example ff_print_debug_info() uses the following formula: (s->mb_width << mv_sample_log2) + (s->codec_id == CODEC_ID_H264 ? 0 : 1) # Finally, is there any way to populate the AVFrame.motion_val array with the motion vectors without having the motion vectors visualized on the decoded video frame? I'm setting AVCodecContext.debug_mv = 1 to obtain the motion vectors but as a side effect that flag also draws the motion vectors. It looks like ff_print_debug_info() always draws the motion vectors when the debug_mv flag is set. It seems wasteful to have to decode the video twice in order to obtain the motion vectors and the video frames without the visualization. Thank you for your time. -pro
On Sat, 28 Jul 2007, Pravin Bhat wrote:
# It looks like you have to divide by 2 (or a higher power of 2) the values in the motion_val array to obtain the true motion vectors. Why is this so?
For example ff_print_debug_info() processes the motion_val vectors in the following manner: motion_val[direction][xy][0]>>shift where shift = 1 + s->quarter_sample. The 'quarter_sample' variable makes sense. It's accounting for some sort of mode where the video is sub-sampled by 2 in the X and Y axis. But I don't understand why the '1' is added to 'shift' in effect dividing all motion vectors by 2.
Sub-sampling is where you delete some of the samples, leaving lower resolution. That is not happening here. Motion vectors are specified to fractional-sample precision, e.g a vector of (0.5, 0) means translate the video by half a pixel (interpolating between samples in the reference frame). quarter_sample=1 means the precision is 0.25, and the alternative is 0.5. The numbers in motion_val[] are fixed-point representations of those fractional vectors.
# For computing the 'mv_stride' variable avcodec.h suggests the following formula: mv_stride= (mb_width << mv_sample_log2) + 1 While I don't understand why that '1' is added mv_stride.
Some internal code is simplified if we put a ring of dummy values around the edge of the array, such that the left neighbor of the leftmost macroblock lands in the dummy values rather than landing on the right edge of the frame.
What worries me is that the addition of that '1' is conditional and I don't understand when not to add that '1'. For example ff_print_debug_info() uses the following formula: (s->mb_width << mv_sample_log2) + (s->codec_id == CODEC_ID_H264 ? 0 : 1)
otoh, ffh264 uses some optimizations that require the addresses of individual motion vectors to be aligned, and it doesn't benefit from the dummy values.
# Finally, is there any way to populate the AVFrame.motion_val array with the motion vectors without having the motion vectors visualized on the decoded video frame? I'm setting AVCodecContext.debug_mv = 1 to obtain the motion vectors but as a side effect that flag also draws the motion vectors. It looks like ff_print_debug_info() always draws the motion vectors when the debug_mv flag is set. It seems wasteful to have to decode the video twice in order to obtain the motion vectors and the video frames without the visualization.
AVFrame.motion_val is always populated, if the codec has motion vectors at all. The motion vectors have to be stored somewhere, so there's no penalty for exporting them. --Loren Merritt
Loren Merritt wrote:
On Sat, 28 Jul 2007, Pravin Bhat wrote:
# It looks like you have to divide by 2 (or a higher power of 2) the values in the motion_val array to obtain the true motion vectors. Why is this so?
For example ff_print_debug_info() processes the motion_val vectors in the following manner: motion_val[direction][xy][0]>>shift where shift = 1 + s->quarter_sample. The 'quarter_sample' variable makes sense. It's accounting for some sort of mode where the video is sub-sampled by 2 in the X and Y axis. But I don't understand why the '1' is added to 'shift' in effect dividing all motion vectors by 2.
Sub-sampling is where you delete some of the samples, leaving lower resolution. That is not happening here. Motion vectors are specified to fractional-sample precision, e.g a vector of (0.5, 0) means translate the video by half a pixel (interpolating between samples in the reference frame). quarter_sample=1 means the precision is 0.25, and the alternative is 0.5. The numbers in motion_val[] are fixed-point representations of those fractional vectors.
Great. That explains a lot! How does one obtain the precision of the motion vectors for a given codec. For example, AVFrame.motion_subsample_log2 is set (hopefully) based on the codec used to decode the frame. Is there a similar field that tells you the precision value used by the codec?
# For computing the 'mv_stride' variable avcodec.h suggests the following formula: mv_stride= (mb_width << mv_sample_log2) + 1 While I don't understand why that '1' is added mv_stride.
Some internal code is simplified if we put a ring of dummy values around the edge of the array, such that the left neighbor of the leftmost macroblock lands in the dummy values rather than landing on the right edge of the frame.
What worries me is that the addition of that '1' is conditional and I don't understand when not to add that '1'. For example ff_print_debug_info() uses the following formula: (s->mb_width << mv_sample_log2) + (s->codec_id == CODEC_ID_H264 ? 0 : 1)
otoh, ffh264 uses some optimizations that require the addresses of individual motion vectors to be aligned, and it doesn't benefit from the dummy values.
Is ffh264 the only codec that doesn't use padding? In other words, is the formula being used by ff_print_debug_info() general enough to handle most codecs?
# Finally, is there any way to populate the AVFrame.motion_val array with the motion vectors without having the motion vectors visualized on the decoded video frame? I'm setting AVCodecContext.debug_mv = 1 to obtain the motion vectors but as a side effect that flag also draws the motion vectors. It looks like ff_print_debug_info() always draws the motion vectors when the debug_mv flag is set. It seems wasteful to have to decode the video twice in order to obtain the motion vectors and the video frames without the visualization.
AVFrame.motion_val is always populated, if the codec has motion vectors at all. The motion vectors have to be stored somewhere, so there's no penalty for exporting them.
Great! Is there a field that gets set if the codec has no motion vectors? Also, I was hoping there was a way to tell which motion vectors are wrong or not used in the frame construction. For example, certain blocks in B-frames are not constructed using the previous frame. It would be nice to be able to determine that the motion vectors for such blocks are invalid. Thanks Loren. Your reply really helped. -pro
On Sun, 29 Jul 2007, Pravin Bhat wrote:
Loren Merritt wrote:
Sub-sampling is where you delete some of the samples, leaving lower resolution. That is not happening here. Motion vectors are specified to fractional-sample precision, e.g a vector of (0.5, 0) means translate the video by half a pixel (interpolating between samples in the reference frame). quarter_sample=1 means the precision is 0.25, and the alternative is 0.5. The numbers in motion_val[] are fixed-point representations of those fractional vectors.
Great. That explains a lot! How does one obtain the precision of the motion vectors for a given codec. For example, AVFrame.motion_subsample_log2 is set (hopefully) based on the codec used to decode the frame. Is there a similar field that tells you the precision value used by the codec?
As you said, s->quarter_sample. But that's not in the public API, and it can't even be derived from the codec name since some codecs make qpel optional.
What worries me is that the addition of that '1' is conditional and I don't understand when not to add that '1'. For example ff_print_debug_info() uses the following formula: (s->mb_width << mv_sample_log2) + (s->codec_id == CODEC_ID_H264 ? 0 : 1)
otoh, ffh264 uses some optimizations that require the addresses of individual motion vectors to be aligned, and it doesn't benefit from the dummy values.
Is ffh264 the only codec that doesn't use padding? In other words, is the formula being used by ff_print_debug_info() general enough to handle most codecs?
yes
# Finally, is there any way to populate the AVFrame.motion_val array with the motion vectors without having the motion vectors visualized on the decoded video frame? I'm setting AVCodecContext.debug_mv = 1 to obtain the motion vectors but as a side effect that flag also draws the motion vectors. It looks like ff_print_debug_info() always draws the motion vectors when the debug_mv flag is set. It seems wasteful to have to decode the video twice in order to obtain the motion vectors and the video frames without the visualization.
AVFrame.motion_val is always populated, if the codec has motion vectors at all. The motion vectors have to be stored somewhere, so there's no penalty for exporting them.
Great! Is there a field that gets set if the codec has no motion vectors? Also, I was hoping there was a way to tell which motion vectors are wrong or not used in the frame construction. For example, certain blocks in B-frames are not constructed using the previous frame. It would be nice to be able to determine that the motion vectors for such blocks are invalid.
AVFrame.motion_val should be NULL if it's not populated. AVFrame.mb_type contains the block types, each of which is a bitmask of some of the MB_TYPE_* flags. So it's a bit complicated to determine which mvs are valid, but it should be codec independent.
Also, I'm working on an algorithm that tries to deblock highly compressed videos. In the paper I would like to compare the algorithm against against existing state-of-art deblocking algorithms. I was wondering: - have you done any quantitative testing of the deblocking performance of x264 (say by comparing the psnr response of the compressed versus the deblocked video). - have you compared the deblocking performance of x264 to other algorithms?
Are you designing a postprocessor, or modifying the codec itself? Because h264's deblocking algorithm only works as an in-loop filter, not as a postprocessor. In particular, it assumes the reference frame was already deblocked, so that any regions of the frame that are only motion compensated without a residual don't need to be deblocked again. Unless they're a border between two motion blocks with different mvs, in which case it still deblocks with less strength than places with a residual. Designing a postprocessor is harder. Blocking artifacts that you dealt with in one frame might still be present in the next, and might no longer be aligned to block positions due to mc. Also note that in h264 there is no significant difference in psnr between any given frame before and after deblocking (although there is a difference in perceived artifacts). It only improves the psnr-per-bitrate of subsequent frames that are predicted from the deblocked frame. --Loren Merritt
Hello. I have a question about motion vectors. As you wrote here "http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2007-August/033535.html", "AVFrame.motion_val is always populated, if the codec has motion vectors at all. The motion vectors have to be stored somewhere, so there's no penalty for exporting them." But, in my program, AVFrame.motion_val is populated only when I'm setting AVCodecContext.debug_mv = 1. But, in this case, motion vectors are drew in frame. So, how to populate AVFrame.motion_val without vizualization? In advance, thank you for answer.
On Wed, Aug 12, 2009 at 11:23:39AM +0400, Oleg wrote:
Hello. I have a question about motion vectors. As you wrote here "http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2007-August/033535.html", "AVFrame.motion_val is always populated, if the codec has motion vectors at all. The motion vectors have to be stored somewhere, so there's no penalty for exporting them."
But, in my program, AVFrame.motion_val is populated only when I'm setting AVCodecContext.debug_mv = 1. But, in this case, motion vectors are drew in frame.
So, how to populate AVFrame.motion_val without vizualization?
try FF_DEBUG_MV and this is the wrong mailinglist for such questions as penalty, if my suggestion works you have to send us a patch improving the documentation so its clearer how to use motion_val [...] -- Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB Avoid a single point of failure, be that a person or equipment.
participants (4)
-
lorenm＠u.washington.edu -
michaelni＠gmx.at -
pro＠cs.washington.edu -
tagrus＠gmail.com