[Ffmpeg-devel] Sample Aspect Ratio & Pan&Scan

[Steve Lhomme]

M?ns Rullg?rd wrote:
> Steve Lhomme <slhomme at divxcorp.com> writes:
> 
>> Hi,
>>
>> So far in DrFFMPEG we used a modified version of mpeg12.c that doesn't
>> take the pan_scan size to set the sample_aspect_ratio. (see around
>> line 2142) But since FFMPEG doesn't seem to have problems to deal with
>> it I think we may do something wrong...
>>
>> Because of DivX certification constraints we only output square
>> pixels. So we have to translate the source aspect ratio to square
>> pixels.
> 
> That paragraph doesn't make sense.

It just means that in what we're doing, we handle the aspect ratio 
before doing the encoding, and we always code square pixels (PAR=1:1).

>> So far we assumed that the sample_aspect_ratio was a pixel aspect
>> ratio and it worked well for most format. But it seems that it's not
>> good enough in the general case.
> 
> sample_aspect_ratio is just that, the aspect ratio of each pixel.

So SAR = PAR ? So we were right so far... But then that means FFMPEG is 
not consistent/clean when dealing with pan&scan.

>> For example I have 2 VOB files. They both use pan & scan and have a
>> pixel aspect ratio of 16:9. But for the first one (chems1.vob) we get
>> a sample aspect ratio of 256:135 (1.896) and the other one has a
>> sample aspect ratio of 32/27 (1.18). For chems1.vob the original pixel
>> size is 720x576 and for the other one it's 720x480. While the source
>> dimensions and pixel aspect ratio are similar, the sample aspect ratio
>> is very different. That's because the pan & scan dimensions are very
>> different (for chems1.vob the width is bigger than the height).
> 
> The meanings of display_horizontal_size and display_vertical_size are
> not defined in the MPEG2 spec.  The fields are there merely as hints
> that the display system may use in whatever way it sees fit.
> 
> Looking at chems1.vob (assuming it is the same file I've gotten from
> you previously), it has a DAR of 16/9 (not SAR as you say above) with
> a coded picture size is 720x576.  The display_horizontal_size and
> display_vertical_size values are 540x576.  One possible interpretation
> of these values (which is suggested in the MPEG2 spec) is that the
> display device has a pixel size of 540x576 and a display aspect ratio
> of 16/9.  That would give a pixel aspect ratio of 5/3 or 1.67.
> Another reasonable interpretation is that the DAR of 16/9 refers to
> the full coded picture, which would give a SAR of 64/27 or 2.37.  It
> is not obvious from the video image what the intent is.

2.37 is wrong. It would give an upscale dimension of 576*2.37=1365 and 
the actual dimension to display is around 1024 pixels which is according 
to MPC (displaying the image correctly, unlike VLC) a DAR of 1.78 (16:9).

>> Our hack so far was not to take the pan & scan in account and it
>> worked well. So is there any reason we should use the pan&scan value ? 
>> From what I understand the pan&scan feature is made to crop the
>> original picture ? On demand ? Maybe it should be optional in FFMPEG ? 
> 
> FFmpeg doesn't process pan&scan information other than exporting it to
> the calling application.  This is as it should be.

Well, this pan&scan information is exported in AVFrame->pan_scan, but as 
it's supposed to be about cropping it should rather be in AVCodecContext 
or something like that. Plus the resulting dimensions of the pan_scan 
values are already applied sample_aspect_ratio. And if 
AVCodecContext->sample_aspect_ratio is supposed to be the PAR then the 
value is wrong.

>> It seems that encoding with FFMPEG works because both the (pixel)
>> aspect_ratio_info is stored and the sample_aspect_ratio. But the
>> aspect_ratio_info field is MpegEncContext specific... Should it be
>> more general ?
> 
> The aspect_ratio_info field in MpegEncContext is the raw encoding of
> aspect ratio used in the bitstream.  Its meaning depends on the codec,
> and exporting it would make no sense.

Actually it's the pan_scan info that should be more general. Why is it 
in AVFrame and not in AVCodecContext ? because an MPEG stream can have 
various values in the same stream ? If so the upper app is supposed to 
change the cropping value during decoding too ?

So to summarize what there is now in the code:

final DAR = sample_aspect_ratio * width / height
width and height being the pixels after the pan_scan is applied. But the 
app doesn't have any info on the pan_scan apart from when a frame is 
decoded.