[FFmpeg-cvslog] h264: Fix maximum reference count check for non-b frames

[Jeff Downs]

ffmpeg | branch: master | Jeff Downs <heydowns at somuchpressure.net> | Thu Jun 28 12:47:10 2012 -0400| [8d9fd58113fa2953269252575bf5f1edce41e550] | committer: Michael Niedermayer

h264: Fix maximum reference count check for non-b frames

Below fixes the maximum reference count check for second reference list in
non-B frames.  There is nothing to prohibit full (field sized) reference
list in this case as far as I can tell, and this fixes several syntax-test
files here (this is a regression caused when this check was made more
stringent by
http://git.videolan.org/?p=ffmpeg.git;a=commit;h=dc9ce40069bde3d28f8d0b3e5bd733ae255fecb5)

Probably a silly corner case seldom seen irl, but thought I'd pass along
in case there was interest in correcting the check.

---------------

h264: Fix maximum reference count check for non-b frames; full range is
technically ok

Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=8d9fd58113fa2953269252575bf5f1edce41e550
---

 libavcodec/h264.c |   10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/libavcodec/h264.c b/libavcodec/h264.c
index cb05508..0c84018 100644
--- a/libavcodec/h264.c
+++ b/libavcodec/h264.c
@@ -3383,7 +3383,8 @@ static int decode_slice_header(H264Context *h, H264Context *h0)
     h->ref_count[1] = h->pps.ref_count[1];
 
     if (h->slice_type_nos != AV_PICTURE_TYPE_I) {
-        unsigned max = s->picture_structure == PICT_FRAME ? 15 : 31;
+        unsigned max[2];
+        max[0] = max[1] = s->picture_structure == PICT_FRAME ? 15 : 31;
 
         if (h->slice_type_nos == AV_PICTURE_TYPE_B)
             h->direct_spatial_mv_pred = get_bits1(&s->gb);
@@ -3393,10 +3394,13 @@ static int decode_slice_header(H264Context *h, H264Context *h0)
             h->ref_count[0] = get_ue_golomb(&s->gb) + 1;
             if (h->slice_type_nos == AV_PICTURE_TYPE_B)
                 h->ref_count[1] = get_ue_golomb(&s->gb) + 1;
+            else
+                // full range is spec-ok in this case, even for frames
+                max[1] = 31;
         }
 
-        if (h->ref_count[0]-1 > max || h->ref_count[1]-1 > max){
-            av_log(h->s.avctx, AV_LOG_ERROR, "reference overflow\n");
+        if (h->ref_count[0]-1 > max[0] || h->ref_count[1]-1 > max[1]){
+            av_log(h->s.avctx, AV_LOG_ERROR, "reference overflow %u > %u or %u > %u\n", h->ref_count[0]-1, max[0], h->ref_count[1]-1, max[1]);
             h->ref_count[0] = h->ref_count[1] = 1;
             return AVERROR_INVALIDDATA;
         }