[FFmpeg-trac] #9222(avfilter:new): A possible divide by zero bug

[FFmpeg]

#9222: A possible divide by zero bug
-----------------------------------+--------------------------------------
             Reporter:  YiyuanGUO  |                     Type:  defect
               Status:  new        |                 Priority:  normal
            Component:  avfilter   |                  Version:  git-master
             Keywords:             |               Blocked By:
             Blocking:             |  Reproduced by developer:  0
Analyzed by developer:  0          |
-----------------------------------+--------------------------------------
 Summary of the bug:
 In libavfilter/vf_palettegen.c, the function ''get_avg_color'' has a
 potential divide by zero problem:

 {{{
     int i;
     const int n = box->len;
     uint64_t r = 0, g = 0, b = 0, div = 0;

     for (i = 0; i < n; i++) {
         const struct color_ref *ref = refs[box->start + i];
         r += (ref->color >> 16 & 0xff) * ref->count;
         g += (ref->color >>  8 & 0xff) * ref->count;
         b += (ref->color       & 0xff) * ref->count;
         div += ref->count;
     }

     r = r / div;
     g = g / div;
     b = b / div;

 }}}

 If ''box->len'' equals to 0, then ''div'' remains 0 after the loop and
 triggers divide by zero problems.

 This may happen through the following call sequences in vf_palettegen.c
 (if ''ctx->priv->nb_refs'' equals to 0):

 [https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_palettegen.c#L496/
 filter_frame] ->
 [https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_palettegen.c#L322/
 get_palette_frame] ->
 [https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_palettegen.c#L201/
 get_avg_color]

 Notice that the ''request_frame'' function has explicitly checked that the
 ''nb_refs'' field is nonzero before calling the function
 ''get_palette_frame'' to avoid such problems (link to the
 [https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_palettegen.c#L517/
 code]):

 {{{
     if (r == AVERROR_EOF && !s->palette_pushed && s->nb_refs &&
 s->stats_mode != STATS_MODE_SINGLE_FRAMES) {
         r = ff_filter_frame(outlink, get_palette_frame(ctx));
     ......
 }}}


 Therefore, I think we need similar checks in the above mentioned call
 sequence.

 This is a potential bug found by static analysis, and currently I don't
 have a POC. Please take a look and check if a fix is needed, thanks!
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/9222>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker