[FFmpeg-devel] [PATCH 2/2] avcodec/pixlet: Fixes: undefined shift in av_mod_uintp2()

[Michael Niedermayer]

On Sat, Aug 19, 2017 at 04:00:18PM +0200, Paul B Mahol wrote:
> On 8/19/17, Michael Niedermayer <michael at niedermayer.cc> wrote:
> > On Fri, Aug 18, 2017 at 06:21:56PM +0200, Paul B Mahol wrote:
> >> On 8/18/17, Michael Niedermayer <michael at niedermayer.cc> wrote:
> >> > Fixes: runtime error: shift exponent 4294967289 is too large for 32-bit
> >> > type
> >> > 'int'
> >> > Fixes: 3030/clusterfuzz-testcase-minimized-4649809254285312
> >> >
> >> > Found-by: continuous fuzzing process
> >> > https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> >> > Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> >> > ---
> >> >  libavcodec/pixlet.c | 2 +-
> >> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >> >
> >> > diff --git a/libavcodec/pixlet.c b/libavcodec/pixlet.c
> >> > index 088226bdda..a9cfe085c9 100644
> >> > --- a/libavcodec/pixlet.c
> >> > +++ b/libavcodec/pixlet.c
> >> > @@ -262,7 +262,7 @@ static int read_high_coeffs(AVCodecContext *avctx,
> >> > uint8_t *src, int16_t *dst, i
> >> >
> >> >          flag = 0;
> >> >
> >> > -        if (state * 4ULL > 0xFF || i >= size)
> >> > +        if ((uint64_t)state > 0xFF / 4 || i >= size)
> >>
> >> This is not exact same behaviour.
> >
> > no, it differs for a small number of cases that have one of the 2
> > most significant bits (of the 64 bits) set in state.
> > I think this was a bug i introduced in a previous commit.
> >
> > If above change is not the correct thing to do, please elaborate.
> > state is passed into ff_clz() later which has no deterministic
> > behavior when these MSBs are set as ff_clz() uses unsigned and state
> > is int64_t and unsigned can be 32 or 64bit.
> 
> Only 32 bits matters for ff_clz() anyway.

your reply is a bit terse.
Does that mean you have no objections to the patch ?

about ff_clz(), with 64bit unsigned a ff_clz(1) should return 63, the
code looks like it expects 31

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If you fake or manipulate statistics in a paper in physics you will never
get a job again.
If you fake or manipulate statistics in a paper in medicin you will get
a job for life at the pharma industry.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20170819/79a2cfea/attachment.sig>