[FFmpeg-trac] #8505(avcodec:new): fast aac encoder produces invalid output for x86 32-bit builds

[FFmpeg]

#8505: fast aac encoder produces invalid output for x86 32-bit builds
-------------------------------------+-------------------------------------
             Reporter:               |                    Owner:
  LeadAssimilator                    |
                 Type:  defect       |                   Status:  new
             Priority:  important    |                Component:  avcodec
              Version:  unspecified  |               Resolution:
             Keywords:  regression   |               Blocked By:
  aac                                |
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------

Comment (by LeadAssimilator):

 There appears to be an issue in
 [http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavcodec/aacenc_quantization.h;h=fc5a46b8754ac861db6d886ec8059f77c1893aee;hb=fcc0424c933742c8fc852371e985d16b6eb4bfe9#l144
 quantize_and_encode_band_cost_template] where it attempts to encode an ESC
 sequence with too few bits.  A minimum coefficient value of 16 is required
 for an ESC sequence, but the computed coefficient is only 15.  Since the
 value 15 is too low, the required number of bits for the combined ESC
 sequence prefix bits and separator bit is also calculated too low at 0
 (minimum is 1), along with an invalid value of -1.  These erroneous values
 are then used in the call to put_bits which writes 0 bits to the output
 instead of the minimum required 1 bit.  Similarly the number of bits for
 the sequence word is also under calculated at 3 bits, versus the minimum 4
 bits. They again are used in the call to put_sbits with the too small
 coefficient 15, that writes 3 bits to the output instead of the minimum
 required 4 bits.  As a result, the entire ESC sequence is only coded as 3
 bits as opposed to the the shortest viable sequence of 5 bits and this
 corrupts the output bitstream.

 It seems that
 [http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavcodec/x86/aacencdsp.asm;h=97af571ec8bb76e3ba654acc0f0df7c845244ae7;hb=fcc0424c933742c8fc852371e985d16b6eb4bfe9#l57
 ff_aac_quantize_bands_sse2] may be partially to blame.  When used, it
 quantizes slightly differently than the non-sse2 version
 [http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavcodec/aacenc_utils.h;h=bef4c103f382a663a07678ce592ce25bc9a815cf;hb=fcc0424c933742c8fc852371e985d16b6eb4bfe9#l65
 quantize_bands], producing larger values.  This might not normally be an
 issue, but later calculations are done with the same inputs using a
 different function,
 [http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavcodec/aacenc_utils.h;h=bef4c103f382a663a07678ce592ce25bc9a815cf;hb=fcc0424c933742c8fc852371e985d16b6eb4bfe9#l59
 quant], which yields different results during the writing of the ESC
 sequence.  In this particular failure case, the input -56421.839844
 (-0x1.b8cbaep+15) quantized to 16 via sse2, 15 via the non-sse2 variant
 quantize_bands (during a working run) and 15 via quant.  This discrepancy
 explains why this issue affects only 32-bit x86 builds, however it can
 happen any time ff_aac_quantize_bands_sse2|quantize_bands produces values
 larger than quant for the same input and the resulting quant output is <
 16.

 I see the following possible solutions:
 1) Use the same exact quantization routines for all involved calculations
 (possible performance impacts)
 2) Clamp the lower bound of the escape sequence coefficient to 16

 It would be great if someone more familiar with the aac encoder or these
 kinds of issues weighs in and/or takes over.  If no one is interested, I
 can take a stab at producing a patch.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/8505#comment:18>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker