[FFmpeg-devel] dca.c: output scale is a bit off

[Alexander E. Patrakov]

Hello,

Recently I have reported a bug against libdca because it had the full scale of 
its output (except the LFE channel) off by a factor of sqrt(2) / 1.5 = 
0.942... I know this because:

1) I encoded a wav file of my own choice with "Surcode DTS DVD" software and 
checked the output of a libdca-based decoder, and 

2) under this assumption on scaling, when encoding just a DC level with my own 
yet-to-be-released encoder, the levels in channels after the forward QMF 
transform for non-LFE channels & lowpass of the LFE channel (i.e., just 
before the quantization) match very well (e.g., 0x16A0A144 in LFE vs 
0x16A09BD3 in other channels for a 50% DC level).

The LFE channel scaling was always OK, and in libdca r84, the scaling factor 
has been fixed for other channels.

The scaling factor in ffmpeg's dca.c was copied from the old and buggy version 
of libdca. Please apply the patch below in order to fix this ffmpeg bug.

Index: libavcodec/dca.c
===================================================================
--- libavcodec/dca.c	(revision 14863)
+++ libavcodec/dca.c	(working copy)
@@ -990,7 +990,7 @@
 /*        static float pcm_to_double[8] =
             {32768.0, 32768.0, 524288.0, 524288.0, 0, 8388608.0, 
8388608.0};*/
          qmf_32_subbands(s, k, subband_samples[k], &s->samples[256 * k],
-                            2.0 / 3 /*pcm_to_double[s->source_pcm_res] */ ,
+                            M_SQRT1_2 /*pcm_to_double[s->source_pcm_res] */ ,
                             0 /*s->bias */ );
     }
 
-- 
Alexander E. Patrakov