[FFmpeg-devel] [PATCH 2/3] avcodec/x86: disable hevc 12b luma deblock

[Andreas Rheinhardt]

J. Dekker:
> Over/underflow in some cases.
> 
> Signed-off-by: J. Dekker <jdek at itanimul.li>
> ---
>  libavcodec/x86/hevcdsp_init.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/libavcodec/x86/hevcdsp_init.c b/libavcodec/x86/hevcdsp_init.c
> index 31e81eb11f..11cb1b3bfd 100644
> --- a/libavcodec/x86/hevcdsp_init.c
> +++ b/libavcodec/x86/hevcdsp_init.c
> @@ -1205,10 +1205,11 @@ void ff_hevc_dsp_init_x86(HEVCDSPContext *c, const int bit_depth)
>          if (EXTERNAL_SSE2(cpu_flags)) {
>              c->hevc_v_loop_filter_chroma = ff_hevc_v_loop_filter_chroma_12_sse2;
>              c->hevc_h_loop_filter_chroma = ff_hevc_h_loop_filter_chroma_12_sse2;
> -            if (ARCH_X86_64) {
> -                c->hevc_v_loop_filter_luma = ff_hevc_v_loop_filter_luma_12_sse2;
> -                c->hevc_h_loop_filter_luma = ff_hevc_h_loop_filter_luma_12_sse2;
> -            }
> +            // FIXME: 12-bit luma deblock over/underflows in some cases
> +            // if (ARCH_X86_64) {
> +            //     c->hevc_v_loop_filter_luma = ff_hevc_v_loop_filter_luma_12_sse2;
> +            //     c->hevc_h_loop_filter_luma = ff_hevc_h_loop_filter_luma_12_sse2;
> +            // }
>              SAO_BAND_INIT(12, sse2);
>              SAO_EDGE_INIT(12, sse2);
>  

If you disable them here, you should also ensure that they are not
assembled at all.

- Andreas