[Ffmpeg-devel] clever 8-bit MMX loop filter ABS test

[Richard Goedeken]

Hi guys,

I saw this piece of code in the list last thursday. While it is a rather
clever way to do the alpha/beta test, it won't produce results that are
identical to the 16-bit code.

 > Here's an example for the test ABS(P0-Q0)<Alpha, using 8b only:
 >
 >  input: mm7 = Alpha value, 8bits, replicated 8 times
 >
 >  movd      mm0, [Q0]		 ; four pixels 'Q0' in lower 32bits
 >  punpckldq mm0, [P0]           ; four pixels 'P0' in  higher 32bits
 >
 >  pshufw  mm1, mm0, 01001110b   ;    P0       | Q0     (Swap P0 and Q0)
 >  paddusb mm0, mm7              ;    Q0+Alpha | P0+Alpha
 >  psubusb mm0, mm1              ; Q0+Alpha-P0 | P0+Alpha-Q0
 >
 > At this point: mm0 contains zeros in the lower 32bits if P0>=Q0+Alpha,
 > and zeros in the higher 32bits if Q0>=P0+Alpha.

The statement directly above is not always true, because of the 8-bit 
saturation. Consider the case when P0 and Q0 are both 255 and Alpha > 0.
In this case, all of the bits of mm0 will be 0 even though
"ABS(P0-Q0)<Alpha" is true. This will lead to a false negative and the
decoder will fail to filter an edge that it should have filtered.

 > Note: you can repeat/pair the above 3 instructions for the other tests
 > (ABS(P1-P0)<Beta, etc...), and accumulate the results in mm0 using
 > a 'por' instruction...
 >
 > In the end, when one wants the final result:
 >
 >  pminub mm0, [One]	      ; mask is now made of '0' or '1'.
 >                             ; [One] is 1, replicated 8 times
 >  pshufw mm1, mm0, 01001110b ; Swap the hi/lo 32 bits
 >  pxor   mm0, [One]          ; flip the bits
 >  pand   mm1, mm0            ; => the higher 4 bytes of mm1 tell
 >                             ; whether the pixels should be filtered or
 >                             ; not.

Also the "pxor mm0, [One]" instruction should not be there. It
effectively neutralizes the absolute-value-ness of doing both the
Q0-P0 and P0-Q0 cases at once.

- Richard