[FFmpeg-devel] [PATCH] swscale sometimes treats PIX_FMT_RGB8 as paletted sometimes not

[Reimar Döffinger]

Hello,
currently
http://movies.apple.com/trailers/fox/ice_age_2/images5/ia2_tsr_land_r4c5a.mov
plays wrong in MPlayer because it uses libswscale to convert it to
non-paletted (ffplay does it somehow else).
The problem is that in swscale.c, line 2521 the palette is only
converted to YUV for PIX_FMT_PAL8 but not PIX_FMT_BGR8 and PIX_FMT_RGB8.
While that is correct if PIX_FMT_BGR8 is supposed to be a different,
non-paletted format that does not fit swscale_template.c, line 2541:
>     else if (srcFormat==PIX_FMT_RGB8 || srcFormat==PIX_FMT_BGR8 || srcFormat==PIX_FMT_PAL8 || srcFormat==PIX_FMT_BGR4_BYTE  || srcFormat==PIX_FMT_RGB4_BYTE)
>        {
>	        RENAME(palToY)(formatConvBuffer, src, srcW, pal);

I would propose attached patch as a quick hack that IMO at least doesn't
make things worse, but feel free to implement a proper solution ;-) (I
will care of the MPlayer side if you make swscale consistent and explain
how things are supposed to work - esp. when e.g. setting destination
format to PIX_FMT_PAL8, things like where it will take the palette from
(generate it, generate it only if the source has none, ...?)).

Greetings,
Reimar D?ffinger
-------------- next part --------------
Index: libswscale/swscale.c
===================================================================
--- libswscale/swscale.c	(revision 23416)
+++ libswscale/swscale.c	(working copy)
@@ -2518,7 +2518,7 @@
         if (srcSliceY == 0) c->sliceDir = 1; else c->sliceDir = -1;
     }
 
-    if (c->srcFormat == PIX_FMT_PAL8){
+    if (c->srcFormat == PIX_FMT_RGB8 || c->srcFormat == PIX_FMT_BGR8 || c->srcFormat == PIX_FMT_PAL8){
         for (i=0; i<256; i++){
             int p= ((uint32_t*)(src[1]))[i];
             int r= (p>>16)&0xFF;