[FFmpeg-trac] #5223(avutil:new): avcodec_find_best_pix_fmt_of_list depends on the order?

[FFmpeg]

#5223: avcodec_find_best_pix_fmt_of_list depends on the order?
--------------------------------+----------------------------------
             Reporter:  MattE   |                    Owner:
                 Type:  defect  |                   Status:  new
             Priority:  normal  |                Component:  avutil
              Version:          |               Resolution:
             Keywords:          |               Blocked By:
             Blocking:          |  Reproduced by developer:  0
Analyzed by developer:  0       |
--------------------------------+----------------------------------

Comment (by devernay):

 Confirmed.
 The "considered" loss mask is not the same for all pixel formats (the loss
 of first format is used a a mask for the second, etc) so that if the first
 pixel format loses resolution, resolution loss is not considered for other
 formats.

 A workaround is to always pass NULL as the last parameter. This really
 returns the best pixel format.

 A real fix is the following code, which uses the same "considered loss"
 mask for all formats in the list:

 {{{
 enum AVPixelFormat avcodec_find_best_pix_fmt_of_list(const enum
 AVPixelFormat *pix_fmt_list,
                                             enum AVPixelFormat
 src_pix_fmt,
                                             int has_alpha, int *loss_ptr){
     int i;

     enum AVPixelFormat best = AV_PIX_FMT_NONE;

     for(i=0; pix_fmt_list[i] != AV_PIX_FMT_NONE; i++) {
         int loss = loss_ptr ? *loss_ptr : 0;
         best = avcodec_find_best_pix_fmt_of_2(best, pix_fmt_list[i],
 src_pix_fmt, has_alpha, &loss);
     }
     if (loss_ptr) {
         *loss_ptr = av_get_pix_fmt_loss(best,  src_pix_fmt, has_alpha);
     }
     return best;
 }
 }}}

--
Ticket URL: <https://trac.ffmpeg.org/ticket/5223#comment:1>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker