[FFmpeg-trac] #3056(undetermined:new): Non-ASCII chars in -passlogfile on Windows

FFmpeg trac at avcodec.org
Fri Oct 18 00:08:33 CEST 2013


#3056: Non-ASCII chars in -passlogfile on Windows
-------------------------------------+-------------------------------------
             Reporter:  lelegard     |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:               |                  Version:  git-
  undetermined                       |  master
             Keywords:               |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 This report describes a problem with ffmpeg on Windows with the
 -passlogfile option using non-ASCII characters in the file path.

 The problem was originally presented in the FFmpeg-user mailing list as
 seen at [http://ffmpeg-users.933282.n4.nabble.com/Weird-characters-in-
 created-passlogfile-on-Windows-td4661845.html]

 I have not tested it yet on Linux but a reply in the mailing list suggests
 that the problem does not exist on FreeBSD. Given the nature of the
 problem, it is probably Windows-specific.

 Consider a file path containing non-ASCII characters like è (e grave) in a
 file or directory name. There are (at least) three different use cases for
 a file name: input file, output file, pass log file.

 In the first two cases, the non-ASCII characters in file names are
 processed correctly. But in the case of the pass log file, the file name
 is transformed into some weird sequence (typical untranslated UTF-8
 sequence). The strange thing is that ffmpeg creates two files, the pass
 log file and the output file, but processes the characters in the
 corresponding file names quite differently. The way the two files are
 created must be completely different.

 See the following example. The host system is Windows 7 64-bit with some
 Unix-like command line tools.

 {{{
 > ls -l x*
 -rw-rw-rw-   1 user     group     9400000 Oct 16 17:08 xè.ts

 > ffmpeg -i xè.ts -pass 1 -passlogfile xè xè.mpg
 ffmpeg version N-57176-g64b3aaf Copyright (c) 2000-2013 the FFmpeg
 developers
   built on Oct 15 2013 21:47:01 with gcc 4.8.1 (GCC)
   configuration: --enable-gpl --enable-version3 --disable-w32threads
 --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r
 --enable-gnutls --enable-iconv --enable-libass --enable-libbluray
 --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc
 --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb
 --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-
 librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex
 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-
 aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx
 --enable-libwavpack --enable-libx264 --enable-libxavs --enable-libxvid
 --enable-zlib
   libavutil      52. 46.101 / 52. 46.101
   libavcodec     55. 37.100 / 55. 37.100
   libavformat    55. 19.102 / 55. 19.102
   libavdevice    55.  4.100 / 55.  4.100
   libavfilter     3. 88.101 /  3. 88.101
   libswscale      2.  5.101 /  2.  5.101
   libswresample   0. 17.103 /  0. 17.103
   libpostproc    52.  3.100 / 52.  3.100
 [h264 @ 000000000249e820] non-existing PPS referenced
 [h264 @ 000000000249e820] non-existing PPS 0 referenced
 [h264 @ 000000000249e820] decode_slice_header error
 [h264 @ 000000000249e820] no frame!
 [h264 @ 000000000249e820] non-existing PPS referenced
 [h264 @ 000000000249e820] non-existing PPS 0 referenced
 [h264 @ 000000000249e820] decode_slice_header error
 [h264 @ 000000000249e820] no frame!
 [h264 @ 000000000249e820] non-existing PPS referenced
 [h264 @ 000000000249e820] non-existing PPS 0 referenced
 [h264 @ 000000000249e820] decode_slice_header error
 [h264 @ 000000000249e820] no frame!
 [h264 @ 000000000249e820] non-existing PPS referenced
 [h264 @ 000000000249e820] non-existing PPS 0 referenced
 [h264 @ 000000000249e820] decode_slice_header error
 [h264 @ 000000000249e820] no frame!
 [h264 @ 000000000249e820] mmco: unref short failure
     Last message repeated 2 times
 [mpegts @ 00000000024aebe0] PES packet size mismatch
     Last message repeated 2 times
 Input #0, mpegts, from 'xè.ts':
   Duration: 00:00:37.04, start: 42851.202589, bitrate: 2030 kb/s
   Program 4006
     Metadata:
       service_name    : ARTE
       service_provider: Harmonic
     Stream #0:0[0x424]: Video: h264 (Main) ([27][0][0][0] / 0x001B),
 yuv420p(tv, bt470bg), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25
  tbr, 90k tbn, 50 tbc
     Stream #0:1[0x425](fra): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 stereo, s16p, 128 kb/s
     Stream #0:2[0x426](eng): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 stereo, s16p, 128 kb/s
     Stream #0:3[0x427](deu): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 mono, s16p, 64 kb/s
     Stream #0:4[0x42b](qad): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 mono, s16p, 64 kb/s (visual impaired)
     Stream #0:5[0x42c](fra): Subtitle: dvb_teletext ([6][0][0][0] /
 0x0006)
 [mpeg @ 00000000048e9980] VBV buffer size not set, muxing may fail
 Output #0, mpeg, to 'xè.mpg':
   Metadata:
     encoder         : Lavf55.19.102
     Stream #0:0: Video: mpeg1video, yuv420p, 720x576 [SAR 16:11 DAR
 20:11], q=2-31, pass 1, 200 kb/s, 90k tbn, 25 tbc
     Stream #0:1(fra): Audio: mp2, 48000 Hz, stereo, s16, 128 kb/s
 Stream mapping:
   Stream #0:0 -> #0:0 (h264 -> mpeg1video)
   Stream #0:1 -> #0:1 (mp2 -> mp2)
 Press [q] to stop, [?] for help
 [h264 @ 0000000004bb2280] reference picture missing during reorder
     Last message repeated 1 times
 [h264 @ 0000000004bb2280] Missing reference picture, default is 0
 [h264 @ 0000000004bb2280] decode_slice_header error
 [h264 @ 00000000048cf8a0] mmco: unref short failure
 [h264 @ 0000000004bb2280] mmco: unref short failure
 [mpegts @ 00000000024aebe0] PES packet size mismatch0:34.60 bitrate=
 451.3kbits/s dup=18 drop=0
 [h264 @ 00000000048d01c0] error while decoding MB 22 28, bytestream
 (-5).8kbits/s dup=18 drop=0
 [mp2 @ 000000000249ec80] incomplete frame
 [h264 @ 00000000048d01c0] concealing 387 DC, 387 AC, 387 MV errors in B
 frame
 Error while decoding stream #0:1: Invalid data found when processing input
 frame=  923 fps=186 q=31.0 Lsize=    2046kB time=00:00:36.88 bitrate=
 454.5kbits/s dup=20 drop=0
 video:1455kB audio:571kB subtitle:0 global headers:0kB muxing overhead
 0.990720%

 > ls -l x*
 -rw-rw-rw-   1 user     group      128580 Oct 16 17:15 xA"-0.log
 -rw-rw-rw-   1 user     group     2095104 Oct 16 17:15 xè.mpg
 -rw-rw-rw-   1 user     group     9400000 Oct 16 17:08 xè.ts
 }}}

 The command contains "-passlogfile xè xè.mpg" but the "xè" in the two
 created files resulted in two different encodings.

 Of course, Windows is subject to the old "character code page" issues. But
 this cannot be the sole explanation. FFmpeg seems to correctly interpret
 the command line arguments since it successfully opens the input file and
 creates the output file with the right encoding. There is no rational
 reason that the arguments "-passlogfile xè" would not be read the same
 way. However, when it comes to creating the log file, the name "xè" is
 specifically transformed into "xA"". When creating the output file
 "xè.mpg", such name transformation does not occur.

 The problem is that the specified file name on the command line does NOT
 correspond to the actual file name in the file system. One may say that we
 don't care since this is a temporary log file.

 But there is a real functional problem when the non-ASCII character is in
 the name of a directory. In that case, ffmpeg transforms the directory
 name and the resulting path is invalid since the transformed path is a
 non-existent directory. And ffmpeg fails...

 See this example:

 {{{
 > mkdir xèdir

 > ls -ld x*
 -rw-rw-rw-   1 user     group     9400000 Oct 16 17:08 xè.ts
 drwxrwxrwx   1 user     group           0 Oct 16 17:15 xèdir

 > ffmpeg -i xè.ts -pass 1 -passlogfile xèdir\log xè.mpg
 ffmpeg version N-57176-g64b3aaf Copyright (c) 2000-2013 the FFmpeg
 developers
   built on Oct 15 2013 21:47:01 with gcc 4.8.1 (GCC)
   configuration: --enable-gpl --enable-version3 --disable-w32threads
 --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r
 --enable-gnutls --enable-iconv --enable-libass --enable-libbluray
 --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc
 --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb
 --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-
 librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex
 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-
 aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx
 --enable-libwavpack --enable-libx264 --enable-libxavs --enable-libxvid
 --enable-zlib
   libavutil      52. 46.101 / 52. 46.101
   libavcodec     55. 37.100 / 55. 37.100
   libavformat    55. 19.102 / 55. 19.102
   libavdevice    55.  4.100 / 55.  4.100
   libavfilter     3. 88.101 /  3. 88.101
   libswscale      2.  5.101 /  2.  5.101
   libswresample   0. 17.103 /  0. 17.103
   libpostproc    52.  3.100 / 52.  3.100
 [h264 @ 000000000031aa40] non-existing PPS referenced
 [h264 @ 000000000031aa40] non-existing PPS 0 referenced
 [h264 @ 000000000031aa40] decode_slice_header error
 [h264 @ 000000000031aa40] no frame!
 [h264 @ 000000000031aa40] non-existing PPS referenced
 [h264 @ 000000000031aa40] non-existing PPS 0 referenced
 [h264 @ 000000000031aa40] decode_slice_header error
 [h264 @ 000000000031aa40] no frame!
 [h264 @ 000000000031aa40] non-existing PPS referenced
 [h264 @ 000000000031aa40] non-existing PPS 0 referenced
 [h264 @ 000000000031aa40] decode_slice_header error
 [h264 @ 000000000031aa40] no frame!
 [h264 @ 000000000031aa40] non-existing PPS referenced
 [h264 @ 000000000031aa40] non-existing PPS 0 referenced
 [h264 @ 000000000031aa40] decode_slice_header error
 [h264 @ 000000000031aa40] no frame!
 [h264 @ 000000000031aa40] mmco: unref short failure
     Last message repeated 2 times
 [mpegts @ 000000000031a4a0] PES packet size mismatch
     Last message repeated 2 times
 Input #0, mpegts, from 'xè.ts':
   Duration: 00:00:37.04, start: 42851.202589, bitrate: 2030 kb/s
   Program 4006
     Metadata:
       service_name    : ARTE
       service_provider: Harmonic
     Stream #0:0[0x424]: Video: h264 (Main) ([27][0][0][0] / 0x001B),
 yuv420p(tv, bt470bg), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25
  tbr, 90k tbn, 50 tbc
     Stream #0:1[0x425](fra): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 stereo, s16p, 128 kb/s
     Stream #0:2[0x426](eng): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 stereo, s16p, 128 kb/s
     Stream #0:3[0x427](deu): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 mono, s16p, 64 kb/s
     Stream #0:4[0x42b](qad): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 mono, s16p, 64 kb/s (visual impaired)
     Stream #0:5[0x42c](fra): Subtitle: dvb_teletext ([6][0][0][0] /
 0x0006)
 Cannot write log file 'xèdir\log-0.log' for pass-1 encoding: No such file
 or directory

 >
 }}}

 In this case, ffmpeg fails since the log file cannot be created.

 Obvious workarounds exist, including using the old DOS 8.3 "short path
 name" for the directory parts which contain non-ASCII characters.

 Example:

 {{{
 > dir/x x*

 2013-10-16  17:08         9 400 000 X2691~1.TS   xè.ts
 2013-10-16  17:15    <DIR>          XDIR~1       xèdir

 > ffmpeg -i xè.ts -pass 1 -passlogfile xdir~1\log xè.mpg
 ffmpeg version N-57176-g64b3aaf Copyright (c) 2000-2013 the FFmpeg
 developers
   built on Oct 15 2013 21:47:01 with gcc 4.8.1 (GCC)
   configuration: --enable-gpl --enable-version3 --disable-w32threads
 --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r
 --enable-gnutls --enable-iconv --enable-libass --enable-libbluray
 --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc
 --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb
 --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-
 librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex
 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-
 aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx
 --enable-libwavpack --enable-libx264 --enable-libxavs --enable-libxvid
 --enable-zlib
   libavutil      52. 46.101 / 52. 46.101
   libavcodec     55. 37.100 / 55. 37.100
   libavformat    55. 19.102 / 55. 19.102
   libavdevice    55.  4.100 / 55.  4.100
   libavfilter     3. 88.101 /  3. 88.101
   libswscale      2.  5.101 /  2.  5.101
   libswresample   0. 17.103 /  0. 17.103
   libpostproc    52.  3.100 / 52.  3.100
 [h264 @ 00000000003aaa40] non-existing PPS referenced
 [h264 @ 00000000003aaa40] non-existing PPS 0 referenced
 [h264 @ 00000000003aaa40] decode_slice_header error
 [h264 @ 00000000003aaa40] no frame!
 [h264 @ 00000000003aaa40] non-existing PPS referenced
 [h264 @ 00000000003aaa40] non-existing PPS 0 referenced
 [h264 @ 00000000003aaa40] decode_slice_header error
 [h264 @ 00000000003aaa40] no frame!
 [h264 @ 00000000003aaa40] non-existing PPS referenced
 [h264 @ 00000000003aaa40] non-existing PPS 0 referenced
 [h264 @ 00000000003aaa40] decode_slice_header error
 [h264 @ 00000000003aaa40] no frame!
 [h264 @ 00000000003aaa40] non-existing PPS referenced
 [h264 @ 00000000003aaa40] non-existing PPS 0 referenced
 [h264 @ 00000000003aaa40] decode_slice_header error
 [h264 @ 00000000003aaa40] no frame!
 [h264 @ 00000000003aaa40] mmco: unref short failure
     Last message repeated 2 times
 [mpegts @ 00000000003aa4a0] PES packet size mismatch
     Last message repeated 2 times
 Input #0, mpegts, from 'xè.ts':
   Duration: 00:00:37.04, start: 42851.202589, bitrate: 2030 kb/s
   Program 4006
     Metadata:
       service_name    : ARTE
       service_provider: Harmonic
     Stream #0:0[0x424]: Video: h264 (Main) ([27][0][0][0] / 0x001B),
 yuv420p(tv, bt470bg), 720x576 [SAR 16:11 DAR 20:11], 25 fps, 25
  tbr, 90k tbn, 50 tbc
     Stream #0:1[0x425](fra): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 stereo, s16p, 128 kb/s
     Stream #0:2[0x426](eng): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 stereo, s16p, 128 kb/s
     Stream #0:3[0x427](deu): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 mono, s16p, 64 kb/s
     Stream #0:4[0x42b](qad): Audio: mp2 ([4][0][0][0] / 0x0004), 48000 Hz,
 mono, s16p, 64 kb/s (visual impaired)
     Stream #0:5[0x42c](fra): Subtitle: dvb_teletext ([6][0][0][0] /
 0x0006)
 [mpeg @ 0000000004b662c0] VBV buffer size not set, muxing may fail
 Output #0, mpeg, to 'xè.mpg':
   Metadata:
     encoder         : Lavf55.19.102
     Stream #0:0: Video: mpeg1video, yuv420p, 720x576 [SAR 16:11 DAR
 20:11], q=2-31, pass 1, 200 kb/s, 90k tbn, 25 tbc
     Stream #0:1(fra): Audio: mp2, 48000 Hz, stereo, s16, 128 kb/s
 Stream mapping:
   Stream #0:0 -> #0:0 (h264 -> mpeg1video)
   Stream #0:1 -> #0:1 (mp2 -> mp2)
 Press [q] to stop, [?] for help
 [h264 @ 0000000004a074c0] reference picture missing during reorder
     Last message repeated 1 times
 [h264 @ 0000000004a074c0] Missing reference picture, default is 0
 [h264 @ 0000000004a074c0] decode_slice_header error
 [h264 @ 0000000004a07de0] mmco: unref short failure
 [h264 @ 0000000004a074c0] mmco: unref short failure
 [mpegts @ 00000000003aa4a0] PES packet size mismatch0:35.36 bitrate=
 453.2kbits/s dup=18 drop=0
 [mp2 @ 000000000249ec40] incomplete frame
 Error while decoding stream #0:1: Invalid data found when processing input
 [h264 @ 0000000004a08700] error while decoding MB 22 28, bytestream (-5)
 [h264 @ 0000000004a08700] concealing 387 DC, 387 AC, 387 MV errors in B
 frame
 frame=  923 fps=432 q=31.0 Lsize=    2046kB time=00:00:36.88 bitrate=
 454.5kbits/s dup=20 drop=0
 video:1455kB audio:571kB subtitle:0 global headers:0kB muxing overhead
 0.990720%

 > ls -ld x*
 -rw-rw-rw-   1 user     group     2095104 Oct 16 17:40 xè.mpg
 -rw-rw-rw-   1 user     group     9400000 Oct 16 17:08 xè.ts
 drwxrwxrwx   1 user     group           0 Oct 16 17:40 xèdir

 > ls -l xèdir
 total 126
 -rw-rw-rw-   1 user     group      128580 Oct 16 17:40 log-0.log

 >
 }}}

 I understand that this kind of Windows oddity may not be too interesting
 to FFmpeg developers but it can be quite annoying for Windows users.

 As far as I am concerned, I use ffmpeg on Windows mainly through the GUI
 front-end named QtlMovie. All ffmpeg commands are generated by the front-
 end and I implemented a specific workaround for this problem. In the
 specific case of -passlogfile on Windows, I first translate the directory
 path of the log file with its DOS 8.3 counterpart. But this is just a non-
 portable workaround, pushing the problem into the ffmpeg user's code.

-- 
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/3056>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker


More information about the FFmpeg-trac mailing list