I'm trying to pass UTF-8 character strings to a stripped-down ffmpeg binary I built, but I can't seem to get av_open_input_file to work. This isn't too surprising, but I'm kind of at the end of my rope trying to figure out how to play with files that have non-latin characters in the name. Simply renaming the desired files isn't exactly suitable, unfortunately. Is there some kind of really obvious solution that I am missing? Thanks, Jon
On Sat, Jul 08, 2006 at 03:32:40AM -0500, Jonathan Beavers wrote:
I'm trying to pass UTF-8 character strings to a stripped-down ffmpeg binary I built, but I can't seem to get av_open_input_file to work. This isn't too surprising,
It is surprising. UTF-8 is completely transparent to apps and should work with no special support whatsoever, except for when it comes to displaying the names on a terminal or in a gui.
but I'm kind of at the end of my rope trying to figure out how to play with files that have non-latin characters in the name.
Perhaps the encoding on disk is not UTF-8? No mangling of filenames is performed so you must use the on-disk encoding. Or... are you perhaps using windows and not telling us that? In that case it's probably an insurmountible limitation unless you can fix mingw/msvcrt.dll to correctly translate the UTF-8 filenames to the windows-internal UTF-16. Rich
On 7/9/06, Rich Felker <dalias at aerifal.cx> wrote:
On Sat, Jul 08, 2006 at 03:32:40AM -0500, Jonathan Beavers wrote:
I'm trying to pass UTF-8 character strings to a stripped-down ffmpeg binary I built, but I can't seem to get av_open_input_file to work. This isn't too surprising,
It is surprising. UTF-8 is completely transparent to apps and should work with no special support whatsoever, except for when it comes to displaying the names on a terminal or in a gui.
I'm pretty new to UTF-8 and Unicode, so I'm probably not a great source on what is surprising or unsurprising! :) At any rate, I was under the impression that UTF-8 was transparent for ASCII characters only, and then non-ASCII characters would require something along the lines of wfopen().
but I'm kind of at the end of my rope trying to figure out how to play with files that have non-latin characters in the name.
Perhaps the encoding on disk is not UTF-8? No mangling of filenames is performed so you must use the on-disk encoding. Or... are you perhaps using windows and not telling us that? In that case it's probably an insurmountible limitation unless you can fix mingw/msvcrt.dll to correctly translate the UTF-8 filenames to the windows-internal UTF-16.
I apologize about this--I'm stuck with Windows. One possible solution that I've seen somewhere on a Cygwin/MinGW mailinglist was that I could continue passing in filenames converted to UTF-8 and then inside my ffmpeg-based program, simply convert the filenames to UTF-16. I'll be sure to send an email to the list with my results, for the sake of posterity... but I'll certainly keep in mind that MinGW just may not play well with non-ASCII text. Thanks, Jon
On Tue, Jul 11, 2006 at 11:43:31AM -0500, Jonathan Beavers wrote:
On 7/9/06, Rich Felker <dalias at aerifal.cx> wrote:
On Sat, Jul 08, 2006 at 03:32:40AM -0500, Jonathan Beavers wrote:
I'm trying to pass UTF-8 character strings to a stripped-down ffmpeg binary I built, but I can't seem to get av_open_input_file to work. This isn't too surprising,
It is surprising. UTF-8 is completely transparent to apps and should work with no special support whatsoever, except for when it comes to displaying the names on a terminal or in a gui.
I'm pretty new to UTF-8 and Unicode, so I'm probably not a great source on what is surprising or unsurprising! :) At any rate, I was under the impression that UTF-8 was transparent for ASCII characters only, and then non-ASCII characters would require something along the lines of wfopen().
This is a windowsism. A long time ago MS thought they had a chance of killing c/unix/inet/open standards by redefining filenames (and all strings) to be UTF-16 and replacing the whole C library with new functions to manipulate them. Meanwhile the original functions continued to work, but only by having a locale-specific 8bit codepage associated with them, and internally converting from the local codepage to UTF-16 and then calling wfopen, etc. On the other hand, unix systems have always had the view that filenames are binary data, aside from '/' and '\0' having special meaning, and that interpretation is up to applications. No idea of wide character filenames was ever introduced. Historically filenames were generally interpreted as representing 8bit character sets dependent on the user's locale, but naturally this is a huge problem if files are to be shared between systems or between users with different languages on the same system. The solution is of course to use UTF-8 for all filenames and markup-less text data. More and more systems are switching to it with generally good results.
I apologize about this--I'm stuck with Windows. One possible solution that I've seen somewhere on a Cygwin/MinGW mailinglist was that I could continue passing in filenames converted to UTF-8 and then inside my ffmpeg-based program, simply convert the filenames to UTF-16. I'll be sure to send an email to the list with my results, for the sake of posterity... but I'll certainly keep in mind that MinGW just may not play well with non-ASCII text.
Ideally mingw would have a C library that treats char string filenames as UTF-8, converts them to UTF-16, and calls the native windows functions for opening UTF-16 filenames. Sadly it doesn't. I don't know what the best workaround is; even what I propose would not "play nice" with other windows apps, I fear. The core problem is that MS made a very very bad choice around 1995 and everyone is living with the effects of it now... Rich
On 7/11/06, Rich Felker <dalias at aerifal.cx> wrote:
I apologize about this--I'm stuck with Windows. One possible solution that I've seen somewhere on a Cygwin/MinGW mailinglist was that I could continue passing in filenames converted to UTF-8 and then inside my ffmpeg-based program, simply convert the filenames to UTF-16. I'll be sure to send an email to the list with my results, for the sake of posterity... but I'll certainly keep in mind that MinGW just may not play well with non-ASCII text.
Ideally mingw would have a C library that treats char string filenames as UTF-8, converts them to UTF-16, and calls the native windows functions for opening UTF-16 filenames. Sadly it doesn't. I don't know what the best workaround is; even what I propose would not "play nice" with other windows apps, I fear. The core problem is that MS made a very very bad choice around 1995 and everyone is living with the effects of it now...
Rich
I was able to come up with a solution to this "Unicode" filename problem on Windows. The solution is not very far from what you had proposed above, Rich. It's actually an unimaginably simple addition to libavformat/file.c. It boils down to something like: #ifdef WIN32 static int file_wopen(URLContext *h, const char *filename, int flags) { wchar_t wPath[ strlen( filename ) + 1 ]; [ ... ] int result = MultiByteToWideChar(CP_UTF8, 0, filename, -1, wPath, sizeof(wPath)); if (!result) { return -1; } fd = _wopen(wPath, access, 0666); } #else [ file_open stuff here ] #endif Somewhat obviously the function pointer in file_protocol should be updated to use file_wopen instead of file_open, and it would be a great idea to include windows.h. :) If there's any demand for a patch or something, I'll happily oblige! At any rate, thanks for the help. Jon
On Mon, Aug 07, 2006 at 03:02:06PM -0500, Jonathan Beavers wrote:
On 7/11/06, Rich Felker <dalias at aerifal.cx> wrote:
I apologize about this--I'm stuck with Windows. One possible solution that I've seen somewhere on a Cygwin/MinGW mailinglist was that I could continue passing in filenames converted to UTF-8 and then inside my ffmpeg-based program, simply convert the filenames to UTF-16. I'll be sure to send an email to the list with my results, for the sake of posterity... but I'll certainly keep in mind that MinGW just may not play well with non-ASCII text.
Ideally mingw would have a C library that treats char string filenames as UTF-8, converts them to UTF-16, and calls the native windows functions for opening UTF-16 filenames. Sadly it doesn't. I don't know what the best workaround is; even what I propose would not "play nice" with other windows apps, I fear. The core problem is that MS made a very very bad choice around 1995 and everyone is living with the effects of it now...
Rich
I was able to come up with a solution to this "Unicode" filename problem on Windows. The solution is not very far from what you had proposed above, Rich. It's actually an unimaginably simple addition to libavformat/file.c.
It boils down to something like: #ifdef WIN32 static int file_wopen(URLContext *h, const char *filename, int flags) { wchar_t wPath[ strlen( filename ) + 1 ]; [ ... ] int result = MultiByteToWideChar(CP_UTF8, 0, filename, -1, wPath, sizeof(wPath)); if (!result) { return -1; } fd = _wopen(wPath, access, 0666); } #else [ file_open stuff here ] #endif
Somewhat obviously the function pointer in file_protocol should be updated to use file_wopen instead of file_open, and it would be a great idea to include windows.h. :) If there's any demand for a patch or something, I'll happily oblige!
At any rate, thanks for the help.
A few comments: 1. Why name the function file_wopen and add the extra complexity of deciding which function to call? Just call it file_open either way.. 2. Does using CP_UTF8 actually work? I suspect the command line (and thus the filenames) arrive in whatever the locale's legacy 8bit encoding is, not UTF-8, but I may be wrong? How do you get the program to receive the command line in UTF-8? 3. Shouldn't wchar_t be TCHAR or whatever the nonsense windows 16bit type is? 4. It's not ok to include windows.h from non-windows-specific files since windows.h might conflict with names used in the source file presently or at a future time. Either prototype the functions you need explicitly or put the windows-specific code in its own .c file. 5. Using strlen(filename) unconditionally as an array size can lead to stack overflow and crash if the user provides an insanely long name. No idea if this is possible but it's still dangerous IMO. You should first test for sane length, something like: size_t l = strlen(filename); wchar_t wPath[l < PATH_MAX ? l+1 : PATH_MAX]; if (l >= PATH_MAX) return -1; With these issues addressed and some testing, a patch for your idea would be well-received I expect. Rich
participants (2)
-
dalias＠aerifal.cx -
jonathan.beavers＠gmail.com