#1163(FFprobe:new): ffprobe can produce invalid XML
#1163: ffprobe can produce invalid XML ----------------------------------+--------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: new Priority: normal | Component: FFprobe Version: 0.10.2 | Keywords: Blocked By: | Blocking: Reproduced by developer: 0 | Analyzed by developer: 0 ----------------------------------+--------------------------------- ffprobe can output invalid XML as xml_escape_str only handles < > ' " and &. For example most escape characters below 32 are invalid UTF-8. This replacement version of the function replaces any invalid UTF-8 characters by the inverted question mark. {{{#!C static const char *xml_escape_str(char **dst, size_t *dst_size, const char *src, void *log_ctx) { // the unknown character (inverted question mark) const unsigned char BAD_CHARACTER_1 = 194, BAD_CHARACTER_2 = 191; const char *p; char *q; int copyAll = 1; size_t size = 1; /* precompute size */ for (p = src; *p;) { int badChar = 0; unsigned char byte; ESCAPE_CHECK_SIZE(src, size, SIZE_MAX-10); byte = (unsigned char)*p; if (byte < 32 && byte != 9 && byte != 10 && byte != 13) { badChar = 1; ++p; } else if (byte < 128) { switch (byte) { case '&' : size += 5; /* & */ copyAll = 0; break; case '<' : size += 4; /* < */ copyAll = 0; break; case '>' : size += 4; /* > */ copyAll = 0; break; case '\"': size += 6; /* " */ copyAll = 0; break; case '\'': size += 6; /* ' */ copyAll = 0; break; default: size++; } ++p; ++size; } else if (byte < 0xC0) { badChar = 1; ++p; } else { int extra; copyAll = 0; if (byte < 0xe0) extra = 1; else if (byte < 0xf0) extra = 2; else if (byte < 0xf8) extra = 3; else badChar = 1; if (badChar) ++p; else { ++p; for (int i = 0; i < extra && *p; ++i, ++p) { byte = (unsigned char)*p; if ((byte & 0xc0) != 0x80) badChar = 1; } if (!badChar) size += extra; } } if (badChar) { size += 2; copyAll = 0; } } ESCAPE_REALLOC_BUF(dst_size, dst, src, size); #define COPY_STR(str) { \ const char *s = str; \ while (*s) \ *q++ = *s++; \ } p = src; q = *dst; if (copyAll) COPY_STR(p) else { while (*p) { int badChar = 0; unsigned char byte; byte = (unsigned char)*p; if (byte < 32 && byte != 9 && byte != 10 && byte != 13) { badChar = 1; ++p; } else if (byte < 128) { switch (byte) { case '&' : COPY_STR("&"); break; case '<' : COPY_STR("<"); break; case '>' : COPY_STR(">"); break; case '\"': COPY_STR("""); break; case '\'': COPY_STR("'"); break; default: *q++ = *p; } ++p; ++size; } else if (byte < 0xC0) { badChar = 1; ++p; } else { int extra; copyAll = 0; if (byte < 0xe0) extra = 1; else if (byte < 0xf0) extra = 2; else if (byte < 0xf8) extra = 3; else badChar = 1; if (badChar) ++p; else { const char *startChar = p; int i; ++p; for (i = 0; i < extra && *p; ++i, ++p) { byte = (unsigned char)*p; if ((byte & 0xc0) != 0x80) badChar = 1; } if (!badChar) { for (i = 0; i < extra;) *q++ = *startChar++; } } } if (badChar) { *q++ = BAD_CHARACTER_1; *q++ = BAD_CHARACTER_2; } } } *q = 0; return *dst; } }}} -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML --------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: new Priority: normal | Component: FFprobe Version: 0.10.2 | Resolution: Keywords: | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------+----------------------------------- Comment (by cehoyos): Is the problem also reproducible with current git head? Please send unified patches to ffmpeg-devel and please provide an example (including console) on how to produce the invalid xml. -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:1> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML --------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: new Priority: normal | Component: FFprobe Version: 0.10.2 | Resolution: Keywords: | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------+----------------------------------- Comment (by Đonny): Hi, I can reproduce it with ffprobe version N-42704-g85761ef built on Jul 20 2012 20:39:19 -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:2> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML --------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: new Priority: normal | Component: FFprobe Version: 0.10.2 | Resolution: Keywords: | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------+----------------------------------- Changes (by nathanaeljones): * cc: n@… (added) Comment: I would be willing to fund development of a patch if it can be completed/integrated soon. Anyone interested? This is currently blocking our integration of ffmpeg thumbnailing within [http://imageresizing.net]. A large percentage (~40%) of videos cause ffprobe to produce invalid xml. -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:3> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML --------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: new Priority: normal | Component: FFprobe Version: 0.10.2 | Resolution: Keywords: | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------+----------------------------------- Comment (by cehoyos): Replying to [comment:3 nathanaeljones]:
I would be willing to fund development of a patch if it can be completed/integrated soon. Anyone interested?
First step would be to make this a reproducible ticket by providing a failing command line together with complete, uncut console output (current git head). -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:4> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML --------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: new Priority: normal | Component: FFprobe Version: 0.10.2 | Resolution: Keywords: | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------+----------------------------------- Comment (by saste): Replying to [comment:3 nathanaeljones]:
I would be willing to fund development of a patch if it can be completed/integrated soon. Anyone interested?
This is currently blocking our integration of ffmpeg thumbnailing within [http://imageresizing.net]. A large percentage (~40%) of videos cause ffprobe to produce invalid xml.
I'm interested in this, but I need more samples. Also the problem seems more related to invalid input data and/or encoding, so it looks more like a libavformat problem. Data should be probably validated and sanitized at the demuxing level, before rendering is performed at the application level. -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:5> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML --------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: new Priority: normal | Component: FFprobe Version: 0.10.2 | Resolution: Keywords: | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------+----------------------------------- Comment (by saste): Replying to [ticket:1163 Ian]:
ffprobe can output invalid XML as xml_escape_str only handles < > ' " and &. For example most escape characters below 32 are invalid UTF-8.
Useful samples are in tickets #2955. -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:6> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML --------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: new Priority: normal | Component: FFprobe Version: 0.10.2 | Resolution: Keywords: | Blocked By: Blocking: | Reproduced by developer: 0 Analyzed by developer: 0 | --------------------------------+----------------------------------- Changes (by eelco): * cc: eml+ffmpeg@… (added) -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:7> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML -------------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: open Priority: normal | Component: FFprobe Version: unspecified | Resolution: Keywords: utf8 | Blocked By: Blocking: | Reproduced by developer: 1 Analyzed by developer: 1 | -------------------------------------+----------------------------------- Changes (by saste): * analyzed: 0 => 1 * keywords: => utf8 * version: 0.10.2 => unspecified * status: new => open * reproduced: 0 => 1 Comment: Please test the patches in attachment and report. -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:8> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML -------------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: open Priority: normal | Component: FFprobe Version: unspecified | Resolution: Keywords: utf8 | Blocked By: Blocking: | Reproduced by developer: 1 Analyzed by developer: 1 | -------------------------------------+----------------------------------- Comment (by mente): Applied on latest version: works like a charm. Thanks @saste! -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:9> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML -------------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: open Priority: normal | Component: FFprobe Version: unspecified | Resolution: Keywords: utf8 | Blocked By: Blocking: | Reproduced by developer: 1 Analyzed by developer: 1 | -------------------------------------+----------------------------------- Comment (by mente): Hm, not so good as I thought. On corrupted files I got segmentation fault. -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:10> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML -------------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: open Priority: normal | Component: FFprobe Version: unspecified | Resolution: Keywords: utf8 | Blocked By: Blocking: | Reproduced by developer: 1 Analyzed by developer: 1 | -------------------------------------+----------------------------------- Comment (by saste): Replying to [comment:10 mente]:
Hm, not so good as I thought. On corrupted files I got segmentation fault.
Check new patchset, in case of error please provide the sample. -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:11> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
#1163: ffprobe can produce invalid XML -------------------------------------+----------------------------------- Reporter: Ian | Owner: stefano Type: defect | Status: closed Priority: normal | Component: FFprobe Version: unspecified | Resolution: fixed Keywords: utf8 | Blocked By: Blocking: | Reproduced by developer: 1 Analyzed by developer: 1 | -------------------------------------+----------------------------------- Changes (by saste): * status: open => closed * resolution: => fixed Comment: I believed this should be fixed in: {{{ commit cbba331aa02f29870581ff0b7ded7477b279ae2c Author: Stefano Sabatini <stefasab@gmail.com> Date: Wed Oct 2 16:22:17 2013 +0200 ffprobe: implement string validation setting This should fix trac tickets #1163, #2502. }}} Feel free to test and reopen the ticket in case of issues. -- Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1163#comment:12> FFmpeg <http://ffmpeg.org> FFmpeg issue tracker
participants (1)
-
FFmpeg