[FFmpeg-trac] #6240(undetermined:new): HandBrake 1.0.3 segfaults or encodes 0 frames with FFmpeg 3.1.7 (works with 3.2.4)

[FFmpeg]

#6240: HandBrake 1.0.3 segfaults or encodes 0 frames with FFmpeg 3.1.7 (works with
3.2.4)
--------------------------------------+----------------------------------
             Reporter:  rathann       |                     Type:  defect
               Status:  new           |                 Priority:  normal
            Component:  undetermined  |                  Version:  3.1.5
             Keywords:                |               Blocked By:
             Blocking:                |  Reproduced by developer:  0
Analyzed by developer:  0             |
--------------------------------------+----------------------------------
 Summary of the bug:
 HandBrake 1.0.3 segfaults in decomb filter (when encoding anything using
 default settings) or encodes 0 frames (if I disable the filter in video
 filters tab) when built with FFmpeg 3.1.7 (works with 3.2.4)
 How to reproduce:

 Case 1:
 1. (On Fedora 25,) build HandBrake 1.0.3 against FFmpeg 3.1.7.
 2. In ghb, open any video file and click "start encoding".
 3. ghb segfaults in decomb filter:

 {{{
 [...]
 Thread 27 "ghb" received signal SIGSEGV, Segmentation fault.
 [Switching to Thread 0x7fffe1016700 (LWP 31173)]
 detect_gamma_combed_segment (segment_stop=204, segment_start=0,
 pv=0x7fffd80008c0) at ../libhb/comb_detect.c:443
 443             int height  = pv->ref[0]->plane[pp].height;
 (gdb) bt full
 #0  detect_gamma_combed_segment (segment_stop=204, segment_start=0,
 pv=0x7fffd80008c0) at ../libhb/comb_detect.c:443
         y = <optimized out>
         x = <optimized out>
         stride = <optimized out>
         width = <optimized out>
         height = -671059792
         mthresh = <optimized out>
         athresh = <optimized out>
         athresh6 = <optimized out>
         pp = <optimized out>
 #1  decomb_filter_thread (thread_args_v=0x7fffd801a870) at
 ../libhb/comb_detect.c:1061
         pp = 0
         pv = <optimized out>
         segment = 0
         segment_start = 0
         segment_stop = 204
         thread_args = 0x7fffd801a870
 #2  0x000000000046887b in hb_thread_func (_t=0x7fffd8001070) at
 ../libhb/ports.c:860
         t = 0x7fffd8001070
 #3  0x00007ffff25006ca in start_thread (arg=0x7fffe1016700) at
 pthread_create.c:333
         __res = <optimized out>
         pd = 0x7fffe1016700
         now = <optimized out>
         unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736968353536,
 3440324452558995082, 0, 140736250525967,
                 140736968354240, 140736968353536, -3440258779170570614,
 -3440300979176309110}, mask_was_saved = 0}},
           priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup
 = 0x0, canceltype = 0}}}
         not_first_call = <optimized out>
         pagesize_m1 = <optimized out>
         sp = <optimized out>
         freesize = <optimized out>
         __PRETTY_FUNCTION__ = "start_thread"
 #4  0x00007fffed607f7f in clone () at
 ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
 No locals.

 }}}
 Case 2:
 1. (On Fedora 25,) build HandBrake 1.0.3 against FFmpeg 3.1.7.
 2. In ghb, open any video file.
 3. Disable deinterlacing filters in video filtering tab.
 4. Click "start encoding".
 5. ghb doesn't crash, but encodes 0 video frames.

 All the above works fine if built against FFmpeg 3.2.4.

 HandBrake upstream has already said using FFmpeg is an unsupported
 configuration as they support libav only:
 https://github.com/HandBrake/HandBrake/issues/631 . However, I'd
 appreciate any help in identifying where the bug is.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/6240>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker