[Libav-user] Problem with h264_videotoolbox and AV_PIX_FMT_YUV420P

[Steve Green]

This is the basic pipeline that I have:

	BGRA frames -> sws_scale() -> YUV frames -> avcodec_encode_video2() -> av_interleaved_write_frame()

.. and this works perfectly with x264 and most of the time with h264_videotoolbox.  The issue with h264_videotoolbox happens when I try to stream at 854x480.  From looking at the output, it appears that someone along the way is not happy with width not being a multiple of 16.  There is a very clear shift/wrapping of pixels.  

As an experiment, I changed my use of av_image_fill_arrays() to align on 16 bytes (with a sufficiently larger buffer) and the resulting image is improved but still not correct.  The basic shift is gone and the image is legible but the colors are off and I can see an artifact over the image.

Given that x264 works and h264_videotoolbox doesn't, I’m tempted to point fingers at either VTCompressionSession or CVPixelBuffers + YUV, which would explain why I want to try AV_PIX_FMT_VIDEOTOOLBOX.  I’ve read pretty much everything I can find on this but I cant find a single mention of 16-byte-multiple widths or strides.

Any idea how to go about debugging this?