[Libav-user] Is a custom I/O read callback allowed to overwrite the buffer if it returns AVERROR_EOF?

Jyrki Vesterinen jyrkive at nekonyansoft.com
Tue Jul 5 21:37:54 EEST 2022

I am developing a game with the Unity game engine. For various reasons, our 
game uses custom asset archives instead of Unity's native assets, and 
utilizes FFmpeg for decoding images and audio inside these custom archives.

To allow FFmpeg to read data from these encrypted asset archives, we give 
it custom I/O read callbacks implemented in C# and made accessible to 
native code via reverse P/Invoke. The C# delegate for the read callback is 
as follows:

delegate int ReadPacket(IntPtr opaque, [Out, 
MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 2)] byte[] buf, int 

The "Out" is a directional attribute informing the C# compiler that the 
caller will use contents of the array but the callee doesn't care about its 
initial content. For more information, see 

This means that FFmpeg will always see what my read callback has explicitly 
written to the buffer, but whatever was in the buffer before FFmpeg called 
the callback is not guaranteed to be preserved, even if the read operation 
didn't fill the whole buffer or even read any bytes at all.

I have discovered that one of our Ogg Vorbis music tracks fails to play 
correctly due to this. Logging FFmpeg's calls to the callbacks shows this 
(note: I have given FFmpeg a 1MB buffer):

1. ReadPacket, 1048576 bytes. The length of the entire track is 1029379 
bytes, so we write it into the buffer and return 1029379.
2. Seek, with "whence" set to AVSEEK_SIZE. We return 1029379.
3. ReadPacket, 1048576 bytes. Since we're already at the end of the file, 
we return AVERROR_EOF and P/Invoke overwrites the entire buffer.

After this, oggdec.c:ogg_read_page() fails to find a sync word and FFmpeg 

I can easily fix this by adding the "In" directional attribute, i.e. 
turning the C# delegate into

delegate int ReadPacket(IntPtr opaque, [In, Out, 
MarshalAs(UnmanagedType.LPArray, SizeParamIndex = 2)] byte[] buf, int 

This instructs P/Invoke to preserve the original content of the array when 
calling the function, and thus it remains after the call too. However, it 
comes with a performance penalty.

My question is: is the read callback supposed to preserve content of the 
array, or is it a bug that the Ogg demuxer assumes it to have remained 
after calling the callback if it returns AVERROR_EOF?

Jyrki Vesterinen
NekoNyan Ltd

More information about the Libav-user mailing list