[FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"

[Carl Eugen Hoyos]

2018-10-12 11:59 GMT+02:00, Jerome Martinez <jerome at mediaarea.net>:
> On 12/10/2018 11:15, Carl Eugen Hoyos wrote:
>> 2018-10-11 14:01 GMT+02:00, Jerome Martinez <jerome at mediaarea.net>:
>>
>>> - why is having a number > INT_MAX an issue? modern machines
>>> are 64-bit and have 8+ GB of RAM
>> So where is the issue?
>
> The issue is that vanilla FFmpeg shows a warning in the scenario I
> indicate (a 128 MB frame, far less than 2 GB), without obvious reason of
> this warning (the multiplier used and the underlying limitation to 2 GB,
> lower than the available RAM, are hard coded without comments about the
> reason).

At a very general level, I believe a warning that very large resolutions
can have a negative effect seems not absurd.
I believe it was already argued that pathological input frames can lead
to multiplied output size.

>> If above is true for you, can can simply remove the check locally, no?
>
> My goal is that **others** don't have this warning when they use a
> vanilla version of FFmpeg.

You seem to be 100% sure that the warning is bad, I am not convinced.

> Before sending a patch for removing the limitation I don't understand
> (or lowering a number I don't understand), I ask if there is a reason
> for it (I guess that if this code is here, there is a reason and I
> imagined that FFmpeg developers could kindly explain the reason), isn't
> it the right thing to do? Do you prefer that I send directly a patch?
>
>> (I hope you agree that what you write is not generally true.)
>
> I don't catch what you want to indicate: do you mean that the only way
> in FFmpeg to be compatible with all machines is to set an hard coded
> limit to 2 GB?
> I don't catch the goal, as you can have machines with 1 GB so this test
> would be useless on them (it will not prevent other parts of the code to
> have to do checks about RAM allocations) as well for machines having lot
> of RAM (they can handle big frames).

I wanted to explain that not every system FFmpeg runs on is a 64 bit
system, I know very well that FFmpeg will have issues on memory-
limited systems, iirc, it hits dnxhd on real-world systems.
(The default user memory limits hit you on AIX afaiu although
system memory is >100G.)

> As I understand, the following test:
>      if ((ret = ff_alloc_packet2(avctx, pkt, maxsize, 0)) < 0)
>          return ret;
>
> permits to check that FFmpeg has enough RAM for storing the max encoded
> frame size, whatever is the size of the RAM (1 GB or 8 GB), so I don't
> understand the reason maxsize is limited to 2 GB before the call to
> ff_alloc_packet2.

> I kindly request more details about how hard coding 2 GB in the code
> helps, for both machines having 1 GB & machines having 8 GB. Looks like
> I am personally not smart enough for understanding that alone.

I believe there are at least two reasons:
One is that on 32bit systems, you cannot allocate larger blocks, the
second reason is that dos is considered a security issue and allocating
more that 2G for one frame seems like a dos approach.

One solution may be to have a configure option that raises the
allocation limit on 64 bit systems or to raise it for every 64bit
system.

Carl Eugen