I'm watching ffmpeg crank out a ProRes from an older format QT at about 3 - 4x real time I'm on a 8-core machine, and it's using all of them + hyper-threading. It's really impressive. Also been converting to mp4 using GPU which is faster still -- mind-blowingly fast. My impression from searching online and from ffmpeg -h encoder=prores_ks that processing through GPU is not an option for ProRes transcoding. Is there something about transcoding to ProRes that makes GPU processing not an option? (and would it even be faster) Or is it possible and I just haven't found it yet?
Am Mo., 24. Feb. 2020 um 00:00 Uhr schrieb MediaMouth <communque@gmail.com>:
Is there something about transcoding to ProRes that makes GPU processing not an option?
"GPU encoding / decoding" nowadays means using a specific piece of hardware inside the GPU (but unrelated to any of its other usages) which is made to support a particular codec. As long as no hardware manufacturer supports ProRes (and this seems unlikely) no "GPU" ProRes transcoding support will be available. Carl Eugen
"GPU encoding / decoding" nowadays means using a specific piece of hardware inside the GPU (but unrelated to any of its other usages) which is made to support a particular codec. As long as no hardware manufacturer supports ProRes (and this seems unlikely) no "GPU" ProRes transcoding support will be available.
Carl Eugen
Thanks for the quick explainer.
As long as no hardware manufacturer supports ProRes (and this seems unlikely) no "GPU" ProRes transcoding support will be available.
Hi, Just for completeness, there is now the Afterburner card dedicated to ProRes decode and render (not a GPU), but a Mac Pro is the only platform that supports it and if you have that setup, videotoolbox will use it automatically. I wonder if it would work at all in an expansion chassis or PC... Regards, Ted Park
Am Mi., 26. Feb. 2020 um 07:31 Uhr schrieb Ted Park <kumowoon1025@gmail.com>:
As long as no hardware manufacturer supports ProRes (and this seems unlikely) no "GPU" ProRes transcoding support will be available.
Just for completeness, there is now the Afterburner card dedicated to ProRes decode and render (not a GPU), but a Mac Pro is the only platform that supports it and if you have that setup, videotoolbox will use it automatically.
FFmpeg's videotoolbox implementation is missing ProRes support. Carl Eugen
On 2/26/2020 5:01 AM, Carl Eugen Hoyos wrote:
Am Mi., 26. Feb. 2020 um 07:31 Uhr schrieb Ted Park <kumowoon1025@gmail.com>:
Just for completeness, there is now the Afterburner card dedicated to ProRes decode and render (not a GPU), but a Mac Pro is the only platform that supports it and if you have that setup, videotoolbox will use it automatically. FFmpeg's videotoolbox implementation is missing ProRes support.
Just curious, is that by intent or has no one implemented/updated the support? If it's intentional, what is the rationale for leaving it out? z!
Am Mi., 26. Feb. 2020 um 17:15 Uhr schrieb Carl Zwanzig <cpz@tuunq.com>:
On 2/26/2020 5:01 AM, Carl Eugen Hoyos wrote:
Am Mi., 26. Feb. 2020 um 07:31 Uhr schrieb Ted Park <kumowoon1025@gmail.com>:
Just for completeness, there is now the Afterburner card dedicated to ProRes decode and render (not a GPU), but a Mac Pro is the only platform that supports it and if you have that setup, videotoolbox will use it automatically.
FFmpeg's videotoolbox implementation is missing ProRes support.
Just curious, is that by intent or has no one implemented/updated the support?
Certainly not by intent. Carl Eugen
curious, why are you dying to get prores ? qt has at best 8 bits h264 encoding, you're just resampble colours to 10 bits. may be nvenc's lossless 10 bits h264 coud fit your needs as well, i'd go ~16x real time... On Wed, Feb 26, 2020 at 5:34 PM Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
Am Mi., 26. Feb. 2020 um 17:15 Uhr schrieb Carl Zwanzig <cpz@tuunq.com>:
On 2/26/2020 5:01 AM, Carl Eugen Hoyos wrote:
Am Mi., 26. Feb. 2020 um 07:31 Uhr schrieb Ted Park <
kumowoon1025@gmail.com>:
Just for completeness, there is now the Afterburner card dedicated to ProRes decode and render (not a GPU), but a Mac Pro is the only platform that supports it and if you have that setup, videotoolbox will use it automatically.
FFmpeg's videotoolbox implementation is missing ProRes support.
Just curious, is that by intent or has no one implemented/updated the support?
Certainly not by intent.
Carl Eugen _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user
To unsubscribe, visit link above, or email ffmpeg-user-request@ffmpeg.org with subject "unsubscribe".
On Wed, Feb 26, 2020 at 5:34 PM Carl Eugen Hoyos <ceffmpeg@gmail.com> wrote:
Am Mi., 26. Feb. 2020 um 17:15 Uhr schrieb Carl Zwanzig <cpz@tuunq.com>:
On 2/26/2020 5:01 AM, Carl Eugen Hoyos wrote:
Am Mi., 26. Feb. 2020 um 07:31 Uhr schrieb Ted Park <
kumowoon1025@gmail.com>:
Just for completeness, there is now the Afterburner card dedicated to ProRes decode and render (not a GPU), but a Mac Pro is the only platform that supports it and if you have that setup, videotoolbox will use it automatically.
FFmpeg's videotoolbox implementation is missing ProRes support.
Just curious, is that by intent or has no one implemented/updated the support?
Certainly not by intent.
On Feb 27, 2020, at 2:02 AM, andrei ka <andrei.k.gml@gmail.com> wrote:
curious, why are you dying to get prores ? qt has at best 8 bits h264 encoding, you're just resampble colours to 10 bits. may be nvenc's lossless 10 bits h264 coud fit your needs as well, i'd go ~16x real time...
RE "dying to get prores" -- so far I haven't encountered a codec worth dying for. The only reason for using it is that a) it's a widely standard in post production environments. b) I hadn't heard of nvenc lossless. Is it actually lossless or just perceptually lossless? My suspicion is it's missing some of features expected in broadcast editing (though to be fair the "professional broadcast world" frequently fails to live up to its own stated standards). h.265 isn't considered an industry standard, but if there are genuine benefits over current standards like ProRes and the DNxHRs and doesn't anger the techs too much, then it could see adoption. Curious to know more about it. RE "qt has at best 8 bits h264 encoding" -- To me this doesn't sound accurate. QT isn't synonymous with h.264. It's just a wrapper that can hold h.264 as well as plenty of other formats. ProRes QTs can handle 10- even 12-bit, as well as Rec 709, and alpha channels.
FFmpeg's videotoolbox implementation is missing ProRes support.
Oh, I never noticed that. It shouldn’t be too difficult right? Though I’m guessing the decoder would be of limited use for the command line tools. I think I will try comparing them on cpu, I was going through the docs and noticed Apple calling ffmpeg out on having an unlicensed implementation...
Am So., 1. März 2020 um 14:11 Uhr schrieb Ted Park <kumowoon1025@gmail.com>:
FFmpeg's videotoolbox implementation is missing ProRes support.
Oh, I never noticed that. It shouldn’t be too difficult right?
Hopefully not.
Though I’m guessing the decoder would be of limited use for the command line tools.
Why? Carl Eugen
On Sun, 1 Mar 2020, 19:55 Carl Eugen Hoyos, <ceffmpeg@gmail.com> wrote:
Am So., 1. März 2020 um 14:11 Uhr schrieb Ted Park <kumowoon1025@gmail.com
:
FFmpeg's videotoolbox implementation is missing ProRes support.
Oh, I never noticed that. It shouldn’t be too difficult right?
Hopefully not.
Though I’m guessing the decoder would be of limited use for the command line tools.
Why?
Carl Eugen
From the context of his email, this mostly infers to any perceived
performance gain H/W accelerated decode brings to the table in an encoding workload. It's not likely to be faster than a software based decoder, but will do the same task without using any CPU cycles nonetheless. Secondly, consider that (almost) zero filters in FFmpeg can take advantage of videotoolbox (at the moment), so even with the command line tools, chaining together workflows with complex filter chains that can take advantage of H/W acceleration on Mac OSX would be limited to the OpenCL (and Vulkan, now that we have a Vulkan HW Context in FFmpeg) based filters only. Pumping up hwupload/hwdownload filters in these chains further slows down things, often offsetting any performance benefit(s) such filters bring to the workflow. This is likely to change in the future if filters that can tap into videotoolbox come into play.
On Mar 1, 2020, at 2:28 PM, Dennis Mungai <dmngaie@gmail.com> wrote:
On Sun, 1 Mar 2020, 19:55 Carl Eugen Hoyos, <ceffmpeg@gmail.com <mailto:ceffmpeg@gmail.com>> wrote:
Am So., 1. März 2020 um 14:11 Uhr schrieb Ted Park <kumowoon1025@gmail.com
:
FFmpeg's videotoolbox implementation is missing ProRes support.
Oh, I never noticed that. It shouldn’t be too difficult right?
Hopefully not.
Though I’m guessing the decoder would be of limited use for the command line tools.
Why?
Carl Eugen
From the context of his email, this mostly infers to any perceived performance gain H/W accelerated decode brings to the table in an encoding workload. It's not likely to be faster than a software based decoder, but will do the same task without using any CPU cycles nonetheless.
Secondly, consider that (almost) zero filters in FFmpeg can take advantage of videotoolbox (at the moment), so even with the command line tools, chaining together workflows with complex filter chains that can take advantage of H/W acceleration on Mac OSX would be limited to the OpenCL (and Vulkan, now that we have a Vulkan HW Context in FFmpeg) based filters only. Pumping up hwupload/hwdownload filters in these chains further slows down things, often offsetting any performance benefit(s) such filters bring to the workflow.
This is likely to change in the future if filters that can tap into videotoolbox come into play.
Part of the original reason for posting CPU vs GPU question in the first place was just a curiosity about how FFmpeg was handling various workloads. Already I use it in automated pipelines as a welcome alternative to Adobe Media Encoder. I only recently started paying attention how it was making use of CPU cores and GPUs. What struck me watching the ProRes transcodes was how the process used all 8 cores and (presumably) hyper-threading (representing as 16 cores being used) Is FFmpeg making the determination how to distribute across the cores, or is the machine (in this case MacOS) making those decisions? Does FFmpeg have any control over these things, and if so, is there any reason an FFmpeg user might want manual control -- say to either maximize processing time ("use 'em al"l) or manage computing resources ("use only cores 1, 2, and 3, but leave 4, 6, and 6 free for something else")? Was also curious: Does anyone know if Media Encoder is using FFmpeg under the hood?
Am 02.03.2020 um 01:45 schrieb MediaMouth <communque@gmail.com>:
Part of the original reason for posting CPU vs GPU question in the first place was just a curiosity about how FFmpeg was handling various workloads. Already I use it in automated pipelines as a welcome alternative to Adobe Media Encoder.
I only recently started paying attention how it was making use of CPU cores and GPUs.
What struck me watching the ProRes transcodes was how the process used all 8 cores and (presumably) hyper-threading (representing as 16 cores being used)
This is very difficult for non-intra codecs but less difficult for ProRes.
Is FFmpeg making the determination how to distribute across the cores, or is the machine (in this case MacOS) making those decisions?
The application has to control the number of threads it uses, the os is scheduling the threads on the available cpu cores.
Does FFmpeg have any control over these things, and if so, is there any reason an FFmpeg user might want manual control --
Yes, because multithreading is not free in general although this is mostly an issue for real (non-intra) codecs.
say to either maximize processing time ("use 'em al"l) or manage computing resources ("use only cores 1, 2, and 3, but leave 4, 6, and 6 free for something else")?
This is the os‘ decision and not related to the application.
Was also curious: Does anyone know if Media Encoder is using FFmpeg under the hood?
I don’t think so (note the different licenses). Carl Eugen
Hi,
So codec engineering companies like NGCodec, MainConcept, Beamr and MulticoreWare turn open source-based, ffmpeg workflows into FPGAs that, when mature firmware implementations, chip companies like Intel & NVIDIA turn into real hardware: masked GPUs. Do I have that right?
I think that is a bit of a stretch… It almost makes it sound like they copy paste a bunch of code from open source projects into a box that turns them into custom silicon designs. Besides, I think the way Intel or NVIDIA might use FPGAs (other than in their FPGA products) is probably very different from the way a software codec is accelerated using an FPGA.
What struck me watching the ProRes transcodes was how the process used all 8 cores and (presumably) hyper-threading (representing as 16 cores being used)
Scalability is a bigger focus in some codecs than others, the tradeoff shows in other areas like compression.
Is FFmpeg making the determination how to distribute across the cores, or is the machine (in this case MacOS) making those decisions? Does FFmpeg have any control over these things, and if so, is there any reason an FFmpeg user might want manual control -- say to either maximize processing time ("use 'em al"l) or manage computing resources ("use only cores 1, 2, and 3, but leave 4, 6, and 6 free for something else")?
Hardware resources like that literally at the level of physical CPU cores is managed at the lowest levels of an OS to more usable abstractions. Maybe Media Encoder not taking up more than a certain amount of CPU time is an example, in many machines it runs alongside and shares resources with e.g. Premier considered higher priority.
Was also curious: Does anyone know if Media Encoder is using FFmpeg under the hood?
I believe AME uses MainConcept. Regards, Ted Park
On 03/01/2020 11:01 PM, Ted Park wrote:
Hi,
So codec engineering companies like NGCodec, MainConcept, Beamr and MulticoreWare turn open source-based, ffmpeg workflows into FPGAs that, when mature firmware implementations, chip companies like Intel & NVIDIA turn into real hardware: masked GPUs. Do I have that right?
I think that is a bit of a stretch… It almost makes it sound like they copy paste a bunch of code from open source projects into a box that turns them into custom silicon designs...
What is implemented in FPGAs is not solely so-called software-in-silicon. Sometimes it is software-in-silicon -- marginal speedup -- but usually software algorithms are realized as systems of FSMs (finite state machines) -- huge speedup -- that simply mimic the original software processes. I presume that what is implemented in GPUs (CUDA cores, for example) are the FSMs with custom hardware tool blocks (transform processors and the like) operating as slaved coprocessors shared by the cores -- even bigger speedup.
...Besides, I think the way Intel or NVIDIA might use FPGAs (other than in their FPGA products) is probably very different from the way a software codec is accelerated using an FPGA. That is undoubtedly true. Some are implemented using a client-server model and some are implemented as subprocesses (just as a main program uses subroutines). So long as the implementation is via FPGA, it doesn't matter whether its done in the own FPGA product or licensed from a software-in-silicon vendor like MGCodec.
Hi,
What is implemented in FPGAs is not solely so-called software-in-silicon. Sometimes it is software-in-silicon -- marginal speedup -- but usually software algorithms are realized as systems of FSMs (finite state machines) -- huge speedup -- that simply mimic the original software processes.
FSMs are pretty abstract in my head, but I meant that simplifying the logic so they mimic original behavior closely enough can’t be a trivial one. I think that with the scale of human effort it would involve, by using the libraries, it would end up contributing indirectly at least. But I may have misunderstood it seems…
That is not what I meant. What I'm getting at is that it appears that the latest streaming protocols & methods are being developed via ffmpeg, then being tested-debugged-retested-etc. via ffmpeg users, then implemented in FPGAs by software-in-silicon vendors like NGCodec, MainConcept, Beamr, etc., and finally cast into FSMs in GPUs.
Which protocols, do you mean like its API?
So codec engineering companies like NGCodec, MainConcept, Beamr and MulticoreWare turn open source-based, ffmpeg workflows into FPGAs that, when mature firmware implementations, chip companies like Intel & NVIDIA turn into real hardware: masked GPUs. Do I have that right?
If they market standalone hardware encoders they probably write firmware for it, but I guess that could be some build of linux with ffmpeg. But I/O isn’t going to be handled by GPUs and FPGAs, not sure what streaming protocols you mean. Software development being driven by advances in hardware is just normal for codecs I feel; I don’t know if FPGAs started being used before new codecs like h.265 were standardized and bitstreams frozen, but there was existing hardware.
I presume that what is implemented in GPUs (CUDA cores, for example) are the FSMs with custom hardware tool blocks (transform processors and the like) operating as slaved coprocessors shared by the cores -- even bigger speedup.
Individual cuda cores could actually be described as a FSM, I think, but it is managed in a group, so they are like the coprocessors with shared scheduler and memory. Regards, Ted Park
I hope what the analogies below will help folks better understand the technology we use. Transistors are to logic gates as quarks are to electrons, protons, and neutrons. Logic gates come in 2 flavors: AND and OR, and it's from them alone that all other digital elements are made. Logic gates are to FSMs (finite state machines) as electrons, protons, and neutrons are to atoms. FSMs are to CPU units (code crackers, execution units, cache controllers) as atoms are to molecules. Microcode is relatively simple sequences that operate on CPU units in the way that RNA sequences operate on ribosomes. CPU units are to CPU cores as ribosomes are to cells. CPU cores are to main boards as cells are to organs. Machine code coordinates the operations of CPU cores in the way that hormones coordinate the operations of organs. Main boards are to computers as organs are to organisms. Programs determine computer behavior in the way that instincts (and sometimes, thoughts) determine the behaviors of organisms. So, you see, programs are layers removed from machine code and machine code is layers removed from microcode. Microcode requires no execution environment. Machine code requires a CPU environment. Programs require a computer environment. FPGAs are collections of configurable logic gates that can be configured to make anything that can be made with logic gates, including any type of processor or sequencer in addition to static logic. Since they are configurable, much capacity is lost to connections not made, for example, to redundant input/output transistors not utilized, to routing channels only partly used, etc. However, in addition to a sea of uncommitted gates, some FPGAs contain hand crafted (compact), commonly needed peripherals (e.g., SPI, HDMI) and even cores (e.g., ARM). SoS (software-on-silicon) licensors such as NGCodec are companies that have developed the particular configurations (i.e., circuitry) to make an FPGA (such as are made by Xilinx, which purchased NGCodec), that are particular to a specific task such as processing video streams or network protocols. So. you see, thinking about the logic gates in FPGAs and what can be built with them is like thinking about electrons, protons, and neutrons and what can be built with them.
I'm not sure I'm hijacking this thread. The subject is actually quite related. Welcome to The Matrix. I had no idea. So codec engineering companies like NGCodec, MainConcept, Beamr and MulticoreWare turn open source-based, ffmpeg workflows into FPGAs that, when mature firmware implementations, chip companies like Intel & NVIDIA turn into real hardware: masked GPUs. Do I have that right? Now I see why so many ffmpeg-user submissions are streaming oriented. All this to reproduce 20th century broadcast media, but with a difference: Tracking & real-time telemetry. Hmmm... Maybe robots making money obsolete won't be a bad thing after all. What a rabbit hole. Twenty years ago, I considered starting an FPGA-based, firmware design business but I couldn't see a specific application aside from contract work, so I retired instead. Oh, well. I again apologize for asking so many seemingly trivial questions. Obviously, I'm just a tourist here. Regards, Mark. Phil Rhodes: If you're reading this, open your eyes. It's all about money. Improved tourist-class, user documentation doesn't make money.
Am Mo., 2. März 2020 um 03:48 Uhr schrieb Mark Filipak <markfilipak.windows+ffmpeg@gmail.com>:
So codec engineering companies like NGCodec, MainConcept, Beamr and MulticoreWare turn open source-based, ffmpeg workflows into FPGAs that, when mature firmware implementations, chip companies like Intel & NVIDIA turn into real hardware: masked GPUs. Do I have that right?
There may be a language barrier but to me this paragraph sounds at least extremely misleading, there is no indication whatsoever that any company ever "turn[ed an] ffmpeg workflow into" hardware. Given that we have a good (or friendly) relationship to some of these companies, I would kindly ask you to refrain from using this wording. Carl Eugen
On 03/02/2020 03:56 AM, Carl Eugen Hoyos wrote:
Am Mo., 2. März 2020 um 03:48 Uhr schrieb Mark Filipak <markfilipak.windows+ffmpeg@gmail.com>:
So codec engineering companies like NGCodec, MainConcept, Beamr and MulticoreWare turn open source-based, ffmpeg workflows into FPGAs that, when mature firmware implementations, chip companies like Intel & NVIDIA turn into real hardware: masked GPUs. Do I have that right?
There may be a language barrier but to me this paragraph sounds at least extremely misleading, there is no indication whatsoever that any company ever "turn[ed an] ffmpeg workflow into" hardware.
Given that we have a good (or friendly) relationship to some of these companies, I would kindly ask you to refrain from using this wording.
Carl Eugen
What you think is important to me, Carl. I used to work at Intel as a product line architect. I sat on 3 divisional planning councils, including graphics processors, and co-chaired one. Later, I represented Wyse Technology in industry technology standards, engineering working groups. I outlined what appeared to me to be the direction of open source video development: That what is developed using open source code is migrated to FPGAs and hence to GPUs. I asked if I have it right (correct). Of course, the ffmpeg crew works to exploit the capabilities that are thereby created. This is all to be celebrated. What is it that you object to? What in what I wrote is wrong or that industry chip providers would find offensive? Of course you have cordial relations with industry. I expect that open source development is also financially supported by industry groups and manufacturers. There's nothing wrong with that. Comments? Regards, Mark.
Am Mo., 2. März 2020 um 10:57 Uhr schrieb Mark Filipak <markfilipak.windows+ffmpeg@gmail.com>:
I used to work at Intel as a product line architect. I sat on 3 divisional planning councils, including graphics processors, and co-chaired one. Later, I represented Wyse Technology in industry technology standards, engineering working groups.
I outlined what appeared to me to be the direction of open source video development: That what is developed using open source code is migrated to FPGAs and hence to GPUs.
(I understand correctly that you are describing your personal experience copying open-source technology into proprietary hardware, right?) I didn't know this, from a technical pov I find it extremely surprising, from a moral pov a little concerning, from a legal perspective I am appalled and I hope the people responsible will get their response one day.
I asked if I have it right (correct).
I wonder how you can ask people who weren't there if something that you witnessed is right... But I guess all this is just the language barrier... Carl Eugen
On 03/02/2020 05:12 AM, Carl Eugen Hoyos wrote:
Am Mo., 2. März 2020 um 10:57 Uhr schrieb Mark Filipak <markfilipak.windows+ffmpeg@gmail.com>:
I used to work at Intel as a product line architect. I sat on 3 divisional planning councils, including graphics processors, and co-chaired one. Later, I represented Wyse Technology in industry technology standards, engineering working groups.
I outlined what appeared to me to be the direction of open source video development: That what is developed using open source code is migrated to FPGAs and hence to GPUs.
(I understand correctly that you are describing your personal experience copying open-source technology into proprietary hardware, right?)
I was in product planning at Intel 40 years ago. There was no open source technology. But computer & chip companies in California and Texas regularly cooperated via industry standards organizations and engineering working groups. It was the same when I worked for Wyse and the Gang of Nine in the mid 1980s, but with cooperation across the world including Epson, Olivetti, and (informally) Philips. All of that was considered Standards work, not collusion, and I expect that the same sort of thing links ffmpeg with the leading video accelerator suppliers.
I didn't know this, from a technical pov I find it extremely surprising, from a moral pov a little concerning, from a legal perspective I am appalled and I hope the people responsible will get their response one day.
Why would you be surprised, concerned, appalled? From your active role in ffmpeg-user, I assumed you were in ffmpeg development. Was that assumption incorrect?
I asked if I have it right (correct).
I wonder how you can ask people who weren't there if something that you witnessed is right...
I haven't witnessed anything. That's why I asked if I was right regarding the video technology development track. To use an American euphemism, I'm just connecting the dots.
On Mon, Mar 02, 2020 at 04:57:23 -0500, Mark Filipak wrote:
I used to work at Intel as a product line architect. I sat on 3 divisional planning councils, including graphics processors, and co-chaired one.
[OT, sorry] BTW, I have worked as Intel Corp. as well, incidentally both in graphics processors ("Petascale Development" ;-)) and in FPGA technology. Both in the same project, and before I was all too avid in video technology.
I outlined what appeared to me to be the direction of open source video development: That what is developed using open source code is migrated to FPGAs and hence to GPUs. I asked if I have it right (correct).
I'd like to pitch in to what Carl Eugen doubted: You hardly want to "convert" software to hardware. Simplified (not sure about all the terms): In software, execution is linear, procedular, while in hardware, everything is parallel by design (while data flow again can be "linear"). You wouldn't want to give up this advantage of hardware over software. You want to mix the best of both worlds. (BTW, for those who aren't aware: hardware, such as chips, ASICs, and also FGPA content, is described in something you could call software. But it's not procedural in the same way.) What hardware would do, is to build accelerated "calculators" for certain operations. This might be a complete codec (raw in, encoded out, like GPUs do, and probably like Amazon AWS does), or just bits of hardware capable of computations which standard CPU components aren't capable of at the same speed. Think MMX, SSE, but specialized for graphics or e.g. vector accelerations. Either of these would be hooked into a software flow, as ffmpeg does with hardware ("GPU") encoding. Many smaller video related operations could already be shifted to GPU hardware, but I believe ffmpeg's GPU accelerations don't take advantage of this. Perhaps its OpenCL and OpenGL uses do, but I don't believe there are many. (Or is there any ffmpeg software encoder assisted by hardware, except for the MMX/SSE/AVX type CPU accelerations?)
Of course, the ffmpeg crew works to exploit the capabilities that are thereby created. This is all to be celebrated. What is it that you object to? What in what I wrote is wrong or that industry chip providers would find offensive?
If someone took the ffmpeg sources to "reimplement" them in hardware, selling that service without open-sourcing their efforts. I believe that is what is meant. Sorry if I drifted off, used some incorrect terms, or have a misunderstanding of this. It's just that my understanding of "FPGA based" seems to be different than yours. Cheers, Moritz
On 03/02/2020 07:24 AM, Moritz Barsnick wrote:
On Mon, Mar 02, 2020 at 04:57:23 -0500, Mark Filipak wrote:
I used to work at Intel as a product line architect. I sat on 3 divisional planning councils, including graphics processors, and co-chaired one.
[OT, sorry]
OT, eh? Just like Congressional testimony.
BTW, I have worked as Intel Corp. as well, incidentally both in graphics processors ("Petascale Development" ;-)) and in FPGA technology. Both in the same project, and before I was all too avid in video technology.
Oh goodie! Hey, Brother, when did you work at Intel? I'll bet in the early 00s. I was at San Tomas 4. I see from streetview that the building isn't even there any more. ;-( I lived in Felton. $-)
I outlined what appeared to me to be the direction of open source video development: That what is developed using open source code is migrated to FPGAs and hence to GPUs. I asked if I have it right (correct).
I'd like to pitch in to what Carl Eugen doubted: You hardly want to "convert" software to hardware...
Agreed; totally. You must not have read the whole thread as I mentioned mimicking procedural code via finite state machines (FSMs). As you note below, such things as MMX & SSE are CPU opcode extensions that, though optimized for graphics, are still code and don't come close to the speed of FSMs. Most FPGAs operate as peripherals that require OS drivers, while some (few) FPGAs are populated entirely by FSMs that are fed (and operate directly on) streams.
...Simplified (not sure about all the terms): In software, execution is linear, procedular, while in hardware, everything is parallel by design (while data flow again can be "linear"). You wouldn't want to give up this advantage of hardware over software. You want to mix the best of both worlds.
(BTW, for those who aren't aware: hardware, such as chips, ASICs, and also FGPA content, is described in something you could call software. But it's not procedural in the same way.)
What hardware would do, is to build accelerated "calculators" for certain operations. This might be a complete codec (raw in, encoded out, like GPUs do, and probably like Amazon AWS does), or just bits of hardware capable of computations which standard CPU components aren't capable of at the same speed. Think MMX, SSE, but specialized for graphics or e.g. vector accelerations. Either of these would be hooked into a software flow, as ffmpeg does with hardware ("GPU") encoding.
Many smaller video related operations could already be shifted to GPU hardware, but I believe ffmpeg's GPU accelerations don't take advantage of this. Perhaps its OpenCL and OpenGL uses do, but I don't believe there are many. (Or is there any ffmpeg software encoder assisted by hardware, except for the MMX/SSE/AVX type CPU accelerations?)
I haven't used them, but I see from posts to this list that some folks are applying CUDA-specific filters.
Of course, the ffmpeg crew works to exploit the capabilities that are thereby created. This is all to be celebrated. What is it that you object to? What in what I wrote is wrong or that industry chip providers would find offensive?
If someone took the ffmpeg sources to "reimplement" them in hardware, selling that service without open-sourcing their efforts. I believe that is what is meant.
That is not what I meant. What I'm getting at is that it appears that the latest streaming protocols & methods are being developed via ffmpeg, then being tested-debugged-retested-etc. via ffmpeg users, then implemented in FPGAs by software-in-silicon vendors like NGCodec, MainConcept, Beamr, etc., and finally cast into FSMs in GPUs. Good for all: Open source developers get support & methods to 'talk' to chips and GPU makers get well developed, proven protocols & methods.
Sorry if I drifted off, used some incorrect terms, or have a misunderstanding of this. It's just that my understanding of "FPGA based" seems to be different than yours.
Well, I'm very familiar with FPGAs and have done many FPGA designs.
Am Mo., 2. März 2020 um 14:35 Uhr schrieb Mark Filipak <markfilipak.windows+ffmpeg@gmail.com>:
That is not what I meant. What I'm getting at is that it appears that the latest streaming protocols & methods are being developed via ffmpeg, then being tested-debugged-retested-etc. via ffmpeg users, then implemented in FPGAs by software-in-silicon vendors like NGCodec, MainConcept, Beamr, etc., and finally cast into FSMs in GPUs. Good for all: Open source developers get support & methods to 'talk' to chips and GPU makers get well developed, proven protocols & methods.
Is there any proof for this claim (that sounds damaging to me)? Carl Eugen
On 03/02/2020 11:14 AM, Carl Eugen Hoyos wrote:
Am Mo., 2. März 2020 um 14:35 Uhr schrieb Mark Filipak <markfilipak.windows+ffmpeg@gmail.com>:
That is not what I meant. What I'm getting at is that it appears that the latest streaming protocols & methods are being developed via ffmpeg, then being tested-debugged-retested-etc. via ffmpeg users, then implemented in FPGAs by software-in-silicon vendors like NGCodec, MainConcept, Beamr, etc., and finally cast into FSMs in GPUs. Good for all: Open source developers get support & methods to 'talk' to chips and GPU makers get well developed, proven protocols & methods.
Is there any proof for this claim (that sounds damaging to me)?
Carl Eugen
No proof. No claim. A conjecture based on what plainly exists. Not damaging at all.
participants (8)
-
andrei ka -
Carl Eugen Hoyos -
Carl Zwanzig -
Dennis Mungai -
Mark Filipak -
MediaMouth -
Moritz Barsnick -
Ted Park