Rav1e 0.5 Brings More Speed-Ups For This Rust AV1 Encoder

[aorth]

I just encoded a 1080p Bluray with ffmpeg and libaom-av1 3.2.0 on cpu-used 4, crf 31, 2-pass, 10-bit, 2x2 tiles, and 12 threads in 20 hours on a Ryzen Zen 3 5600G. Quality looks excellent. According to perf stat the encode actually only used 3.6 CPUs. So the parallelism is not great...

[MadeUpName]

I have a feeling that some where along the line some one will take a profiler to the AV1 code and we will get orders of magnitude in improvement like what happened to opengl on Linux once the game companies got serious about it or the recent news about improvements in IOPS per core. I just feels like AV1 spends a lot of time twiddling it's thumbs.

[arQon]

Even at $100 per 4TB, there is the issue of backups. I use H.265 to get my entire collection to 2.2 TB, so I can fit a complete copy onto four different disks. That seems paranoid, but the rip-and-reencode process took over a year and I'll be furious if I lost everything.

Right, that's the important part if you're not just pirating stuff: the amount of time you invest in getting everything ripped and encoded in the first place.
For audio, it's trivial *now* as there's exactly one correct option, i.e. FLAC, but when I first ripped my CDs nearly 20 years ago I had to use ogg - and thus had to do all of them again a decade later once HDDs got large enough to make lossless an option.
For video though, even VERY low-res stuff like DVDs, you're already starting with a lossy source, and whatever you encode to - unless you can afford to burn 6-8GB per disc for the raw MPEG2 - needs to be "good enough forever" or the generational loss means you have to start over again from scratch.

> Our 800 films (most DVD) are encoded at what I believe is a quality level good enough that people can't see a difference. I know I can't.

That's really the only metric that matters.
For DVDs, I still went with h264 over h265, simply because of the time involved in encoding them. The very low resolution of DVD means anything can play them back regardless of the format, but also means that even in h264 the point at which you reach something indistinguishable from the source itself ("transparent", in the jargon of the field) the files are small enough that the additional compression of h265 doesn't really matter enough (or at least, didn't in my case) to make it worth the process taking 3x as long in elapsed time.

> One thing to note is that I think most newer gadgets have H.265 hardware decode.

Sure: but you're talking about a standard that is NOW 8 years old, and has had an enormous amount of marketing behind it and very little in the way of genuine competition for the first several years of its life. That's my point re h266 (and to some extent, AV1) - it takes a long time for HW decode to actually make its way into all the devices you may want to use for playback, and in some cases (like VPx) it often simply never happens at all, because by the time it becomes "worth" doing (if ever) based on demand, that demand is already moving on to whatever the next "standard" is instead, completely skipping whatever you chose. (And because of the vagaries of video formats, even if the format you used IS technically supported, it may well be so for only a handful of specific subsets, i.e. 8-bit only, or 4:2:0 only, etc etc).

So, realistically, encoding anything *non-transient* to some random "cutting edge" format is basically just gambling, and hoping that that format will actually do well enough to be viable for you 5 years from now. The Pi4, for example, has HEVC decode hardware but no AV1 support, to pick the easiest example of why the handful of people here who imagine that AV1 support is "everywhere" are delusional; and why there's a lot more to encoding your video library "well" - i.e. in a format that is useful to you *now* while also providing the best compression you can get - than just jumping on the bandwagon of whatever is currently getting the most hype.

> I ran my own H.264 vs. H.265 comparisons using equivalent CRF values (I think H.264 and H.265 have different CRF scales).

Correct. Transparency is ~16-18 on h264, and ~22-23 on h265. (It does depend quite a lot on the bit depths involved too, but I think that's getting too involved for this post).

> The H.265 took three or four times as long to encode, but always gave equal image quality for less disk space.

My notes from the time had h265 as roughly 2.6x slower, but also show that neither of the encoders scaled meaningfully past 4 cores. That may have improved by now, but I expect the ratios aren't much different even if so.

I think you made the right choice for your specific circumstances (which like I say isn't as easy as it seems at first glance, so well done) but those circumstances are different enough from one person to the next for that to not automatically mean that it would be the right choice even for someone in a very similar position. For me, the availability of many TB of total storage meant I could forgo h265 completely, and "waste" several hundred GB by using h264 for the DVDs, rather than "waste" 3x the electricity squeezing every last byte out of the transcoded files instead. Considering how laughably slow each successive generation of new codecs is compared to prior ones, those tradeoffs, as well as the issue of what can and can't actually PLAY those formats, are likely to remain relevant indefinitely.

[Michael_S]

arQon - thanks for that, great post. I don't have anything significant to add.

Maybe, with the benefit of hindsight, it would have been smarter for me to stick with H.264.

It's funny that you mention that the encoders don't meaningfully scale past four cores. I always ran ffmpeg with cpulimit keeping it at 4 cores, so I could do other things on my machine while the encoder ran. Maybe I stumbled across the optimal setting by dumb luck.