SVT-AV1 0.5 Released As Intel's Speedy AV1 Video Encoder

[dwagner]

Jan Ozer has an article looking at Netflix's VMAF here:

Regardless of how great the measurement method is that Netflix uses, they will certainly use less bandwidth for encoding than necessary to avoid visible compression artifacts. And that is because of the measure "profit", which rises with each bit/s they shave off.

[dwagner]

x264 --preset veryslow --tune ssim --crf 16 -o test.x264.crf16.264 orig.i420.y4m

For the uninitiated, the "very slow" preset is considered the "mastering" quality preset and crf 15 is generally considered visually lossless to the source; for this test they used "very slow" and crf 16 which means it would be nearly impossible for the average person to tell the difference between the source and the encoded version.

Further, the "very slow" preset, as the name implies is indeed very slow, encoding times are glacial.

If SVT-AV1 matched x264+veryslow+crf 16 with -q 20 -enc-mode 3 (as the command line says) then it doesn't get much better than that because with settings as aggressive as x264's SVT-AV1 would smoke it.

I don't see what you are trying to say here. If SVT-AV1 is as slow as "x264 --preset veryslow", while not providing better quality at the same bit rate, then clearly x264 is the better choice for that bit rate. As others already stated: A comparison of only speed without the other dimensions considered is useless.

I would say that Intel's SVT family of encoders are the future

Only if their combination of result dimensions becomes more competitive.
Currently, I see a lot of cases where even H.264 is preferable over HEVC, simply because spending lots of extra encoding CPU time comes with a very slim, if at all, advantage in image quality. Especially when plenty of bit rate is available, HECV isn't preferable to H.264, and AV1 will certainly be much worse for a long time.

[andreano]

objective analysis

I've had luck with sitkevij/ffmpeg:

docker run --rm -it sitkevij/ffmpeg:3.3-vmaf -i output.mkv -s 1920x1080 -r 50 -i input.yuv -lavfi libvmaf -f null -

Unfortunately, the bundled ffmpeg was too old to support AV1, so you would have to rebuild it. There was also a very apparent memory leak, which may or may not be fixed by upgrading ffmpeg.

[edwaleni]

Jan Ozer has an article looking at Netflix's VMAF here:

https://www.linkedin.com/pulse/finding-just-noticeable-difference-netflix-vmaf-jan-ozer

[edwaleni]

PSNR is beyond terrible as a metric (it favors blur, which is the most horrible thing you can do to human eyes), even SSIM is bad. They tell you *nothing* regarding actual video quality. What the psnr and ssim tunings in x264 do is they *disable* encoder tools that make the video look good! The ssim tuning disables psychovisual rate-distortion decisions, while the psnr tuning additionally disables adaptive quantization, those two being _the_ most important x264 tools that make it such a great encoder. Why does x264 do that? Simple, disabling these tools results in higher psnr/ssim scores. Which is completely backwards. So those two metrics are useless. An encoder comparison that deals with PSNR and SSIM is therefore completely worthless.

So that commandline in Spooktra's post - "x264 --preset veryslow --tune ssim --crf 16 -o test.x264.crf16.264 orig.i420.y4m" (emphasis mine)... yeah.

Netflix developed a new metric that takes a different approach, the magic of machine learning. It's called VMAF. But even VMAF isn't really it. There's just no substitute for actually looking at the encoded video. So, the people here asking for an "objective quality benchmark"... it doesn't exist.

I am only aware of two commercial products that perform objective analysis:

Moscow University's Visual Quality Comparison Tool (VQMT)
SSIMWave Video Quality-of-Experience Monitor (SQM)

Both of these are north of a $1000.

[Gusar]

PSNR is beyond terrible as a metric (it favors blur, which is the most horrible thing you can do to human eyes), even SSIM is bad. They tell you *nothing* regarding actual video quality. What the psnr and ssim tunings in x264 do is they *disable* encoder tools that make the video look good! The ssim tuning disables psychovisual rate-distortion decisions, while the psnr tuning additionally disables adaptive quantization, those two being _the_ most important x264 tools that make it such a great encoder. Why does x264 do that? Simple, disabling these tools results in higher psnr/ssim scores. Which is completely backwards. So those two metrics are useless. An encoder comparison that deals with PSNR and SSIM is therefore completely worthless.

So that commandline in Spooktra's post - "x264 --preset veryslow --tune ssim --crf 16 -o test.x264.crf16.264 orig.i420.y4m" (emphasis mine)... yeah.

Netflix developed a new metric that takes a different approach, the magic of machine learning. It's called VMAF. But even VMAF isn't really it. There's just no substitute for actually looking at the encoded video. So, the people here asking for an "objective quality benchmark"... it doesn't exist.

[edwaleni]

https://ffmpeg.org/ffmpeg-filters.html#psnr

https://ffmpeg.org/ffmpeg-filters.html#ssim

According to this they use "standard" SSIM. There are like 5 flavors of SSIM now days, but I wasn't aware that ffmpeg had the capability.

[Spooktra]

Do you happen to know what level of SSIM that they use?

3 component or multi scale?

https://ffmpeg.org/ffmpeg-filters.html#psnr

https://ffmpeg.org/ffmpeg-filters.html#ssim

[edwaleni]

FFMPEG includes tools to measure both SSIM and PSNR; SSIM varies between -1 (completely different) to 1 (exactly the same); SSIM values greater than .99 will usually correlate with pristine quality, .98 will correlate with acceptable quality and .97 will correlate with poor quality; with PSNR it is measured in dB and a measurement of 50 dB or better is considered mastering quality.

If you were to compare an encoded file to the source and calculate both a PSNR of 50 dB or greater AND an SSIM value of .99 or greater, then chances are you would have a hard time telling the encoded version from the source, but note to accomplish such a thing requires very high bit rate even with the best encoders.

Do you happen to know what level of SSIM that they use?

3 component or multi scale?

[Spooktra]

"Objective quality benchmarks are algorithms that compare the compressed video with the source and render a value that predicts how the compressed file would fare in subjective tests."

What open source tools would be recommended to perform an objective quality benchmark?

FFMPEG includes tools to measure both SSIM and PSNR; SSIM varies between -1 (completely different) to 1 (exactly the same); SSIM values greater than .99 will usually correlate with pristine quality, .98 will correlate with acceptable quality and .97 will correlate with poor quality; with PSNR it is measured in dB and a measurement of 50 dB or better is considered mastering quality.

If you were to compare an encoded file to the source and calculate both a PSNR of 50 dB or greater AND an SSIM value of .99 or greater, then chances are you would have a hard time telling the encoded version from the source, but note to accomplish such a thing requires very high bit rate even with the best encoders.