SVT-AV1 0.5 Released As Intel's Speedy AV1 Video Encoder

[wswartzendruber]

If Intel keeps this up, they're going to be a software company.

[andreano]

three dimensions:

output size × output quality (many automated metrics or perceptual) × encoding speed

To ease visualization you usually equalize one dimension, like trying to configure all encoders to produce same size, or same quality according to some automated metric, and then plot the other two dimensions.

This is not a focus of Phoronix.

Well said, but I know a very practical methodology that might be within Michael's reach (or someone could help add to PTS if he's busy):

For each encoder and bitrate, find the slowest speed setting that is faster than real-time. Now we are down to 2 dimensions; bitrate and quality, which can be measured and plotted.

[edwaleni]

"Objective quality benchmarks are algorithms that compare the compressed video with the source and render a value that predicts how the compressed file would fare in subjective tests."

What open source tools would be recommended to perform an objective quality benchmark?

[Spooktra]

"Objective quality benchmarks are algorithms that compare the compressed video with the source and render a value that predicts how the compressed file would fare in subjective tests."

What open source tools would be recommended to perform an objective quality benchmark?

FFMPEG includes tools to measure both SSIM and PSNR; SSIM varies between -1 (completely different) to 1 (exactly the same); SSIM values greater than .99 will usually correlate with pristine quality, .98 will correlate with acceptable quality and .97 will correlate with poor quality; with PSNR it is measured in dB and a measurement of 50 dB or better is considered mastering quality.

If you were to compare an encoded file to the source and calculate both a PSNR of 50 dB or greater AND an SSIM value of .99 or greater, then chances are you would have a hard time telling the encoded version from the source, but note to accomplish such a thing requires very high bit rate even with the best encoders.

[edwaleni]

FFMPEG includes tools to measure both SSIM and PSNR; SSIM varies between -1 (completely different) to 1 (exactly the same); SSIM values greater than .99 will usually correlate with pristine quality, .98 will correlate with acceptable quality and .97 will correlate with poor quality; with PSNR it is measured in dB and a measurement of 50 dB or better is considered mastering quality.

If you were to compare an encoded file to the source and calculate both a PSNR of 50 dB or greater AND an SSIM value of .99 or greater, then chances are you would have a hard time telling the encoded version from the source, but note to accomplish such a thing requires very high bit rate even with the best encoders.

Do you happen to know what level of SSIM that they use?

3 component or multi scale?

[Spooktra]

Do you happen to know what level of SSIM that they use?

3 component or multi scale?

https://ffmpeg.org/ffmpeg-filters.html#psnr

https://ffmpeg.org/ffmpeg-filters.html#ssim

[edwaleni]

https://ffmpeg.org/ffmpeg-filters.html#psnr

https://ffmpeg.org/ffmpeg-filters.html#ssim

According to this they use "standard" SSIM. There are like 5 flavors of SSIM now days, but I wasn't aware that ffmpeg had the capability.

[Gusar]

PSNR is beyond terrible as a metric (it favors blur, which is the most horrible thing you can do to human eyes), even SSIM is bad. They tell you *nothing* regarding actual video quality. What the psnr and ssim tunings in x264 do is they *disable* encoder tools that make the video look good! The ssim tuning disables psychovisual rate-distortion decisions, while the psnr tuning additionally disables adaptive quantization, those two being _the_ most important x264 tools that make it such a great encoder. Why does x264 do that? Simple, disabling these tools results in higher psnr/ssim scores. Which is completely backwards. So those two metrics are useless. An encoder comparison that deals with PSNR and SSIM is therefore completely worthless.

So that commandline in Spooktra's post - "x264 --preset veryslow --tune ssim --crf 16 -o test.x264.crf16.264 orig.i420.y4m" (emphasis mine)... yeah.

Netflix developed a new metric that takes a different approach, the magic of machine learning. It's called VMAF. But even VMAF isn't really it. There's just no substitute for actually looking at the encoded video. So, the people here asking for an "objective quality benchmark"... it doesn't exist.

[edwaleni]

PSNR is beyond terrible as a metric (it favors blur, which is the most horrible thing you can do to human eyes), even SSIM is bad. They tell you *nothing* regarding actual video quality. What the psnr and ssim tunings in x264 do is they *disable* encoder tools that make the video look good! The ssim tuning disables psychovisual rate-distortion decisions, while the psnr tuning additionally disables adaptive quantization, those two being _the_ most important x264 tools that make it such a great encoder. Why does x264 do that? Simple, disabling these tools results in higher psnr/ssim scores. Which is completely backwards. So those two metrics are useless. An encoder comparison that deals with PSNR and SSIM is therefore completely worthless.

So that commandline in Spooktra's post - "x264 --preset veryslow --tune ssim --crf 16 -o test.x264.crf16.264 orig.i420.y4m" (emphasis mine)... yeah.

Netflix developed a new metric that takes a different approach, the magic of machine learning. It's called VMAF. But even VMAF isn't really it. There's just no substitute for actually looking at the encoded video. So, the people here asking for an "objective quality benchmark"... it doesn't exist.

I am only aware of two commercial products that perform objective analysis:

Moscow University's Visual Quality Comparison Tool (VQMT)
SSIMWave Video Quality-of-Experience Monitor (SQM)

Both of these are north of a $1000.

[edwaleni]

Jan Ozer has an article looking at Netflix's VMAF here:

https://www.linkedin.com/pulse/finding-just-noticeable-difference-netflix-vmaf-jan-ozer