A few things - this is one step in a long, LONG path. AV2 is currently unusable in its current state (the encoder typically runs at around 1fps on good hardware), and likely will remain so til ~2028 when the first av2 hardware accelerated chips start dropping. Even then, I wouldn't expect AV2 streams to be common til 2030.

IMO, if it were just the efficiency gains on the table (which are substantial - ~20-30% over AV1), I'd say that AV2 isn't worth it. The biggest thing it does add though is multi-stream support, which will be a big win for VR and live sports. The other fun thing is you can send an alpha channel as a separate stream, which the file will then composite for proper transparent video support.

Based on AV1's trajectory, hardware encode isn't necessary (though it is nice). The current encoder is a reference encoder. Now that the spec is finalized, expect significant speed improvements from production encoders (realtime likely won't happen until we get it in hardware though)

I strongly disagree with it not being required. I run a small social news site - AV1 is still prohibatively expensive both for the server and clients for software encoding/decoding. Without hardware encoding, the tradeoff for better compression ratios in exchange for massive battery use + very long processing times for encoding simply isn't worth it. In order to get AV1 out, I have to often process a h264 version of a video first anyway, just so the client isn't left waiting for their video upload to finish encoding. This means to support AV1 I'm not saving anything on the storage side. Even youtube only does AV1 encodes for extremely popular videos - it only makes sense to do at significant scale.

I love AV1, don't get me wrong, and I can't wait til I can switch over to it as a single unified format for both images and video, but for now the cost is too high until hardware acceleration becomes ubiquitous

I went and checked some youtube videos on my front page, A video with 15k views had an AV1 encode, while a video with 160 views was h264 only. So extremely popular videos is not how I would describe it, probably by views, almost everything you watch on youtube is AV1. But they skip the extra encodes for videos relatively no one watches.

Last time I checked they do it for new videos only. Older videos with 1M views aren't even on AV1.

Makes sense, new videos are where most of the streams are happening, I wouldn't be surprised if they start to reduce the number of transcodes to save space as a video drops in popularity. h264 will work on everything so they need that as a minimum, with AV1 just being there to save on data transfer.

Thanks You. I have been saying this since the launch of AV1 on HN, Doom9 and other places. I wanted to mention even Google uses custom dedicated hardware ASIC for AV1 encode.

I wish LCEVC is more widespread. For the same H.264 encoding time you get 50% to 60% Bitrate reduction using it with H.264.

Hardware encode is required if you want things like video calls, camera recording and such to use it.

It isn’t required for content distribution platforms which aren’t realtime and the cost of encode is easily made up by hundreds of thousands of streams.

One of the interesting usage of AV1 was specifically for low bitrate calls, and software encoding was perfectly fine, even on mobile.

With low enough resolution, framerate and bitrate, you can get a quality stream without significant encoding artifacts compared to any other codec. It is in production right now and has been for a while.

The tradeoff CPU / bandwidth is quite advantageous in situations like this. And no, AV1 HW encoders cannot usually be used, they are not designed for a tight bitrate control or realtime communications like software encoding is usually.

> One of the interesting usage of AV1 was specifically for low bitrate calls, and software encoding was perfectly fine, even on mobile.

You really want hardware decoding on mobile, otherwise you end up with 40 minutes battery life. Fortunately, for typical videoconference resolutions, VP8 and H.264 are just fine. AV1 is nice to have, though, due to excellent support for synthetic content (screen sharing), and for scalable video coding (a much more elegant solution than simulcast, IMHO).

In the world I live in, the general plan is to stick to VP8 and H.264 for the time being, and to skip to AV1 when it's universally available on mobile. I haven't seen any features of AV2 which would justify waiting for it.

No, you do NOT want hardware anything on mobile if you are targeting smaller bitrate that are not that taxing on the CPU, when the conditions are otherwise so bad that the call would either drop or be unusable. HW encoders produce bad results at low bitrate. HW decoders usually have issues with the temporal encodings used and they may also just not accept those streams (a lot of test scenarios are movies, and the RTC tools are poorly supported).

I worked on shipping it to Chromium, WebRTC and Google Meet many years ago and we had many publications about it: - https://blog.google/products-and-platforms/products/duo/4-ne... - https://webrtchacks.com/the-hidden-av1-gift-in-google-meet/

The use case is not screensharing or a large conference room, but mainly a simpler talking face for a 1:1 chat, but with good quality as packet loss is then not as impactful on a 30KBps stream with AV1 than a 50KBps VP8 stream.

> we had many publications about it

I'd be interested in learning more, but the links you provide are just advertising copy. Could you please provide links to actual technical articles on your conclusions?

The internals are usually confidential and it's hard to find an engineer willing to make a comprehensive write-up about those: they want to make tech and not spend time proofing a tech write-up for public consumption (they already had to make an internal one!).

So the middle ground is that you have those "marketing" copies that demo the tech. One of the telling part of those is how you can get a fine usable 30KBps stream at very low bitrate with AV1 compared to a higher bitrate H264 that is unusable. It doesn't tell you that because you are using a lot less bytes, you will be trading CPU power consumption for radio power consumption and it's a tricky comparison, but in general, it's a favorable trade for the user who has very bad network conditions and is trying to make a call. The goal is to make the call work at all cost, not to save the battery and having a useless stream of data transferred.

> HW encoders produce bad results at low bitrate.

Is that poor implementation or is it inherently harder to implement in hw encoders?

There's a few reasons, I suspect fixed resource depth might be factor in poor hardware single pass encoding ...

What does limit them, though, is pseudo real time single pass pipelines.

I see the best encoding results from two pass - one fast run to work out the easy compress and hard compress parts of a video and then a second pass to get the optimal results on a stream that's already got a budget in mind for each section through the advantage of foresight as to what's left to do.

As someone else said, it's poor single pass encoding performance targeted for the tools used in real-time communications. This type of usage is "new" to hardware manufacturers and they poorly test it as it's easier to make a chip good enough for decoding the general case for watching your favorite movie platform than do something comprehensive.

One aspect of real-time encoding is that the frames are not ordered or structured the same linear way as they used to be in older format. Now, we have temporal and spatial encoding, which allows for better frame drops or efficiency or a stream that is decodable at multiple resolutions at the same time.

An example of temporal encoding is that you have a sequence of frames at 15fps (T0) that are all referencing the previous one, and sometimes an I-frame that is a full independent picture you can start decoding from. Then, you can have another temporal layer (T1) , where for every frame at the base 15 fps layer (T0), you insert a new frame that depends on it. You end up having a 30 fps stream! And if your network connection is worse, or you hardware can't keep up, you can drop the T1 layer and only use the T0 layer. This works great for real-time! In the specs, you could have more layers with more complex dependency chains, but 3 layers is as high as you want to go.

Spatial encoding is a bit different, you will have frame at the highest resolution, but they reference another frame at half the resolution (who may also do the same). Each higher layer means just adding more details over the smaller size frame that you have at the base. To decode an image, you need to have all the frames available. This can also be combined with the temporal encoding above. While this isn't useful for a 1-1 communication, in conference rooms, it's a great optimization as while you may send your full HD picture to the server, you may not want to send that to everyone when you're just a thumbnail who is not actively speaking. So the conference server will not send the full HD picture, but the lower resolution only. And since you don't want to do the encoding on the server (it's expensive, slow and you need to trust the intermediate service with your secret stuff), doing spatial encoding on the client side is better.

Those techniques are all advanced ones that would be used if available universally. Unfortunately, a lot of hardware decoders choke on those, despite being part of the specs. And it's not that they can't generate a stream with those, they also sometimes can't decode them (breaking the spec).

And finally, the hardware encoders are tuned for higher bitrate work. Ask them to do a 3MBps stream, they'll do fine. Ask them for a 30KBps stream, they'll make garbage most of the time.

Have you said this for Audio Codec I would have agreed. I do not know a single Smartphone Video Conferencing software that uses CPU encoding rather than hardware encoding. Neither WhatsApp or FaceTime, perhaps the largest of the two real time Video Call uses AV1.

Yeah, no production or large scale VC system is running software AV1 encoders on smartphones. You will drain a full phone battery in 1-2 hours of calls.

It just doesn’t make sense and will result in extraordinary power/battery drainage at best, or output that’s worse than hardware encoding.

The only way you could get AV1 to software encode in realtime AND low latency on a mid-range Android chip is by disabling or skipping nearly all of the compression/encoding features that make it good at low bitrate.

> Yeah, no production or large scale VC system is running software AV1 encoders on smartphones. You will drain a full phone battery in 1-2 hours of calls.

Yeah but, half jokingly, Zoom does that (draining the battery in an hour) already :P

So, status remains quo, the commons remain tragic, and glory to H.264 forever?

> tragic

H.264 isn't even that bad at all, if not the best depending on how you look at it. Our Internet bandwidth, both on the backend and front end on Mobile 5G is increasing with plenty more room to grow. While computation decoding and storage isn't.

i.e If bandwidth is infinite and free, and we are only optimising for decoding power usage. H.264 wins in a lot of this scenario.

H264 is lacking a lot of features (behind patents) that are essential to real-time communications. It's available, but by far, the worst offender for call quality. Modern call technology will want to use temporal and spatial scaling which are not available in the profiles supported by most H264 encoders and decoders.

Those tools are available for VP8 (temporal only), VP9 and AV1 and improve the quality of calls quite a lot when used right. I don't know about about the internals of H265 and H266 as those are also behind patents and no one wanted to touch them in the real-time conferencing space.

At least until a better codec has widespread enough hardware support, I think.

Google Meet can do it. You don't want the full conference with AV1, just use it for very low bitrate scenarios with a high packet loss possibly. Phones are a good target system. And I know this is quite opposite to expectations.

It is a lot better to send a stable and visually ok stream with AV1 at 30KBps than fail to send a VP8 50KBps stream that is unusable anyway and is subject to twice as many packet lost than a lower bitrate solution.

It is possible they use AV1 in other scenarios now, but I left the team a while back now and I haven't checked what they are now using under the hood.

Anything running on a battery will need hardware acceleration

Even without encoding, as long as decoding is supported for AV2, streaming sites like Youtube can always transcode uploads. The encoder on mobile hardware is more of a nice bonus as long as we have an AV1 encoder available in the meantime.

Youtube is doing this now. Most semi popular videos have an AV1 transcode, something interesting is I've seen youtube chooses to use the AV1 format even on my macbook which doesn't have a hardware decoder, I had a look at the CPU usage and there is a 50% load on one thread on my M1, but aside from extra battery usage, this is basically negligible since I'm likely not doing any other CPU heavy tasks while watching video.

That's fine and not anything new for codecs, they always take a long time before mass adoption.

Take a look at AV1 itself, you can't even say it's really ubiquitous on all hardware. It's quite well along in adoption compared to early days, but some mobile devices are still lacking hardware acceleration for it.

It is still ironic to me that my Steam Deck has decode AV1 acceleration, on a really old CPU/GPU combo.

AMD added AV1 encoding only in later SoCs though. Next Steam Deck will have both.

Why is it ironic?

> The biggest thing it does add though is multi-stream support

I would have thought this would be a part of the container format rather than the video codec?

Google H264 SVC

The way things are going, we can pretty much forget about AV2 hardware encoders in PCs anytime soon. All the newest, best chip capacity is being completely hogged by Apple and AI companies.

Unless chipmakers port the AV2 design to older, cheaper nodes, it’s just not happening for average users. We’ll probably see some Chinese TV chip makers throw in an AV2 decoder just to check a box, but as an actual encoder? I wouldn't count on it anytime soon.

I wouldn't be so pessimistic, Intel and AMD aren't going to stop making CPUs, and if their integrated graphics adds AV2 it will be motivation enough for others to follow.

Ram and Ssd is just the start capitalism dictates profits will suck all production to AI. CPu have 3-5 year production timeline ie cpu prices will start going up a lot more as previous production contracts expire you might even get shortages new consumer cpus won't be affordable or spread as fast as you think.

PCs don’t need hardware encoders. There are no realtime jobs on these. You can let it encode in however long it takes.

You need hardware encoders for things like cameras because they need to encode in real time since the buffer would quickly overflow otherwise.

Use cases for hardware encoders in PCs:

- Video calls

- Screen/webcam recording

- Live streaming

- Real-time transcoding for media servers (don’t know much about this but I’ve heard it’s a thing)

- Game streaming

- Video editing (making exporting less frustrating)

Everything here is really niche, except video calls (and even that...).

In other words, unless on smart phones, don't expect broadly distributed AV2 encoding hardware.

If it does happen on PC, it will be most likely some courtesy of the hardware chip designers.

Video calls are niche, and offline encoding of video isn't?

Are you sure this isn't just “things I do are commonplace, and things I don't are incredibly niche”?

What? All are niche, and video calls are at best quite rare. People still prefer audio calls or even text messages mostly.

Video calls is just a feature expected on smart phones, and it means there we better have an hardware encoder.

I don't know if AV2 is defining them, but it needs hard "profiles", that to give a scope for hardware implementations.

Well, it all depends on how AV2 is designed, and defining those profiles is very probably a feedback loop between the AV2 designers and hardware implementors.

I recently stayed with a non-technical friend for a few weeks; they easily spent a third of their working hours on zoom calls.

I think virtually everyone who works an office job uses Teams/Zoom very often nowadays, I can’t think of anyone I know who wouldn’t

Screen recording is not niche, anyone who works remotely probably does video calls daily and the others are somewhat more niche but I'd guess that at least 50-70% of people do at least one of these things

I think, screen recording is ultra-niche, and I don't know why, I feel better when I know my OS is unable to do just that.

(As a side project, I am writting my walyand compositor, and I know that if it supports screen recording, that's going to be something certainly hacky and I can disable at compile time).

Do you not ever want to show someone something on your screen? For a bug report or to demonstrate a project you’re working on? Or do a screen share to ask for help with setting something up (which is admittedly a bit different from a recording but only in that it’s live)?

I can see a lot of non-techy people not knowing how to do this but among people who know how to do it I bet it’s above 50% who use it

> The biggest thing it does add though is multi-stream support

This was supported in H.264 MVC but only saw real use for 3D movies on physical BluRays. With almost no content available outside that.

Looking at how GPU development has been sidetracked for NPU, I worry that this is 2035 target at best. Manufacturers will push for maximising matrix operation silicon area. In the era of trillion dollar investments into datacenters, traffic cost is afterthought. The only benefitors might be YouTube, Netflix and such, but on their scale investment into ISP level caches might be cheaper.

In HW accelerated chips, what part of the calculations they usually accelerate? Could it be possible to repurpose old HW?

Essentially all of the processing of the video data, barring the container format which the CPU uses to know what part of the data to send to the GPU or the Audio chip or codec.

And HW acceleration is generally a preset baked in version of the encoder or decoder. These are mostly codec specific.

So, no using hardware from previous versions.

Now, you can see some software that tries to use the GPU itself, instead of the dedicated hardware acceleration, to decode, but that isn't the HW accelerated, and may not operate in real time.

At the same time, that will consume much more power, eliminating some of the advantages or the pure HW rendition, especially important for mobile.

I could see an argument being made for encoding, if it is 2x or faster than the CPU, but I haven't looked at any in a while, so don't know the speeds.

One of the biggest gains of having dedicated hardware is that the computation doesn’t happen on the general hardware.

This is what makes it viable on mobile devices where system responsiveness and power efficiency are high priority.

Generally these hardware decoders haven’t been retoolalble.

Typically things like quantization and motion estimation.

> The biggest thing it does add though is multi-stream support, which will be a big win for VR and live sports.

AV1 supports it too ?

> enabling high-quality video delivery at significantly lower bitrates

> likely will remain so til ~2028 when the first av2 hardware accelerated chips start dropping

This might sound dumb, but whats the point if its intended for slower devices, but those slower devices don't even exist yet?

i believe you would need new hardware to ENCODE AV2 video but DECODING AV2 video should work on current and a fair bit of older technology. The folks of Videolan are developing an AV2 decoder called "Dav2d" that should be able to play (decode) AV2 on modest hardware today.

https://www-test.videolan.org/projects/dav2d/

It's not for slower devices, it's for lower data transfer bills for providers like YouTube.

It's a win for consumers as well if you can get better video quality or more reliable calls on a slower connection.

so that new devices can adopt it

They can't adopt it if it doesn't exist.

And what about old devices? I'm sure someone out there is still using an s5 as a daily driver... Future proofing is great and all, but 240p on modern devices looks like trash, even worse than tube tv.

240p for someone’s webcam thumbnail in Microsoft teams is perfectly sufficient though.

They can keep using whatever they're currently using.

This doesn't take away anything. It's a new standard.

Based on your argument, one should add new safety standards to cars like seat belts, because old cars might not have them.

Well, I just noticed I said 'should' and not 'shouldn't' which I wanted to say

But to your point,

While they might not be required to retrofit, one shouldn't stop defining new safety standards.

one shouldn't*

One shouldn't add new safety standards*

writing before coffee...

I feel like in 2030 it's more likely that we send 480p and just upscale with ai on the other end

480p ? More like 200i. There's a race to the bottom driven by those "up to" codecs.

Where do you see information about the efficiency gains over AV1?

It's at the end on the conclusion slide @24:40:

https://www.youtube.com/watch?v=Se8E_SUlU3w&t=1480

Does anyone know what is so costly to calculate in AV2?

In general they just increase the numbers on everything when they go up a generation.

e.g. if you check in 4 directions to see if you can reuse a chunk then make it check in 8 or 16.

Faster encoders will have smart heuristics on when to use these new abilities and when to skip them but the reference encoder will try everything in a dumb way to eke out a tiny win to maximize a theoretical advantage and map out the extreme best case.

I feel like these encoders that require acceleration are a big reason why good hardware obsoletes so quickly.

What I'm interested in is seeing how this will improve the AVIF image format. AVIF stomps the competition for low-bitrate still images (where chroma subsampling is used). For lossless images, not so much. Lossless JPEG XL and lossless WEBP make lossless AVIF look like a joke.

AVIF is for sure my favorite image format right now. No other format has the quadfecta of lossless, HDR, transparency, browser support. Plus as you said, for very compressed images it looks amazing. It blows my mind how small AVIF files can be. Also, unlike HEIC and Ultra HDR JPEG, it actually supports HDR natively as part of the file format rather than doing the hacky sidecar gain map trick. I know it doesn't matter to everyone, but I just love HDR and AVIF is the only format that I feel like really takes it seriously.

I really don't.

1. Lossless AVIF is a joke often beaten by WebP and even PNG. Even worse for grayscale.

2. Chroma subsampling remains a bad idea for still images unless the resolution is high enough to hide the artifacts.

3. Tooling is the worst part, AV1 encoders are basically focused 99% on video and leave a measly 1% to image; unlike JXL, of course. SVT-AV1 still doesn't do YUV444 and libaom was unusable. Fortunately, the unpaid enthusiasts were here: https://giannirosato.com/blog/post/the-multimedia-renaissanc... (and more recently https://giannirosato.com/blog/post/oavif/)

I don't see AVIF being used for lossless, which is the largest reason I'd prefer JXL to win: one codec to rule them all sure is an alluring future.

> 2. Chroma subsampling remains a bad idea for still images unless the resolution is high enough to hide the artifacts.

Hmm, I don't think so. I think at a fixed file size, chroma subsampling usually allows you to have fewer noticeable artifacts. Humans are so much more sensitive to luma that it doesn't make sense to treat it equally to chroma with respect to lossy compression. That said, if you don't like it, AVIF supports 4:4:4 just fine.

In my tests, AVIF beats PNG easily for lossless compression of actual photographs (for things like charts and screenshots, PNG wins of course). And for lossy, it's much smaller than jpeg and supports HDR unlike WebP. So if you need HDR and are doing lossy compression on the web, it's your best option as far as I know.

> Hmm, I don't think so. I think at a fixed file size, chroma subsampling usually allows you to have fewer noticeable artifacts

At low bpp, certainly. Though "certainly" is to be quantified since chroma is quite cheap in AV1, thanks to CfL.

> Humans are so much more sensitive to luma that it doesn't make sense to treat it equally to chroma with respect to lossy compression

The problem is that this is completely dependent on material. Sharp and/or bright red is too common a killer sample (cf https://gitlab.com/AOMediaCodec/SVT-AV1/-/work_items/2211). Make sense for video where you'll have a hard time seeing it, but for still pictures it's too problematic to apply indiscriminately unless you're encoding at potato quality anyway.

> That said, if you don't like it, AVIF supports 4:4:4 just fine.

I know, but libaom is basically a reference codec, SVT-AV1 is the only "real" one we got and it doesn't =(

> In my tests, AVIF beats PNG easily for lossless compression of actual photographs

You're right, I wrongly put photographs aside where AVIF certainly is better. It did "okay" in my tests (NB: ImageMagick doesn't do "lossless" RGB AVIF even with `-quality 100` unless you add `-define heic:chroma=444 -define heic:cicp=1/13/0/1`; you can verify with `magick compare -metric AE ref.png out.avif /dev/null`).

> And for lossy, it's much smaller than jpeg

At decent quality, is it that much better than jpegli (https://opensource.googleblog.com/2024/04/introducing-jpegli...) or even mozjpeg ? If we add FGS to the equation, AVIF has the potential to be much better, though.

Some followup tests for lossless photo encoding

  $ real_time() { command time -p sh -c '{ "$@"; } 2>&3 >&2' argv0 "$@" 3>&2 2>&1 | sed -n 's#^real ##p'; }
  $ magick identify DSC00191.ppm
  DSC00191.ppm PPM 5492x3672 5492x3672+0+0 8-bit sRGB 57.6972MiB 0.090u 0:00.042
  $ for s in 0 3 6; do real_time magick DSC00191.ppm -quality 100 -define heic:chroma=444 -define heic:cicp=1/13/0/1 -define heic:speed=$s DSC00191_s$s.avif; done
  76.71
  12.80
  1.73
  $ real_time magick DSC00191.ppm -quality 0 DSC00191.jp2
  3.315
  $ real_time sh -c 'magick "$1" -define png:compression-level=0 "$2" && oxipng -q -o2 -s "$2"' argv0 DSC00191.{ppm,png}
  7.746
  $ for e in 3 6 9; do real_time magick DSC00191.ppm -quality 100 -define jxl:effort=$e DSC00191_e$e.jxl; done
  0.53
  1.33
  7.71
  $ for m in 0 3 6; do real_time magick DSC00191.ppm -quality 100 -define webp:lossless=true -define webp:method=$m DSC00191_m$m.webp; done
  5.86
  10.41
  46.03
  $ du -k DSC00191* | sort -n
  20068   DSC00191_e9.jxl
  20608   DSC00191_e6.jxl
  21324   DSC00191_m6.webp
  21340   DSC00191_m3.webp
  21772   DSC00191_e3.jxl
  23744   DSC00191.jp2
  23848   DSC00191_s0.avif
  23896   DSC00191_s3.avif
  24212   DSC00191_s6.avif
  24556   DSC00191.png
  26428   DSC00191_m0.webp
  59084   DSC00191.ppm

Not fabulous, really. If you've got a YUV source, everything changes, of course.

Damn, I didn't realize jxl was so great. I agree, AVIF lossless isn't competitive with it at all.

> At decent quality, is it that much better than jpegli

I was curious so I gave it a try and switched my photo editing site [0] to jpegli. Here's a comparison between a 29kb avif file (left) and a 146kb jpeg file (right), as produced by my site: https://files.catbox.moe/wdo9gf.png . The avif looks much better to my eye, and is of cource much smaller

[0]: https://pictolab.io

>but I just love HDR and AVIF is the only format that I feel like really takes it seriously.

JPEG XL would like a word.

He would like a word, but browsers said no?

Firefox is getting the flag to turn it on in the next release, chromium just added it back now there is a rust decoder. The wheels are turning again. Browser support for jpeg xl is very much in progress again.

And it is already present in Safari (and more generally in iOS and macOS, as part of the standard OS graphics library):

* https://developer.apple.com/documentation/avfoundation/avvid...

* https://petapixel.com/2024/09/18/why-apple-uses-jpeg-xl-in-t...

Do you think advanced users should consider AVIF/AVIF2 along JPEGXL for long term pictures archival ?

Which kind of encode settings do you suggest for conversion from high resolution RAWs or JPEGs ?

For long term archival, JXL is better, the only issue with it is browser and device support

Long term archival is often also about long term support and there just going with the most popular/supported ones might be a safer bet, eg in the extreme case if I wanted to save some digital photos in a time capsule I would likely choose PNG and JPEG

I have been using JXL for all my personal photos. My photo server Immich will just transcode a JPEG to display on devices which don't natively support JXL.

Depends on if you want lossy or lossless.

Honestly AVIF2 is the last thing we need now. There are way too many minority image formats already.

There aren’t _that_ many. HEIF has been an unusually large pain in the ass just because it’s both patent encumbered and incredibly popular since the iPhone and many cameras use it.

JPEG is woefully outdated with the lack of HDR and modern compression, HEIF can’t be used without paying a license, webp was designed just for extremely efficient small images rather than local storage, avif I’ve never seen used ever, and JPEG XL is on track to be the next major format.

I agree we don’t need an avif2, but until jpeg xl there really weren’t any decent alternatives for jpeg.

> There aren’t _that_ many.

https://en.wikipedia.org/wiki/Comparison_of_graphics_file_fo...

Almost all of these are historic relics of no relevance today. FLIF for example was a research project that was later the basis of JPEG-XL.

Pretty much only the ones I listed previously are serious contenders in the space right now.

It's a silly idea to not improve on things for something as "too many" formats. There's too many ARM processors out there, should companies stop development and keep using old, less efficient CPUs?

The market can handle one new format around every ten years. JPEG XL just came out so everybody should bide their time for a while instead of trying to immediately churn to a slightly better format.

Not always https://tonisagrista.com/blog/2023/jpegxl-vs-avif/

Maybe libavif will pick up AV2 support, so all one needs to do is update it to v2.0.0 or something and bam! Extra efficiency!

AV1 is being actively claim-charted by a lot of companies right now, and lawsuits are almost certainly coming. The same process is already starting for AV2, but most players are waiting for the AV1 cases to mature first.

People keep calling the AV-family codecs “royalty free,” but in practice they increasingly look like a legal and financial gamble.

People have been saying this for decades now.

I've never understood why some people seem to cheer this on like a corporation owning some maths was their local sports team.

For a while I assumed some people had put in a lot of effort on H.264 encoders and so the digital sharecroppers were angry and jealous that someone might be advocating for messy freedom.

But some people seem to just enjoy the thought of corporations putting a tax on video distribution.

Luckily those greedy corporations have repeatedly shot themselves on the foot and so their influence is waning.

How long has it been since AV1 was released? About eight years, and there's still no credible patent holder. The vultures are always circling around compression standards. You shouldn't take that too seriously. Even if a lawsuit is filed, there's a legal defence fund to protect against baseless claims.

Most importantly, there is no alternative. As kasabali said, the patent situation for the other codecs is a mess. Additionally, they could be hit by the same "No FRAND" problem. If someone comes forward with a patent that was used unintentionally, the situation is the same for all codecs.

> People keep calling the AV-family codecs “royalty free,” but in practice they increasingly look like a legal and financial gamble.

And the alternative is… ?

For H.265 there are two HEVC licensing pools you have to sign with plus at least two non-pool companies:

* https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding#P...

Going with a non-AVx codec is no less complicated and fraught with lawsuit risk AFAICT.

> in practice they increasingly look like a legal and financial gamble

As opposed to what, like HEVC? Where you need to pay 3 different patent pools to be sure (which all has different terms), then there's still other patent holders that aren't in any pools and come and hit you with loyalty requests any time under terms however they like to?

There should be a stronger push to fight these trolls. AOM has enough resources to bust their garbage patents and make them pay damages for patent racketeering. At least until software patents are banned for good.

It should be not possible to patent communication standards. The opportunity for abuse through lock-in effects is just too big.

Mostly a joke... I've been waiting for the AV1 Apple TV, so now I'm just waiting for AV2 support as Apple TV as well now.

My 10 year old iPhone 7 can play 1080p AV1 video in software for more than 200 minutes with VLC. The iPhone 7 was released a year and a half before AV1 was.

So I think it's a safe bet the current Apple TV devices are capable of playing AV1 video in software. There's a VLC release for Apple TV:

https://www.videolan.org/vlc/download-appletv.html

https://apps.apple.com/us/app/vlc-media-player/id650377962?p...

Not especially relevant, as the obvious use of AV1 on the AppleTV is streaming, and the OS frameworks don't request AV1 without hardware decoding. Services which provide their own video decoding (are there any?) don't seem interested providing their own software decoder for the ATV, despite the bandwidth savings.

Six years ago Apple TV added support for playing 4K YouTube video:

https://9to5mac.com/2020/06/22/tvos-14-brings-support-for-st...

YouTube's 4K videos are only available in VP9 and AV1.

So the YouTube app on tvOS has supported at least VP9 for six years and I wouldn't be surprised if it supports AV1 today.

Apple A17 Pro / A18 include AV1 hardware decode.

The latest Apple TV is on the A15:

https://en.wikipedia.org/wiki/Apple_TV_(device)#4K_(3rd_gene...

There may be a new Apple TV released this year.

> There may be a new Apple TV released this year.

The evergreen prediction in the Apple TV world :p. IIRC, Mark Gruman initially predicted the next model would be out H1 of 2024

Outside the apple ecosystem, AV1 is supported nearly everywhere.

They’ve had hardware decoding since M3 and equivalent A cpus. So I’d say it’s pretty well supported.

It works even without hardware decode. I noticed youtube is using AV1 on my M1.

AV1 already was a big leap toward efficient and open video formats. I'm awaiting AV2 since a long time.

Sure it'll take a while since it's implemented in chips and hardware so we got efficient and fast hardware encoding/decoding.

But a ~25% higher efficiency sounds very promising in times of increasing storage prices and chip crises.

Congrats!

How is the case of fighting off Dolby's patent racketeering going? They tried to attack Snapchat for using AV1.

Last update seems to be "lawsuit was filed" with zero updates since then. That stuff tends to move slowly.

Hopefully their patents will be busted and preferably Dolby will be also forced to pay damages for filing invalid lawsuits. That's the only way to teach patent trolls proper lessons.

While I agree with the general sentiment, I would not classify Dolby as a "patent troll" — they still do invest in research and developing products, right?

I'd classify any patent racketeer as a troll. Dolby has nothing to do with AV1, they just want a parasitic rent on it. Whether the troll actually does anything else besides the trolling part isn't really relevant.

Patent mob seems more fitting.

Another example is Qualcomm

How is Dolby a patent troll? :D

Because they have nothing to do with AV1, they simply want to leech on it. It's fitting to call them a patent troll.

they're a company that doesn't make any products. they just get actual hardware engineers to pay them to put a logo on their devices and to prevent Dolby from suing

Dolby's patents are garbage, and they know it. They managed to patent entropy coding somehow. This will not stand to challenges, and they managed to poke a well-resourced company.

This will always happen. There are just some entities that can't stand not seeking rent.

It may always happen but it would happen less if we updated patent laws to fine people who filed invalid patents or enforced some kind of similar punishment. If you file a patent, it's up to you to verify that your patent is actually valid, and the courts shouldn't have to do that legwork for you. It also doesn't help that the patent office/components of governments don't review patents as thoroughly as they used to. Same with trademarks.

I generally don't like the current patent law but it sounds a bit off to pay the government & wait for them to review your patent claim and then get fined by the government when both of you were wrong about it. There are already processes to additionally fine a company bringing about a truly frivolous patent lawsuit, it's just rare because usually it's not so cut and dry as we'd like it to be.

I mean my idea isn't the only one in that solution space. My reasoning was to ensure that the government actually reviewed the patent and ensured it was valid instead of rubber stamping it. Or, even better, the filer of the patent application would do that. Although the best is probably to make software unpatentable anyway.

Yeah, I wouldn't mind getting rid of software patents. Or, at least, making the patent reviews more rigorous before assignment.

Even better, software patents should not be allowed in the first place.

Why patents is a problem, please explain. If you build something that has been patented, then, well, you pay the per piece fee on it.

> In the 1980s, when IBM accused Sun of violating seven patents, Sun examined the patents and argued that IBM didn't have a case. The reply of IBM's lawyers was "maybe you don't infringe these seven patents. But we have 10,000 U.S. patents. Do you really want us to go back to Armonk [IBM headquarters in New York] and find seven patents you do infringe? Or do you want to make this easy and just pay us $20 million?" And Sun paid out.[4]

Big corpo shenanigans, from the 80s. Any recent quotes?

It's pretty much the modus operandi of any patent troll or in more general case - of any racketeer. Nothing new about it.

https://www.youtube.com/watch?v=Zz0_r4PDyCs&t=1m30s

That doesn't warrant a general ban on patents, just because some out of touch living in a fantasy reality kids have an erection saying this.

Ban on software patent is more than warranted, only trolls themselves don't like such idea obviously.

But what also needed is persecution these patent trolls for racket like any mob racketeers. There were attempts of persecuting them this way in the past, but they should be renewed.

I'm not a expert on video encoding

But i wonder if the future could depend less on fixed-function compression methods and more on AI networks that recreate the video but weight much less that a compressed video.

Neural codecs such as github.com/Orange-OpenSource/Cool-Chic

It will probably depend on whether NPUs are universally available in smartphones, and whether we get a standard API for accessing NPUs. But I don't know whether AI-based codecs can have battery usage competitive with fixed-function hardware.

Agreed, unless AI progress slows down enough that it becomes reasonable to bake weights into circuitry, the conventional approach will probably remain preferable for encoding, at least on power constrained devices.

Dav2d doesn't have the same nice ring to it. I hope there's someone with a decent repo-name punning skill who'll contribute before that.

avi2ude? av2go?

It was difficult to find a nice name, with av2 :(

It works in French d2vid (Deuvid)

Also in French, different similarity, av2il hav2e

Slightly worse in English, av2age mav2ick

At least it's not D4vd.

2av2quit?

I like it - not as punny as the first but pretty straightforward.

daviid - or trim to davii and pronounce it "davey". But tbh I quite like dav2d.

e2ed

And how long will it take before someone implements this standard and gets sued because Adobe or Dolby or whoever wanted to get slapped down? My knowledge may be out of date but if this is as "open" as AV1, I'm very skeptical that the individual companies will actually allow that. Greed and all that.

It took 7 years for the first patent assertion claim against AV1 to go to the courts and it will probably take a while for that case to resolve. Funnily enough, it wasn't from the pool constantly putting itself in the news about it over the years. I.e. it can take quite a while before attempts are made.

i'll be surprised if anything comes of it because like... https://aomedia.org/about/members/ if the legal departments of all these corporations haven't made them drop av1 in all these years there's probably not much there?

I'd like to think so as well (well, I don't know they'd necessarilymake them "drop" it as much as license those particular patents/accept the risk they may have to pay in the future) but on the other hand history has shown VP8 had a lot of companies behind it too before Google signed to sublicense patents from the MPEG LA & other holders for it.

VP8 was developed by a small company and bought and open sourced by Google on a fairly short timescale because the proprietary codec group had tried to start exerting their control.

So it's not accurate to say that had a lot of companies behind it. That usage came later and even then it was mostly in odd corners of the video industry.

Who originally developed the codec is only half the story. In 2011, 17 companies created a CCL agreement around VP8+WebM. In 2013 Google signed the sub-licensing agreement to keep it free. Any of the legal teams at those 17 companies had the chance to catch it before then. That there were a lot of legal team reviews by the big tech companies involved did not catch everything. The system isn't designed in a way to make that a solid guarantee like it should be.

> It took 7 years for the first patent assertion claim against AV1

This might just mean that if the claim is found valid, there's seven years' worth of inertia slowing down any effort to move on. Seven years in which HW and SW manufacturers worked to build in the support, and you the user developed your processes or workflows around assumptions specifically tied to that solution. I'd rather know on day one if I should go that way or not.

On that topic: https://arstechnica.com/gadgets/2026/03/av1s-open-royalty-fr...

Patent trolls are nasty. How long will it take for one to get the full support of those shaking the independence of the US judiciary for their own gain? Let's hope the rot stops before that.

No one can predict patent trolls - they can surface at any time randomly. But there should be a more organized fight against them. I assume AOM should be backing that.

Looking forward to a decently speedy encoder coming around. The reference one for AV1 is really not that great, and the same is true here. But as soon as we get SVT-AV2 or whatever, I'll be a very happy camper.

Now the real question: will The Industry™ again need for nerds and digital privateers to actually write and graft all the psy coding tools that makes their encoders usable (and I mean the word, using x264 as benchmark) for the quality-conscious and not just CPU-efficient VoD blurred to death with marketing yelling "PSNR! PSNR! PSNR!" at the top of their lungs? Will FGS be more usable?

Take a look at https://gitlab.com/AOMediaCodec/SVT-AV1/-/work_items/2269 for details (PS: SVT-AV1 claims to be suitable for AVIF yet doesn't support YUV444, lel)

anyone performed a encoding and decoding benchmarking with the reference codec yet? I'd expect encoding to be dreadful, but maybe the decode is already usable

It takes a few years for vendors to support hardware decoding for a new standard, so we won't see it in widespread use anytime soon.

I’m curious how much AV2 will actually help older hardware in practice.

I’m on a 2019 Intel MacBook Pro: 2.6 GHz 6-core i7, 64 GB RAM. The machine is still more than powerful enough for normal desktop work and software dev, but YouTube in Chrome has become borderline unusable for me. My internet is fine, Safari plays the same videos smoothly, and YouTube “Stats for nerds” shows plenty of buffer but the decoding makes youtube unusable in chrome for me.

I use Firefox but YouTube has recently started giving me a pop-up occasionally telling me that they are intentionally slowing down the site because they don't like some of the browser extensions I use.

Has nothing to do with video codec.

Download the video with yt-dlp & play it in mpv you'll see it even flies on a potato.

Play the same in browser and it'll be dropping frames left and right.

Sound like a Chrome/Youtube problem. My 2012 Macbook Pro plays 1080p AV1 just fine in VLC (pretty sure Youtube works fine too in Firefox, but I didn't check whether or not it was AV1 or H264).

For reference: dav1d 0.5 can decode 143 FPS of a 1080p 8-bit video on a third gen core i7.[1] I doubt there's been much in the way of regressions since then. 10-bit and 4k is obviously a lot more heavy, but not really relevant to older devices.

[1] https://www.phoronix.com/news/dav1d-0.5 (it's mislabled as a core i3, but the 3770K is a core i7).

use the enhanced h264-ify to block the av1 stream, av1 takes a lot of cpu

Unfortunately for you, newer codecs use more CPU than older ones so AV2 would probably be even worse.

Newer codecs has generally introduced more ways a video can be encoded, so that the encoder need to work much harder to encode a video, so that it actually achieve the gains that the newer codec allows much more processing will be required. Decoding on the other hand, will mostly stay the same or increase only slightly. It's not likely do decrease though, so if you struggle playing av1 today, you will also struggle with av2.

For encoding, you can always write a simple encoder that use only the features that were present in mpeg2, and it will be about as efficient as mpeg2 as well. Newer codecs has more features that allows more efficient encoding, at the cost of more processing.

I gave up using Chrome a decade ago. It’s a power sucking pig. Safari has its own issues, but at least it’s usable. When I need something that isn’t Safari, I use Firefox.

Chrome has used less power than Safari for a few years now. As far as I know, the only good reason to use Safari is to use FairPlay DRM for sites that need it. For all other purposes, Firefox or Chrome is superior. https://birchtree.me/blog/everyone-says-chrome-devastates-ma...

Interesting, thanks. I’ll give it another try. It certainly used to be a pig. I remember times when the fan on my MacBook sounded like a freaking hovercraft and you’d go into system monitor and see a Chrome process just sucking down every CPU cycle. You’d quit Chrome and the fans would spin down and stop. But if they’ve fixed that, maybe it’s worth another go.

Safari has a terrible developer experience and has been behind in implementing the various browser API for years, including AV1 support.

Safari has had AV1 support for a long time. Absolute majority of "various browser API" that they don't support are all kinds of "Web Bluetooth"-like crap that I don't need and which only introduces attack and tracking surface when supported.

Like I said, Safari has its own issues. But huge power consumption isn’t one of them.

Is there an AV1 2.0? I'm not using this codec if they can't do basic semantic versioning right.

Codecs are final (the only exception being H.265 with a couple of later additions/extensions that required new codec IP and were not that widely deployed or supported). They are not software. You cannot have upgrades for fixed function HW decoders (which is a must for pretty much any established codec and AV2 will be also one of them).

Neat but the real world still, and for decades to come - lives on h264 https://www.wink.co/documentation/Why-H264-Is-Almost-Always-...

Did you read what you shared?

It’s talking about security cameras, low-bitrate video.

FTA: "Myth Busted: At security camera bitrates (400-800 Kbps), H.265 provides negligible compression benefits. The marketing claims of "50% savings" apply only to high-bitrate content like 4K movies at 25+ Mbps, not security cameras."

That's also bullshit anyway. Not surprising because the article is slop.