Announcement

Collapse
No announcement yet.

AMD Releases Brotli-G For GPU-Accelerated Brotli Compression

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #21
    Originally posted by bug77 View Post

    If you use brotli with big files, it compresses less than gzip.
    Parallelism may be a thing, but I'm not sure how that would work. Clouds do not rent GPU for web traffic compression offloading, afaik. Maybe for custom deployments?
    Like I explained, afaik all data that goes through the web is split into chunks so if something gets lost, you only have to resend those chunks instead of the whole file. So brotli combined with http won't need to compress big files, only lots of chunks of bigger files. And as the article states, it's not GPU only. But if a GPU could free up many CPUs, cloud and CDN providers might switch to just using GPU acceleration. And maybe AMD is even working on accelerating more stuff through GPUs so it might make more sense in the future. After all, they probably wouldn't mind if they could sell their GPUs in the server market too.

    Comment


    • #22
      Originally posted by bug77 View Post

      If you use brotli with big files, it compresses less than gzip.
      Parallelism may be a thing, but I'm not sure how that would work. Clouds do not rent GPU for web traffic compression offloading, afaik. Maybe for custom deployments?
      Ok, I'm at an even bigger loss now. After reading their description on GitHub and reading their article https://gpuopen.com/brotli-g-sdk-announce/ it seems the GPU implementation isn't meant for compressing (at least not mainly), but for decompression. They want Windows users to use a DirectX Shader for parallelized decompression. Sure, when loading why web page using brotli will result in many brotli compressed packages that need to be decompressed in order to display the page, but why build it for Windows then? Those systems usually have enough horse power to do it the old fashion way. I would rather see an implementation for Android and iOS be more helpful. After all, they are on a much stricter power budget. So this makes just no sense at all. And who in the Windows world is supposed to implement that? That would need to be browsers, but I don't see any implementing it. Not when it doesn't work in all platforms.

      Comment


      • #23
        Originally posted by Artim View Post
        cloud and CDN providers might switch to just using GPU acceleration
        Nah, they just store the file compressed in the first place, so they don´t have to compress on-the-fly

        B.t.w. this is a completly "new" format from a practical point of view as:
        One thing for developers to note is that assets that have already been compressed with Brotli cannot be decompressed with Brotli-G decompressor implementations. To take advantage of Brotli-G, assets must be recompressed using a Brotli-G compatible compressor (also supplied in the SDK).
        This essentially means, that one has to introduce a new format in the HTTP header like Content-Encoding: br-g
        as you can´t pass a brotli compresse bitstream to the Brotli-G decoder.. So essentially the current ecosystem is almost 0.
        Last edited by Spacefish; 22 November 2022, 06:36 PM.

        Comment


        • #24
          To me it looks like this is targeted at Windows' DirectStorage feature meant for faster decompression of assets on GPUs, hence the HLSL usage.

          Right now, that uses "GDeflate" developed by nVidia.
          This is AMD's answer to that.

          Care to chime in, bridgman?

          Comment


          • #25
            Originally posted by bug77 View Post
            I thought brotli was designed to be easy on hardware (being able to run on phones and such). What am I missing here? Why does it need to run on GPUs now?
            brotli is just a compression algo, it can be used on whatever you really want it to. there is a brotli tool that can be used to make brotli compressed files. if you have run across .tar.br files, those are brotli files (pretty rare though)

            this is designed for various assets, but this should work for anything that needs to be utilized on the gpu, we need to compress resouces going to gpus, because the stuff we punt onto the gpus is far out pacing the amount of vram that increases as time goes on.

            compression is necessary for this, but since you are under the assumption that decompression needs to happen real time, and that this will also that this will be used in gpu intense scenarios, so you want the performance impact to be as little as possible, brotli is easily the best choice for this.

            this could be used for direct storage, but its not necessarily only useful for that, this could be useful for compute, this could be useful to trim that 8.5gb of vram usage to 8. but more importantly, we don't really have a good way to crossplatform supported, losslessly compress assets, as far as I know, I'm not sure if we even have any good lossless texture formats for gpu.

            Comment


            • #26
              Originally posted by Spacefish View Post
              Nah, they just store the file compressed in the first place, so they don´t have to compress on-the-fly

              B.t.w. this is a completly "new" format from a practical point of view as:


              This essentially means, that one has to introduce a new format in the HTTP header like Content-Encoding: br-g
              as you can´t pass a brotli compresse bitstream to the Brotli-G decoder.. So essentially the current ecosystem is almost 0.
              The question merely is if you need to indicate the format or if the decompressors are able to switch on demand between brotli and brotli-g when the latter fails.

              Comment


              • #27
                Originally posted by bug77 View Post

                What would you keep in VRAM that would benefit from brotli? Brotli is good with small files, it sucks with something as large as a texture. Maybe this is aimed at some sort of compute scenario?
                Game assets. Textures, and anything else you are sending to vram would still benefit.

                Maybe there are gpgpu scenarios where the dataset is highly crompressible (where saving pcie bandwidth savings are enough to offset the additional compute overhead).
                Last edited by brucethemoose; 23 November 2022, 01:37 PM.

                Comment


                • #28
                  Michael "just not AMD Radeon graphics products" you probably meant "not just AMD Radeon graphics products". Has a different meaning

                  Comment


                  • #29
                    Nvidia has it it's own library for GPU accelerated compression for deflate, LZ4, zstd, etc.. The latest versions are not open source thought.
                    https://github.com/NVIDIA/nvcomp

                    Comment


                    • #30
                      Originally posted by abu_shawarib View Post
                      Nvidia has it it's own library for GPU accelerated compression for deflate, LZ4, zstd, etc.. The latest versions are not open source thought.
                      https://github.com/NVIDIA/nvcomp
                      while gpu accelerated lz4 encode and decode could be highly interesting, unless it's in a cross platform ecosystems entirely useless to me sadly

                      Comment

                      Working...
                      X