TGSI On-Disk Shader Cache For Mesa: Caching Comes To R600g/RadeonSI

[Hi-Angel]

There have been other structural and implementation changes based on feedback and now the TGSI stuff, also some of the cleanup patches have already been upstreamed

V7:
- support for SSO added.
- rebased/reworked on top of my work to rework the gl structs, many previous
hack and work arounds removed. Fallback support is now much cleaner and
less hacky.
- Added fallback support for programs that use FragData bindings.
- Fixed off by 1 bug with attribute binding fallback as the
string_to_uint_map hash table offsets everything by 1 which we didn't allow
for.
- Lots of other clean-ups and improvements.

V6:
- rewrote a bunch of the i965 shader cache functions to reduce
duplication.
- fixed some i965 bugs that only happen when falling back to a full
recompile when a cache object is deemed invalid (created with different
mesa version).
- added env var to more easily debug bugs as described in previous dot
point.
- Store all pointers as uint64_t.

V5:
- added support for caching shaders with transform feedback
enabled by the in shader xfb qualifiers now ARB_enhanced_layouts
is enabled.
- rebased on last 3 months of Mesa changes

V4:
- lots of reworking patches to remove code churn should be much nicer now
- fixed fallback when shader has been detached
- fixed a couple of bugs with UBOs
- no more printfs, debug info is behind an environment var
- various cleanups, tweaks and fixes

V3:
- add support for geometry and tessellation stages
- cache clip planes
- reserve parameter storage before restoring list
- stop losing buffer blocks on cache fallback
- lots of little fixes I cant remember

V2:
- rebased on master
- add support for encoding doubles
- renamed skip_cache params to is_cache_fallback, and fix related bug
when disabling shader cache for xfb.

I see. But isn't it better to merge the work, and improve it afterwards? E.g. in the case of cache it'd be the same 7 revisions, except that by virtue of having the code in master rebasing would be unneeded, and also other devs could occasionally improve something whilst working on irrelevant things.

[FireBurn]

I see. But isn't it better to merge the work, and improve it afterwards? E.g. in the case of cache it'd be the same 7 revisions, except that by virtue of having the code in master rebasing would be unneeded, and also other devs could occasionally improve something whilst working on irrelevant things.

That's up to the maintainers, but seeing how invasive this is, they might be taking a softly softly approach. Also it still hasn't been reviewed yet. If an earlier version had been reviewed entirely - it might have happened that way

[Adarion]

Can't wait for it to hit official MESA. That will probably be a nie enhancement towards user experience. Maybe the lack of this is why I sometimes have so long loading times, but the HDD LED is hardly showing activity.

[boxie]

If I understood it correctly, he was talking about his TGSI cache implementation, and the IR representation still needs to be compiled to hardware specific binaries. So you get rid of actually parsing the shader code and the "top layer" of the shader compilation (to IR), but the lower part still has to be done.
I guess you can compare it to SPIRV->hardware specific binaries, the most low-level stuff needs compiling, but you get rid of most of the "text parsing".

When the RadeonSI specific back-end is merged, the driver can load those binaries directly into memory (based on some checksum I assume, for security and stability), and the rest of the compilation overhead goes away (apart from having to read data from disk).

Correct me if I'm wrong.

seems plausible