Recently I found out that if I set VLC to decode using "VA-API video decoder via DRM" it wouldn't work, while the one with "via X11"...