9 Years After Starting, AppStream 1.0 Is Coming For Cross-Distribution Package Metadata

[phoronix]

Phoronix: 9 Years After Starting, AppStream 1.0 Is Coming For Cross-Distribution Package Metadata

AppStream was started in 2011 as a means of drawing up cross-distribution (XML-based) standards for describing software components/packages metadata and for repositories to describe software collections. Now nearly a decade later, AppStream 1.0 should be coming in the next few months...

http://www.phoronix.com/scan.php?page=news_item&px=AppStream-1.0-Coming

[the_scx]

It is worth mentioning that Flatpak doesn't use AppStream at all. It uses fork of the AppStream-Glib, which is not fully compatible with AppStream.

Currently, we have 3 "standards":

• appstream by ximion - Reference implementation maintained by Debian developer, barely adopted. To validate, use this: appstreamcli validate --pedantic com.website.AppName.appdata.xml or appstreamcli validate com.website.AppName.appdata.xml
• appstream-glib by hughsie - Glib (*coughs* GNOME *coughs*) variant, used by GNOME and Red Hat, and adopted by others as well. To validate, use this: appstream-util validate-strict com.website.AppName.appdata.xml or appstream-util validate com.website.AppName.appdata.xml or appstream-util validate-relax com.website.AppName.appdata.xml (widely used by EL and Fedora)
• org.freedesktop.appstream-glib by Flathub members - fork of appstream-glib, with relaxed style rules, but with additional requirements, like OARS. To validate, use this: flatpak run org.freedesktop.appstream-glib validate-strict com.website.AppName.appdata.xml or flatpak run org.freedesktop.appstream-glib validate com.website.AppName.appdata.xml (this is currently required on Flathub) or flatpak run org.freedesktop.appstream-glib validate-relax com.website.AppName.appdata.xml

This is similar to Office Open XML. We have the ISO standard and the MS implementation that deviates from it.
Anyway, both appstream-glib and org.freedesktop.appstream-glib (technically speaking, it is the flatpak package for appstream-glib, with additional patches) are completely undocumented (I mean changes in relation to the appstream). What is worse, their rules changes several times a year, which really pisses off developers. What's worse, appstream-glib is not fully portable across Linux distribution, or rather the same AppData file can be valid on one system and incorrect on another, even if they use the same version of appstream-glib, which by the way is a rare situation.

[nanonyme]

On the nice side, the last one *will* work the same everywhere.

[the_scx]

On the nice side, the last one *will* work the same everywhere.

Well, it is still possible to create an AppData file which will be valid for appstream and invalid for appstream-glib/org.freedesktop.appstream-glib.
I can also provide a valid file for org.freedesktop.appstream-glib, which is invalid for appstream-glib, or vice versa.
The AppStream 1.0 release does not change anything. The differences between the three "standards" are still visible.

[caligula]

Nice, they've adopted XML when everyone else switched to JSON.

[discordian]

Nice, they've adopted XML when everyone else switched to JSON.

Or YAML or TOML. If you pick any of those, some people will be happy, others angry.
The correct way is to use XML so everyone will hate you

[skeevy420]

Or YAML or TOML. If you pick any of those, some people will be happy, others angry.
The correct way is to use XML so everyone will hate you

There's an INI joke in there somewhere.

[Citan]

Nice, they've adopted XML when everyone else switched to JSON.

Indeed, it's nice to see some people are still attached to using robust and certifiable formats in an era of so many lazy developers going for "straight flow of whatnot mixed junk" data...

[Ximion]

Hi!

[...]
Currently, we have 3 "standards":

• appstream by ximion - Reference implementation maintained by Debian developer, barely adopted. To validate, use this: appstreamcli validate --pedantic com.website.AppName.appdata.xml or appstreamcli validate com.website.AppName.appdata.xml

"Barely adopted" depends on your point of view. The AppStream reference implementation is used by default in all of KDE, by Elementary Linux for the AppCenter, by Debian, Ubuntu and Arch Linux to generate, validate and maintain their metadata and at Purism for metadata generation and online presentation. That's actually quite a lot of users, and there are possibly more that I currently don't recall. Not to mention projects using the tools directly to maintain their metainfo files.

• appstream-glib by hughsie - Glib (*coughs* GNOME *coughs*) variant, used by GNOME and Red Hat, and adopted by others as well. To validate, use this: appstream-util validate-strict com.website.AppName.appdata.xml or appstream-util validate com.website.AppName.appdata.xml or appstream-util validate-relax com.website.AppName.appdata.xml (widely used by EL and Fedora)

It's unfortunate that there are two implementations, but Richard and I talk regularly and we worked on quite a few projects together. In the long run we maybe will be able to unify the projects, but since they have very different internals that's much harder than it might look, especially now that existing stuff depends on it. Both implementations parse AppStream properly though and are standard-compliant. AppStream-GLib will permit a few non-standard things that shouldn't be used and its validator is more brutal on styling checks (as it was basically meant to check for "looks good in GNOME Software", while AppStream itself checks for "is it standard compliant?" and then might give info-hints to improve the looks of app metadata), but that's about it.

• org.freedesktop.appstream-glib by Flathub members - fork of appstream-glib, with relaxed style rules, but with additional requirements, like OARS. To validate, use this: flatpak run org.freedesktop.appstream-glib validate-strict com.website.AppName.appdata.xml or flatpak run org.freedesktop.appstream-glib validate com.website.AppName.appdata.xml (this is currently required on Flathub) or flatpak run org.freedesktop.appstream-glib validate-relax com.website.AppName.appdata.xml

This is a thing I wasn't aware of... When validating with `appstreamcli`, there is unofficial support for modifying the relevance of certain hints to customize the result (e.g. some projects may want to make certain style checks that aren't fatal fatal). That could maybe help here (I will have a closer look at the actual divergences, it's likely not an issue - a fork sounds a bit drastic).

This is similar to Office Open XML. We have the ISO standard and the MS implementation that deviates from it.
Anyway, both appstream-glib and org.freedesktop.appstream-glib (technically speaking, it is the flatpak package for appstream-glib, with additional patches) are completely undocumented (I mean changes in relation to the appstream). What is worse, their rules changes several times a year, which really pisses off developers. What's worse, appstream-glib is not fully portable across Linux distribution, or rather the same AppData file can be valid on one system and incorrect on another, even if they use the same version of appstream-glib, which by the way is a rare situation.

The latter part sounds like unintentional behavior in AppStream-GLib, maybe it's worth reporting a bug for that? In any case, the situation isn't comparable with OOXML, as both implementations will read the same stuff and interpret it the same way (if something doesn't behave spec compliant, it's a bug that will be fixed). They do make different requirements for validation though, which is due to the different scope of the projects. As per specification, an AppStream Metainfo file without a content_rating is perfectly compliant, so `appstreamcli` will pass the validation. If the validation tool has additional agenda though, e.g. having requirements for Flathub, they may make that tag mandatory and fail validation if it isn't present.

This is precisely the reason why `appstreamcli` has an issue severity system for its validator hints, to make it a bit less opinionated: Errors and Warnings fail the validation for serious violations of the specification, and other issue severities are hints for the developer to help them improve their metadata and are not fatal by default. That includes e.g. style checks like summary-ends-with-dot.

I hope that clarifies a few things :-)

[Ximion]

Or YAML or TOML. If you pick any of those, some people will be happy, others angry.
The correct way is to use XML so everyone will hate you

Hehe ^^ I wasn't the one who picked XML originally, but it definitely is a solid choice for the purpose of AppStream, as it is translatable with existing tools and can be extended without breaking backwards compatibility far easier than a format like YAML or JSON. AppStream also is pretty straightforward XML, not the insanity that you can create with it.
There is no objectively best markup format, there's always tradeoffs involved which make a format more or less suitable for a particular task. And for XML: You don't have to love it to acknowledge that it is useful sometimes ;-)