OpenZFS 2.3-rc3 Adds JSON Output For Commonly Used Commands

[mobadboy]

one step closer to deprecating btrfs

[mb_q]

There are some people who would argue that that is quite a radical position, given the history of Unix, and the general habit of piping the output of one program into another for further processing.

Sure, I use coreutils and pipes all the time, but this is ad hoc processing, where you just want the answer for a particular case. I have a problem with using tools created for ad hoc work in systems where side cases matter, like elaborate management or monitoring tools, because it ends badly; consider autotools, a simple framework for using built-in tools to just install stuff: zero functional C but dozens of "compile this to ensure that gcc is gcc".

IMHO JSON is not helpful in ad hoc work, but encourages those dubious integrations.

[mb_q]

Those claims are not true. If your data doesn't contain characters outside the ascii domain, json output can be pure 7-bit ascii. Also there are schema systems built on top of json. E.g. if you define the schema in high level typed language you can derive the serialization / deserialization process. Json serialization also doesn't require a full library. You can build a special purpose json serialization for a cmd line tool with just printf and some juggling.

If I understand correctly, you want to put invalid UTF-8 in JSON string by escaping it? But this is not going to be a valid JSON. Some parsers will eat it, some won't, and some will do strange things. I know there are schema systems for JSON, the problem is that they are useless without an ecosystem, i.e., one Linux or POSIX standard providing one schema in a particular schema system for all system utils for follow -- otherwise you have to adapt the parser to the tool anyway, so there is no gain from the fact that the output is JSON.

[caligula]

Sigh. Fundamentally, all data can be represented by binary. Everything else is just a level of abstraction built on top, with compromises built in to the abstraction method.
The point about UTF-8 is that no-one is seriously proposing a new byte-level representation of character sets. You can point to problems, but the world has voted that it is good enough, bar a few exceptions, such as personal names written in obsolete Chinese characters, or artificially created character sets for academic/hobby-project languages.

Ok, the thing I was referring to was 'it just doesn't support integers and binary data (you cannot even represent arbitrary Linux path in JSON due to the fact that JSON requires UTF-8 and Linux doesn't) and there is no standard schema system and often no schema whatsoever, so the conversion still has to be manual' (mb_q's comment). Some consider UTF-8 as a problem if they want arbitrary binary output. For instance file names can be non-conformant random binary sequences.

The fact that people find the need to write articles like this .. tells you that being forced to do type conversion to use a data interchange format is insane. A standard data-interchange format needs a far richer set of representations of numeric data than JSON currently has, together with a way of defining exceptions/additions where needed.

Aye, that's bad. However json is quite often used even if numbers need to be reinterpreted or rounded.

JSON is just one of many serialized data-interchange formats (Wikipedia: Comparison of data-serialization formats) - and the mere fact that there is a plethora tells you that JSON is not the agreed 'standard' method. It can be improved upon, and really ought to be.

Yes but that page doesn't cover all the issues. E.g. XML based ones are often considered quite bloated. I haven't studied this topic recently, but for example Amazon Ion looks quite interesting and if it's a superset of JSON, it could act as an upgrade path now that json has become pretty ubiquitous.

[Old Grouch]

Those claims are not true. If your data doesn't contain characters outside the ascii domain, json output can be pure 7-bit ascii. Also there are schema systems built on top of json. E.g. if you define the schema in high level typed language you can derive the serialization / deserialization process. Json serialization also doesn't require a full library. You can build a special purpose json serialization for a cmd line tool with just printf and some juggling.

Sigh. Fundamentally, all data can be represented by binary*. Everything else is just a level of abstraction built on top, with compromises built in to the abstraction method.
The point about UTF-8 is that no-one is seriously proposing a new byte-level representation of character sets. You can point to problems, but the world has voted that it is good enough, bar a few exceptions, such as personal names written in obsolete Chinese characters, or artificially created character sets for academic/hobby-project languages.

As for JSON, it does not have distinct types for integers and floating-point values, which is less than useful. Naturally, one can remedy this by designing a higher-level schema that uses workarounds to represent different types in a JSON serialized way, but you only need to look at the differing implementations, and questions on how to achieve what 'ought' to be trivial tasks to realize that JSON + schema is insufficiently standard, or understood, even now.

The fact that people find the need to write articles like this:

https://blog.phakorn.com/jsons-numeric-boundaries-the-lesser-known-reality-of-inaccurate-figures

tells you that being forced to do type conversion to use a data interchange format is insane. A standard data-interchange format needs a far richer set of representations of numeric data than JSON currently has, together with a way of defining exceptions/additions where needed.

JSON is just one of many serialized data-interchange formats (Wikipedia: Comparison of data-serialization formats) - and the mere fact that there is a plethora tells you that JSON is not the agreed 'standard' method. It can be improved upon, and really ought to be.


*To be fair, one can argue that Alonzo Church's lambda-calculus represents numbers in unary. People in general tend to be more familiar with binary, and Turing machines (which, ironically, often represent numbers in unary in examples), rather than with lambda-calculus.

[caligula]

Personally, I think it would be useful to have tools be able to provide output in a well-formed canonical output for use in subsequent processing. Having pretty much agreed on the use of UTF-8, it would be nice if we could agree on a workable (serialised) data representation. JSON is not it, for the reasons others have pointed out.

Those claims are not true. If your data doesn't contain characters outside the ascii domain, json output can be pure 7-bit ascii. Also there are schema systems built on top of json. E.g. if you define the schema in high level typed language you can derive the serialization / deserialization process. Json serialization also doesn't require a full library. You can build a special purpose json serialization for a cmd line tool with just printf and some juggling.

[Old Grouch]

CLI tools should not enforce bad behaviour of using them from other programs.

There are some people who would argue that that is quite a radical position, given the history of Unix, and the general habit of piping the output of one program into another for further processing.

Personally, I think it would be useful to have tools be able to provide output in a well-formed canonical output for use in subsequent processing. Having pretty much agreed on the use of UTF-8, it would be nice if we could agree on a workable (serialised) data representation. JSON is not it, for the reasons others have pointed out.

I'm certainly not against the use of 'other methods'. Being able to use generic standard APIs for CLI tools would be excellent.

[mb_q]

Given those apparent failings of JSON, and in the absence of OS APIs, what other data representation would you suggest?

There are API, sys tree, syscalls & ioctls; most tools that you want to produce JSON have library versions that wrap them. Or you can use systemd stack and its DBUS controls. CLI tools should not enforce bad behaviour of using them from other programs.

[Old Grouch]

Yeah, it just doesn't support integers and binary data (you cannot even represent arbitrary Linux path in JSON due to the fact that JSON requires UTF-8 and Linux doesn't) and there is no standard schema system and often no schema whatsoever, so the conversion still has to be manual. The robust solution for programmable administration is to use OS APIs in you language, not to autodrive CLI tools.

Given those apparent failings of JSON, and in the absence of OS APIs, what other data representation would you suggest?

[caligula]

Or do you mean that it's funny that this didn't happen before JSON even though there have been other easy to parse formats before?

Yes. E.g. try to create csv reports with powertop. This is a great example of unix crap.