Apple Designs New File-System To Succeed HFS+

[liam]

The posix-compliant-ness affects the userspace interface of course.
Being posix-compliant for a filesystem means supporting a bunch of features needed by posix programs to work on it.
The only one that I remember atm is renaming or deleting an open file. On ext4 I can do it all the time and nothing breaks, on NTFS (on windows) nope.
Look up on google, there are more requirements than this.

Yup, no arguments here.

As for the "what interfaces the kernel provides", consider that btrfs is also pretty self-contained (does not rely on mdadm, lvm, and company), it asks only the most basic kernel interfaces.
This makes porting it in the same ballpark than porting ZFS on linux. Annoying but NOT ANYWHERE NEAR making something similar from scratch.

I'm not certain about this. You might be right, but I'm genuinely not sure.
IMHO, the massive layer violating nature of btrfs means that it has to tailor itself to linux-isms, rather than the well defined (even if not unchanging) system internal interfaces.
ZFS, aiui, was a very special case. It was written with a compatibility layer (http://open-zfs.org/wiki/Platform_co...0on.C2.A0Linux) for, at least, a large amount of its code. I THINK this is why zfs, on linux, doesn't have a reclaimable page cache.
A better example would be to look at the recent, partial, btrfs port to the window's environment. I haven't looked at that, at all, but how they managed that would be very demonstrative of the difficulties.

On FreeNAS forums the standard is "don't go lower than 8GB as it can be hazardous" (this on a Unix system), better if more. And this ram isn't cache. It's used by the filesystem. Also in the official site http://www.freenas.org/hardware-requirements/

Also, unless you have at least a 2-disk array you aren't using most of ZFS features (checksums can detect corruption but there isn't any redundancy to fix it), this is true also for btrfs, although with btrfs there is the "make a raid1 inside the same partition" option, which is a bit masochistic but would work.

I don't know what is happening in freenas-land b/c according to http://www.freenas.org/hardware-requirements/ the 32 bit version of freenas that used UFS had a 4GB requirement. For a system that is supposed to be a well tuned dedicated file server, those requirements seem insane.
As for making best use of zfs, I won't argue any of that. Btrfs, however, can save multiple copies of a data block on single device (using -d dup when making the fs).

Current CoW filesystems are not suited for anything that isn't a server, ok with that.

For "embedded" we need to make a difference, mobile devices are increasingly getting in collision with PC userbase (also for Apple, remember Cook saying their iPad Pro shoud replace PCs?), so in a near-ish future it might.

Although the trend I'm seeing is more Google-ish, "providing devices with low storage and offering large cloud storage services", which isn't even bad (if you have a decent Internet connection), as handling RAIDs, making backups, and doing other kinds of shit is beyond the average user's skill level anyway.
Data in the cloud is much less hard to lose from the common accidents that I see people lose data from (malware, hardware failure, idiots installing Linux wrong).

Yeah, I was deliberating whether to specify "embedded mobile".
The ipad pro, and the various surface products, are good examples of convergent devices (the surface, though, is REALLY just a laptop with a detachable keyboard). I have to imagine that apfs (or whatever it's called) had something like the ipad pro in mind. Smaller devices, like phones, especially iphones, have relatively simple requirements, but once you start using heavy local apps with multitasking and large working sets, you start to need a more expansive set of features.
I also agree with your assessment of our cloud-based future. That's largely why I think that btrfs is simply here too late. The future,imho, are these clustering filesystems (ceph, iirc, has even expressed scalability concerns with the vfs/io interface) which store/distribute/checksum our data in the cloud, and scale across datacenters.


EDIT:
I just read your post above. I'm surprised they only started this in 2014, but zfs, with its far larger feature set, took only a year or two longer.
I suppose this shows that if you have a DEDICATED (only working on this on thing) group (some number between two and much more, but small enough that everyone knows what everyone else is doing) of developers, you can make something as subtle as a new, non-trivial, fs in pretty short order.
I still hope something comes of bcachefs

[starshipeleven]

If you're looking for a little more detail on APFS - http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/

Thanks.

a quick tl;dr for others (the stuff in "" is quoted directly):

-they claim it's from scratch, and it started in 2014, given the following featureset I'd say it's ok.
-encryption
-btrfs-like snapshots
-btrfs-like volumes with their own encryption key and policies and stuff, this is probably useful for app sandboxing
-hardlinks-like deduplication (copy one file and you are just copying metadata, not physical data, you are basically making links to the same data blocks), a feature they themselves admit to not have a specific use for. Making hardlinks instead of copies works in any decent filesystem since a while ago tho. The fact that this works only if done by file manager but not by commandline cp tells me the logic for this is not implemented at filesystem level, akin to most NTFS features.
-designed with flash logical structure in mind (akin to f2fs), of course not suited for raw flash.
-TRIM support (no duh)
-latency QoS, some kind of logic that prioritizes user applications over background applications
-no RAID support whatsoever in the filesystem itself, you must rely on hardware raid or whatever you were using until now (easymode development turned ON, understandable though, they don't have any RAID in their main products anyway)
-"novel copy-on-write metadata scheme" (unknown), "APFS does not employ the ZFS mechanism of copying all metadata above changed user data which allows for a single, atomic update of the file system structure" (don't know enough to say if this is good or not)
-there is a filesystem checker for apfs
-no checksum whatsoever, (easymode development over 9000) "The APFS engineers I talked to cited strong ECC protection within Apple storage devices." (ECC checks happen all day and all the night in any decent disk controller since the dawn of time, dafuq is "strong"? Delegating again to hardware something that can't be dealt with by software?), "Apple engineers I spoke with claimed that bit rot was not a problem for users of their devices," (Breaking news: it's not for the average Windows or even linux user either)


Heh, a modern filesystem for consumer devices, cool for its niche, but the lack of checksumming and RAID makes it completely unworthy of going beyond single-disk PCs and mobile segment.

btrfs is still our lord and saviour. ALL HAIL BTRFS!!!!

[ctwise]

If you're looking for a little more detail on APFS - http://dtrace.org/blogs/ahl/2016/06/19/apfs-part1/

[starshipeleven]

Claiming that Canonical is a Microsoft "sockpuppet" and calling Shuttleworth a "sex slave" isn't "pretty much just stating facts".
It's slander.

You probably missed the obvious sarcastic tone in "(none in his right mind would sue Canonical, because they are the sockpuppet of MS... ah crap I said it, I wanted to say because they have never made a dime off linux, so apart from getting Shuttleworth as a sex slave they cannot really offer anyway in case they lost)"

I said that Canonical hasn't made a dime off Ubuntu so they have nothing to offer apart Shuttleworth's service as a sex slave (something sounding like an obvious exaggeration).

I did not call Shuttleworth a sex slave. Stop drug abuse.

I did joke about Canonical being a sockpuppet of MS. Canonical does indeed channel The True MS Way(tm) in many things, and I'm not the first noticing this.

[starshipeleven]

I've been looking at the kernel interface for module development since that's what btrfs would be using. I don't think that posix says anything about what internal interface the kernel uses, but rather what interfaces the kernel's services provide. So, being posix compliant might not mean much (anything?) when the question is "how easy would it be to port this kernel feature from one posix-y kernel to another?".

The posix-compliant-ness affects the userspace interface of course.
Being posix-compliant for a filesystem means supporting a bunch of features needed by posix programs to work on it.
The only one that I remember atm is renaming or deleting an open file. On ext4 I can do it all the time and nothing breaks, on NTFS (on windows) nope.
Look up on google, there are more requirements than this.

As for the "what interfaces the kernel provides", consider that btrfs is also pretty self-contained (does not rely on mdadm, lvm, and company), it asks only the most basic kernel interfaces.
This makes porting it in the same ballpark than porting ZFS on linux. Annoying but NOT ANYWHERE NEAR making something similar from scratch.

From all accounts, zfs only needs a lot of ram if you are using its dedupe feature.

On FreeNAS forums the standard is "don't go lower than 8GB as it can be hazardous" (this on a Unix system), better if more. And this ram isn't cache. It's used by the filesystem. Also in the official site http://www.freenas.org/hardware-requirements/

Also, unless you have at least a 2-disk array you aren't using most of ZFS features (checksums can detect corruption but there isn't any redundancy to fix it), this is true also for btrfs, although with btrfs there is the "make a raid1 inside the same partition" option, which is a bit masochistic but would work.

Generally speaking, I think a cow fs isn't well suited to embedded environments.

Current CoW filesystems are not suited for anything that isn't a server, ok with that.

For "embedded" we need to make a difference, mobile devices are increasingly getting in collision with PC userbase (also for Apple, remember Cook saying their iPad Pro shoud replace PCs?), so in a near-ish future it might.

Although the trend I'm seeing is more Google-ish, "providing devices with low storage and offering large cloud storage services", which isn't even bad (if you have a decent Internet connection), as handling RAIDs, making backups, and doing other kinds of shit is beyond the average user's skill level anyway.
Data in the cloud is much less hard to lose from the common accidents that I see people lose data from (malware, hardware failure, idiots installing Linux wrong).

[starshipeleven]

So.. what your saying now is that all software should be GPL?

No, just that code that is integrated with other GPL code must be GPL-compliant or it violates the GPL of the other code.
ZFS is permissive license so if it went in mainline would violate the kernel's GPL.

Mozilla Firefox isn't a kernel patch compiled with the kernel so it can be whatever the fuck license it wants and it does not violate the kernel's GPL.

Nice respin tho. Are you out of ammo now?

[unixfan2001]

Let's not mix things up please.
In the other thread (you know which) I was just having some fun trolling permissive license trolls, sometimes I find funny to do that while sounding like a ravaging lunatic, but I assure you I'm just playing.

Here I'm pretty much just stating facts.

Claiming that Canonical is a Microsoft "sockpuppet" and calling Shuttleworth a "sex slave" isn't "pretty much just stating facts".
It's slander.

[liam]

givet that it's apple, they will shove half-baked crap down customers throats, like applemaps. and make "you are holding it wrong" excuses

Apple maps was a rather unusual product from apple.
I'd say that apple, more often than not, deliver well designed products that are pretty forward looking.

[liam]

Apple is anything but unpredictable after Steve (our lord and saviour) died.

They focus on people that have money and care about looks, which for the most part don't know a damn about IT, and may not even use OSX at all.
All data indicates that Apple hardware is selling like hot cakes and is outselling minor OEMs, while usage share of OSX is stationary, which means obviously one thing.
People install windows on them for a long list of reasons (they need to run programs and virtualizing isn't enough), and I can testify that. I lost the count on the apple laptops I had to install Windows on (natively, which is a bit more involved than using their crappy guided system).

... hilariously wrong since the rumour was clearly total bs.
ZFS is a huge monster that needs dozens of GB of ECC ram to run even relatively small arrays.
Also without AES acceleration (which is common now but not years ago) it kills the processor with checksum calculations.

ZFS was designed with large datacenter use in mind, where ECC RAM, processor power and AES acceleration are plentiful. It's NOT and I repeat NOT anywhere near suited for small-scale use.

ZFS has some tradeoffs that simplified the development but made it more limited in scope, like assuming that RAM was ECC and plentiful, or that the arrays could not be changed easily.

btrfs which is the evolution of ZFS supposed to run in both small and big scale, and be flexible, isn't an easy beast to tame.

Apple's needs are of a flexible file system they can deploy on both mobile and PC, and with advanced features.

These local arrays can stay with HFS+ much longer since they don't need encryption nor are flash-based.

We are talking of a filesystem driver that would be shared between a mostly-posix OS (linux) and a posix OS (OSX)

What they need isn't narrowly-defined, they need an all-purpose filesystem able to scale decently in their mobile and PC sector, while being flash friendly and other features. That's not easy as making a filesystem for just datacenters like ZFS.

F2FS isn't btrfs. btrfs is unfinished, f2fs is already usable as-is.

I'm not sure how productive this dialogue can be given our divergent views regarding some aspects of Apple.
So, ignoring those things, I've been looking at the kernel interface for module development since that's what btrfs would be using. I don't think that posix says anything about what internal interface the kernel uses, but rather what interfaces the kernel's services provide. So, being posix compliant might not mean much (anything?) when the question is "how easy would it be to port this kernel feature from one posix-y kernel to another?".
From all accounts, zfs only needs a lot of ram if you are using its dedupe feature.
Generally speaking, I think a cow fs isn't well suited to embedded environments.

[k1e0x]

You guess wrong.
I'm always saying zfs is ok as a separate prject where it is due to its own license, while you are whining about it not being in mainline and handwaving a solution for legal risks or assuming they don't exist. Ah, and giving the fault to linux kernel license even when it is a project that predates ZFS.

Uhh.. I think your more interested in arguing with me than anything else.

ZFS actually has a permissive licence. Linux does not.

So.. what your saying now is that all software should be GPL? Your making my point about the GPL creating barriers for me. Tell me.. are you reading this in Firefox or "Iceweasel"? Linux predates Mozilla too. How horrible of them to choose the MPL.