Apple Designs New File-System To Succeed HFS+

[starshipeleven]

I think your interpretation is based on a cost/benefit analysis, which is fine, but unless you are making the same judgments of those values (and the actual value of those values) as apple, you can't really replicate their reasoning.

Apple is anything but unpredictable after Steve (our lord and saviour) died.

They aren't focusing on the it crowd (though, from my experience, a very large percentage of developers/engineers use apple hardware/software---they may not be it, but they are a highly technical audience), but that doesn't matter. What apple cares about is customer satisfaction (perhaps indirectly, but that is the effect we see).

They focus on people that have money and care about looks, which for the most part don't know a damn about IT, and may not even use OSX at all.
All data indicates that Apple hardware is selling like hot cakes and is outselling minor OEMs, while usage share of OSX is stationary, which means obviously one thing.
People install windows on them for a long list of reasons (they need to run programs and virtualizing isn't enough), and I can testify that. I lost the count on the apple laptops I had to install Windows on (natively, which is a bit more involved than using their crappy guided system).

Do you recall the rumors, from a number of years ago, about apple adopting zfs? My guess is...

... hilariously wrong since the rumour was clearly total bs.
ZFS is a huge monster that needs dozens of GB of ECC ram to run even relatively small arrays.
Also without AES acceleration (which is common now but not years ago) it kills the processor with checksum calculations.

ZFS was designed with large datacenter use in mind, where ECC RAM, processor power and AES acceleration are plentiful. It's NOT and I repeat NOT anywhere near suited for small-scale use.

The cost of development wouldn't necessarily be that large. Sun managed to develop, and deploy, ZFS in about 5 five years.

ZFS has some tradeoffs that simplified the development but made it more limited in scope, like assuming that RAM was ECC and plentiful, or that the arrays could not be changed easily.

btrfs which is the evolution of ZFS supposed to run in both small and big scale, and be flexible, isn't an easy beast to tame.

Apple's needs are of a flexible file system they can deploy on both mobile and PC, and with advanced features.

You're forgetting that apple has a huge chunk of the av market, and editing hd/4k video, in an intermediate format, takes up HUGE amounts of space (they literally use TBs for scratch space).
If it's a big operation, they are using a san, and then the fs doesn't matter, but the majority of av folks aren't working at those places, and they get by with local arrays attached through thunderbolt.

These local arrays can stay with HFS+ much longer since they don't need encryption nor are flash-based.

I'm referring to the impedance mismatch between the two OSs, here.

We are talking of a filesystem driver that would be shared between a mostly-posix OS (linux) and a posix OS (OSX)

It might be better, but I think you are overestimating the work delta between building a narrowly defined fs product, from scratch, and porting an unfinished fs from another os, and ensuring it meets their reliability expectations.

What they need isn't narrowly-defined, they need an all-purpose filesystem able to scale decently in their mobile and PC sector, while being flash friendly and other features. That's not easy as making a filesystem for just datacenters like ZFS.

F2FS isn't btrfs. btrfs is unfinished, f2fs is already usable as-is.

[k1e0x]

Your definition of "everything" is weird.

ZFS runs only on Unix.

Linux has an unofficial port of ZFS, Windows has jackshit even if it would be legal to make a ZFS driver for windows.

It needs a Posix compliant system. So anything that's posix compliant can in theory use it (So for instance ZFS on Haiku is probably possible if people want to do the work) It would be legal to make a ZFS windows driver, as the license isn't the problem the problem is windows isn't posix compliant. So you have code that is more legal to use on Windows than it is on Linux.. thus making my point about the GPL creating barriers for open source.

Now lets get real for a second here.. because ZFS on Linux isn't some kind of sketchy "unofficial port". ZoL is a participant to the OpenZFS Project the same as Illumos and FreeBSD. It is as mainline as the other two. The legalities of it are unknown because it depends if you define ZFS as a derivative work of Linux or not. Ubuntu and many other distros ship ZFS included now because ZFS is defined (quite properly) as a derivative work of Illumos. Clearly that is the case, can anyone argue that? So since that's true there is no licence incompatibility with the GPL. Will it be included in the kernel? No, but neither will a lot of other things that make up a running Linux system like glibc for instance and your kernel won't get much done without that. So in the end the "legal issues" on ZFS amount to fud.

It doesn't solve every problem but it is the best file system we have and it's miles ahead of anything a competing operating system has (: cough : NTFS). I don't understand why you wouldn't want to use this advantage you have and you know the Linux community would be wise to adopt it because there are things out there like FreeBSD and Illumos that ARE being used for containers that are a natural fit for ZFS. Illumos can even execute native Linux Docker containers and run them securely in a Zone (think FreeBSD Jail) on bare metal. So.. you know.. suit yourself.. you can sit around and talk about imaginary licence incompatibilities and lowering boot times with systemd while better designed operating systems run over Linux. : shrug :

[liam]

starshipeleven bug77

I think it's unlikely they started work behind close door and at some point they decided to announce it even if it's not complete. It just doesn't make sense to change strategies. My money's on a project that just recently started.

Hi bug77,

It also seems unlikely that they would announce a product that wasn't very close to completion.
Given that this is their first fs in...however long ago hfs got it's + ago, I think they are trying to hew close to their preferred announcement/release cadence while still providing themselves, and their customers, with a bit of protection via the development release status.

One, we don't know how long they've been working on this project. Given that it's apple, it could've been in the works for years, with a team of hundreds working around the clock

Let's ignore for a moment that modern filesystems require substantial testing before being declared anywhere near stable, and you simply cannot do that in-house unless you have a few hundred million people in your basement.

That said, Apple isn't a charity, and making THE BEST FILESYSTEM EVAR isn't going to boost their sales by any amount, since they target the a userbase that does not kow shit about IT.
Rational approach is simply hack some new features in their current filesystem, or take soemthing off-the-shelf and expand on it like they did for everything.

A modern filesystem is not something you can develop from scratch in a pinch over a weekend in a garage, that large development cost has to be justified.

Hi starshipeleven,

I think your interpretation is based on a cost/benefit analysis, which is fine, but unless you are making the same judgments of those values (and the actual value of those values) as apple, you can't really replicate their reasoning.
They aren't focusing on the it crowd (though, from my experience, a very large percentage of developers/engineers use apple hardware/software---they may not be it, but they are a highly technical audience), but that doesn't matter. What apple cares about is customer satisfaction (perhaps indirectly, but that is the effect we see). They are aware of the limitations of hfs+ and have been looking to replace it for years. Do you recall the rumors, from a number of years ago, about apple adopting zfs? My guess is that they've had people looking at this, and working on it, for the better part of a decade, and, being apple, you just don't here about it until they release it.
The cost of development wouldn't necessarily be that large. Sun managed to develop, and deploy, ZFS in about 5 five years.

https://en.wikipedia.org/wiki/ZFS#History

And Apple has hugely more resources than sun even had.

Two, F2FS might work except for that 16TB volume limitation. However, I'd imagine that'll get fixed before long.

The point was taking a project with similar goals that exists already and expanding on it. Currently, given the sizes of Apple storage, 16TB will be enough for a long while, plenty of time to fix that limitation.

You're forgetting that apple has a huge chunk of the av market, and editing hd/4k video, in an intermediate format, takes up HUGE amounts of space (they literally use TBs for scratch space).
If it's a big operation, they are using a san, and then the fs doesn't matter, but the majority of av folks aren't working at those places, and they get by with local arrays attached through thunderbolt.

Three, how would working on a linux fs help osx?

if they adopt it as their own the fs isn't "a linux fs" anymore, like say CUPS, it's mostly developed by Apple, but it is used by Linux too.

I'm referring to the impedance mismatch between the two OSs, here.

Yeah, they could port it, and they'd be less work than designing something from scratch, but that's still a pretty significant amount of work for something that isn't "done" yet.

I don't understand what you mean here, porting and expanding a bit something that is "not done yet" remains better than designing it from scratch. We aren't talking of re-implementing shit and simple fs like UDF or FAT.

Which is why "apple develops a new modern filesystem from scratch" is imho ruled out

That's why it's 99% likely they chose the easiest way, respinning their own HFS+ by adding some features around.

It might be better, but I think you are overestimating the work delta between building a narrowly defined fs product, from scratch, and porting an unfinished fs from another os, and ensuring it meets their reliability expectations.

Whatever the case, I'd expect we'll know the answer in the next few months.

[bug77]

smitty3268 pal666 starshipeleven Thaodan bug77

One, we don't know how long they've been working on this project. Given that it's apple, it could've been in the works for years, with a team of hundreds working around the clock
...

I think it's unlikely they started work behind close door and at some point they decided to announce it even if it's not complete. It just doesn't make sense to change strategies. My money's on a project that just recently started.

[starshipeleven]

One, we don't know how long they've been working on this project. Given that it's apple, it could've been in the works for years, with a team of hundreds working around the clock

Let's ignore for a moment that modern filesystems require substantial testing before being declared anywhere near stable, and you simply cannot do that in-house unless you have a few hundred million people in your basement.

That said, Apple isn't a charity, and making THE BEST FILESYSTEM EVAR isn't going to boost their sales by any amount, since they target the a userbase that does not kow shit about IT.
Rational approach is simply hack some new features in their current filesystem, or take soemthing off-the-shelf and expand on it like they did for everything.

A modern filesystem is not something you can develop from scratch in a pinch over a weekend in a garage, that large development cost has to be justified.

Two, F2FS might work except for that 16TB volume limitation. However, I'd imagine that'll get fixed before long.

The point was taking a project with similar goals that exists already and expanding on it. Currently, given the sizes of Apple storage, 16TB will be enough for a long while, plenty of time to fix that limitation.

Three, how would working on a linux fs help osx?

if they adopt it as their own the fs isn't "a linux fs" anymore, like say CUPS, it's mostly developed by Apple, but it is used by Linux too.

Yeah, they could port it, and they'd be less work than designing something from scratch, but that's still a pretty significant amount of work for something that isn't "done" yet.

I don't understand what you mean here, porting and expanding a bit something that is "not done yet" remains better than designing it from scratch. We aren't talking of re-implementing shit and simple fs like UDF or FAT.

Which is why "apple develops a new modern filesystem from scratch" is imho ruled out

That's why it's 99% likely they chose the easiest way, respinning their own HFS+ by adding some features around.

[starshipeleven]

ZFS actually has a permissive licence. Linux does not. That's why everything except Linux already has it. - An this is REALLY a problem. I highly doubt the GNU's goal was to create barriers for software the way it has. Open source won and now we are squabbling over technicalities that few lawyers even understand. I really think that weak copyleft is the right approach, strong copyleft does nothing but bite Linux in the ass and deny it from progressing.

Your definition of "everything" is weird.

ZFS runs only on Unix.

Linux has an unofficial port of ZFS, Windows has jackshit even if it would be legal to make a ZFS driver for windows.

[unixfan2001]

Sure it is, it isn't open source "code" but it is open source "documentation". Besides that statement was very carefully worded we really don't know what will happen a year or two from now with the code. Given that if the documentation is good, a compatible Linux interface could be written.

From a legal standpoint, "open documentation" doesn't equate to anything.
There are plenty of products which are "openly documented" but that the creators will happily sue you for if you dare re-implement them.

[fuzz]

GPL is hard to even accept as an open source license, mainly because you aren't free to use the software as you please.

Actually, it does enable you to use the software as you please and ensures that you continue to be able to do so. Developers are so fucking selfish, but in the end it's the users that matter and open code is and always will be in their best interests.

Jeeze, your argument is like saying the Constitution infringes upon my rights to do whatever I want to people. Fuck the Constitution, I want to enslave others!!! Get real.

[liam]

smitty3268 pal666 starshipeleven Thaodan bug77

One, we don't know how long they've been working on this project. Given that it's apple, it could've been in the works for years, with a team of hundreds working around the clock
Two, F2FS might work except for that 16TB volume limitation. However, I'd imagine that'll get fixed before long.
Three, how would working on a linux fs help osx? Yeah, they could port it, and they'd be less work than designing something from scratch, but that's still a pretty significant amount of work for something that isn't "done" yet.

[bug77]

Actually it is well known in the Mac community that Apples graphics drivers suck! For the majority of users it doesn't matter as they aren't gaming.

Well known is one thing. Is the Mac community as vocal demanding proper drivers as the Linux community that doesn't loose a chance to crucify Nvidia for their closed source drivers or AMD for their lacking performance?