Google Is Still Striving To Upstream Incremental FS In Linux

[starshipeleven]

Wow.

This seems like an incredibly complicated solution, prone to myriads of problems and errors, to solve a minor annoyance.

I can't help but think that, depending upon the actual file download speed and fragments received, that all of these special tracking and deferment operations could quickly evolve into an incalculable mess, with a high likelihood of extensive delays and even lockups.

Don't get me wrong, it's not that I think it's impossible. Just that it's so complex and rife with unforeseen scenarios that it seems it would take a very long time to actually be proved stable and reliable.

In fact by the time it was Internet speeds might be so fast that the whole point of the exercise could be moot.

Anything coming from Android has been out-of-tree for years already, I guess they have ironed out a lot of issues.

[starshipeleven]

Why is it always Google? I wanna see DuckDuckGo start contributing, so we can praise them too.

DuckDuckGo does not develop a smartphone OS and probably never will

[Palu Macil]

And you think all of those 144,096 Google employees work on the Linux kernel? Their Linux team is very small as well (unless you want to count ChromeOS and Android employees as well).

Not counting two teams that do upstream lots of code to the kernel is a bit weird, but my point was more that kernel code isn't something everyone has resources to contribute to, and if they do, there isn't much reason to think you'll have a reason to contribute. If I owned a web company ten times the size of DuckDuckGo but didn't manufacture ARM devices I might not have a single kernel dev.

[gregzeng]

Perhaps some writer should explain some terms used here. Google is one of this planet's largest contributor to the open source world: Chromium web browser & operating systems, Android operating systems, etc. It finally realized that "do no evil" was bad, because the doing often has unexpected consequences that upset some people.
Some here seem to not realize that "upstream" has many events, including "systemd". If these upstr4eam events are open sourced, via "The Linux Foundation" (financed by Google, Microsoft & others), then the benefits will be given to all those "downstream".

The dedicated communists here seem to think that commercial enterprises should not be allowed in the open source community. When these commercial businesses want to open source their industrial secrets, we should welcome their trust. All operating systems, open source or not, face similar problems. Google's ideas seem similar to "systemd", allowing prioritizing of multi-processes, especially in the initial stages. Slow thinkers might be happy with the status quo. However, there are many here who need or want things to become more rapid and safer.

Part of my reluctance to completely use Linux is because Linux is so slow, compared to Windows. If & when Linux speeds up its multi-tasking, and helps developers to work with Linux's slowness, then we can all benefit.

[ssokolow]

i wonder why it might be the case

Because Google owns the world's biggest ad network by far and has had decades to get where it is?

[pal666]

The point is that DuckDuckGo doesn't really have the money to have a Linux team while Google does.

i wonder why it might be the case

[pal666]

Their Linux team is very small as well (unless you want to count ChromeOS and Android employees as well).

so, considering subj is android, their linux team is not very small?

[david-nk]

Wow.

This seems like an incredibly complicated solution, prone to myriads of problems and errors, to solve a minor annoyance.

It isn't that complicated, I'd estimate it would take me about 30 minutes to implement a basic, unoptimized version of this file system using FUSE.

The more complicated parts are prefetching, preloading and profiling. You'll want to prefetch data when an application is doing sequential reads to increase throughput, you will want to preload missing data even when there is no filesystem activity and you will want to profile which data applications tend to read first. To use the Final Fantasy example, the filesystem should be able to figure out that it should preload the data for chapter 2 before the data for chapter 3.
Even with these optimizations, it's probably still doable as a weekend project.

Still, while I can see some usage cases for such a file system, I still don't quite get why Google thinks they need this for Android.

I can't help but think that, depending upon the actual file download speed and fragments received, that all of these special tracking and deferment operations could quickly evolve into an incalculable mess, with a high likelihood of extensive delays and even lockups.

Doubtful. I mean yeah, fetching the data over network will generally have higher latency than fetching them from a hard disk, but this isn't going to be any worse than existing solutions such as iSCSI.

[muncrief]

It's not just a minor annoyance. Consider a game like Final Fantasy with multiple chapters. In that case, instead of having to download an entire 500 MB game, you could download the first chapter and base game files, say 60 MB and then download successive chapters in the background as needed.

There are many other reasons for a feature like this.

I can certainly see getting some initially needed data first as an advantage, but in the case you forward why not just send what the user needs first as normal files?

And I think that gets down to my basic problem with all of this.

If there's some enormous amount of data a user ultimately needs, but they only need some part of it first, why not just send the parts they need first as regular files?

And please understand, I'm not trying to be difficult, and am not against this Virtual FS for any burning philosophical reasons, I'm just sincerely trying to understand why such an incredibly complicated mechanism is necessary to send urgently needed or prioritized data.

Why wouldn't the sender just transfer it first as normal files?

[Spacefish]

I guess this is pretty useful if you want to speedup an app start after installation and the devs of the app did not care to implement lazy loading of resources, but instead packed everything into the main app package.. In that case the system can transparently lazy load those things.

I/O should be done on a background thread anyway, so even if the virtual filesystem blocks a littlebit longer this should ne lock up the UI / rendering.

Another thing: Some apps might pack a lot of files a user does not need, like different language versions of resources (soundfiles, images?!) or different pre-compiled shaders and so on.. With that FS only the ones really used get downloaded.. This saves google bandwith, the user storage space on their device and startup after install is faster.