FUSE Read/Write Passthrough Updated For Much Better File-System Performance

[boeroboy]

The interesting thing is that most bugs came from the complication associated with mitigating the perf issues by 'caching'.
Need to list the contents of a moderately-big directory? S3 api requires you to make one thousand http calls; latency will kill you unless you cache the results. Same for quickly opening one thousand small files. Cache them locally or suffer big time.

Exactly. I ended up caching metadata in some of mine as well. Plus you need big reads and big writes otherwise you'll only get 4k blocks at a time which kills performance on a good day.

[miquels]

Need to list the contents of a moderately-big directory? S3 api requires you to make one thousand http calls; latency will kill you unless you cache the results.

Just wondering ... what if you use HTTP/2 and just send out all http calls in parallel instead of sequentially?

[starshipeleven]

Hmmhmm. this is only for merge/bindmount like things, right? It would be nice having sort of this functionality for things like ntfs-3g as well.

E.g., if the user opens a file, ntfs-3g can check whether a 1:1 passthrough is possible (ie. not fragmented, not compressed, not encrypted, not sparse etc.) - and then tell the fuse kernel part "you can read/write directly to /dev/sda1 offset 10240000 as long you don't read/write past 10480000" (this way passthrough can be used at least for the first continuous part of the file). In the next iteration the user space part could give the kernel part the runlist of the fragmented parts of the file. The kernel could delay requesting the runlists until a few accesses to the opened files to save resources.

that would be a ntfs-3g-specific thing

Also, let's be honest, how many files in NTFS are "not fragmented, not compressed, not encrypted, not sparse"?

[mifritscher]

This could use every fuse driver which has a block device behind and where a 1:1 mapping is possible. ntfs, exfat etc. btrfs/zfs would be possible as well - at least ro. (write by a user space call for allocating, then the to be written data can be kept completely in kernel space)
Fragmented files can be done in V2.0, also sparse files (excluding writes to sparse areas which aren't allocated yet). But at least accessing the first fragment could be done in V1.0 quite easily.