Linux 6.12 NFS Adds LOCALIO Protocol For "Extreme" Performance Boost

[oiaohm]

I'm surprised people still take NFS seriously when the only form of security it provides is checking the source IP address, and it transmits the data unencrypted over the network. Yes, I know there is a way to add data encryption. I've looked into it and it seems to be a convoluted mess to implement with very little good documentation.

There is a problem here. Its a disconnect.

https://wiki.archlinux.org/title/NFS/Kerberos

https://wiki.archlinux.org/title/NFS#TLS_encryption

Well first of all it's not a complete package. You need add-ons from Oracle to make it work. And the very limited information about the encryption add-on leads me to believe that few people are actually using it. Hence the surprise that NFS is still a thing in today's highly security-focused environment.

Might as well bring Telnet back.

This is wrong and right. You don't need addons from oracle to make secure NFS work it is just one of the options. You using NFS TLS encryption you use ktls-utils that currently maintained by Orcale to use Linux kernel TLS because it a helper application to the Linux in kernel TLS. Yes that helper application is need when ever you decide to use the Linux kernel TLS. Yes arch and other write this up as a valid option but you kind have to be insane to use software that marked as expermental like ktls-utils and the in kernel TLS.

Next is use the cross platform Kerberos NFS option.
To use Kerberos NFS in decent sane way you need gssproxy this can encrypted identify as one option or complete encrypted the traffic as another..

https://github.com/gssapi/gssproxy/tree/main/docs

Might come clear reading above. The reality is Kerberos NFS 4 the one that support full encrypted and lot more identification is an Kerberos application that used gssproxy for the Kerberos tasks.

Then you have a lot of older legacy systems that don't support NFS TLS or NFS Kerberos at all and most of those have older worse performing protocol options with security that no longer works any how. So you have to do elvis solution.

Yes the reality is NFS does not implement encryption itself instead it off load to some party being either gssproxy(for Kerberos) or inkernel TLS.

The NFS documentation is very short or encryption and security because simple NFS protocol does not implement encryption heck barely does user identification instead these are offloaded to software that provides TLS and Kerberos support.

Yes setting up NFS to use gssproxy you only need a few fine steps. Setting up detailed Kerberos configuration for NFS is setting up Kerberos server that kind of out side NFS then setting up all the fine settings that is all gssproxy items that is the same for gssproxy Kerberos applications. See very quickly this stops being NFS.

Chugworth NFS has always been developed with do not reinvent the wheel model leading to you want to do X you need to get to know how to configure applications that are not NFS because NFS implementation will have offloaded the feature.

The most common reason why NFSv3 or NFSv4 is used without encryption is performance. Yes NFSv4 does not always perform better than NFSv3 this is because NFSv4 does have better locking so yes you can gain performance using NFSv3 with high risk of messed up data.

The most common use encrypted form of NFS is the Kerberos form because it cross platform and you don't end up having issues with the in kernel TLS. Yes issue with in kernel TLS is if other applicaitons also need to be using it. gssproxy was designed from the ground up that you would have multi different Kerberos applications running at the same time..

[mobadboy]

there are better ways to do mounting....

[elvis]

I'm surprised people still take NFS seriously when the only form of security it provides is checking the source IP address, and it transmits the data unencrypted over the network. Yes, I know there is a way to add data encryption. I've looked into it and it seems to be a convoluted mess to implement with very little good documentation.

HPC/VFX systems architect here. Yup, we know it's "insecure". But NFSv3 is still far, far more performant than other protocols.

SMB3 has nice stuff in it, but really requires a lot of effort and annoying network design to utilise multi channel and get performance up to the levels required. SMB over QUIC is very exciting, but that's going to take a while before we see it arrive for easy consumption in ksmbd and on clients. SMB over RDMA is just painful, and breaks constantly.

NFSv4 is nice and secure, but the performance overheads are annoying. If you're pushing 100GbE+ workloads, that matters. Likewise pNFS has its upsides, but clustered storage is always tricky for things that need good IO performance, versus just ensuring you don't saturate a single point in your cluster. For smaller (single digit petabyte) workloads, it's quite easy just to fill a device full of NVME drives and export that over traditional NFSv3 to overcome those bottlenecks.

Same "clustering is annoying" comments apply to things like GPFS, Lustre, etc. Yes these are marvellous at very large scale where you want a single namespace. But people like Weta still use lots and lots of individual NFSv3 NAS devices and a globally managed autofs via Puppet/Ansible/Salt type tools, simple because it's cheaper, easier and more performant for their industry.

S3 (the protocol, not the AWS service) and object storage in general are great, if your applications can use them. We're trying very hard to migrate people over to object storage, but HPC and VFX industries still have a huge volume of tools and applications that demand "old fashioned" POSIX style file systems. We try to push users and vendors alike, but that's a long road, and we're only at the start.

Would I expose NFSv3 to an untrusted network? Hell no. Would I roll out NFSv3 in 2024 on a controlled, locked down, internal, storage-dedicated network with thousands of private compute nodes smashing my storage? Yup, you bet. Am doing just that right now. Coupled with 100GbE+ and jumbo frames, NFSv3 still beats the pants off anything else.

Like literally anything in computing, there is no "one size fits all" solution. If you think someone using something is surprising, go and ask the people using it why they do. You'll probably learn about a tonne of industry-specific requirements you never even knew existed. This is business diversity in action.

[trapexit]

I wonder if there’s any scenario where LOCALIO would work but bind-mounting directories wouldn’t. I also wonder what kind of overhead this will have for non-LOCALIO scenarios.

You should not be using the underlying filesystem directly when exported via NFS. For some reason this isn't widely mentioned because perhaps it's not a common need for people but as the author of mergerfs I see folks do this *all the time* and don't think twice. With SMB, mergerfs, NFS, overlayfs, etc. mergerfs at least can be configured so caching is largely disabled so there is little risk in normal usage but if you have some sort of layered filesystem setup you should *not* be accessing the underlying filesystem out of band.

In this case if you have containers that could live anywhere you don't really want to conditionally manage the storage. NFS share lives on host X... doesn't matter where the container lives it mounts from X. No "if local bindmount, else nfs"

[Chugworth]

Tell me you have no idea without telling you have no idea.



Right. If we ignore authentication and encryption, then NFS has no authentication and encryption.

Well first of all it's not a complete package. You need add-ons from Oracle to make it work. And the very limited information about the encryption add-on leads me to believe that few people are actually using it. Hence the surprise that NFS is still a thing in today's highly security-focused environment.

Might as well bring Telnet back.

[pgoetz]

I'm surprised people still take NFS seriously when the only form of security it provides is checking the source IP address, and it transmits the data unencrypted over the network. Yes, I know there is a way to add data encryption. I've looked into it and it seems to be a convoluted mess to implement with very little good documentation.

While I completely agree with about the lack of good documentation, the secure version of NFS is designed to work with Kerberos and isn't particularly convoluted. Not only is NFS widely used, particularly in the HPC/academic world, but the software is under very active development, as evidenced by the scores of patches posted to the NFS developers list.

[dragorth]

I could see this as being useful on a Proxmox host with one Open Media Vault container and then other containers that use that as the backing storage, such as NextCloud. Cool.

[EphemeralEft]

That doesn't strike me as something I'm going to be able to make much use of. What I'm currently exporting is already directly accessible on the server.

I wonder if there’s any scenario where LOCALIO would work but bind-mounting directories wouldn’t. I also wonder what kind of overhead this will have for non-LOCALIO scenarios.

[NateHubbard]

in scenarios where the NFS client and server are on the same host.

That doesn't strike me as something I'm going to be able to make much use of. What I'm currently exporting is already directly accessible on the server.
But, yeah, I get it. Seeing that "extreme performance" headline got my attention though.

[intelfx]

I'm surprised people still take NFS seriously when the only form of security it provides is checking the source IP address, and it transmits the data unencrypted over the network.

Tell me you have no idea without telling you have no idea.

Yes, I know there is a way to add data encryption. I've looked into it and it seems to be a convoluted mess to implement with very little good documentation.

Right. If we ignore authentication and encryption, then NFS has no authentication and encryption.