Persistent Memory Was A Popular Topic At This Week's LinuxCon Europe

[starshipeleven]

I have no idea what Palm were doing. But most of the commenters here (including you) don't seem to appreciate the technical points that are stake here.

Finally someone that grasps what "persistent memory" is about, in this context.

[lejeczek]

Intel team looked like all-Polish team?

[DrYak]

I have no idea what Palm were doing. But most of the commenters here (including you) don't seem to appreciate the technical points that are stake here.

I know I tend to make long post, and that this length tends to discourage people. So here's the TL;DR:
- All PalmOS-powered PDA, from the first Pilot, all the way to somewhere during the PalmOS 5.x ARM era :
DID NOT HAVE ANY MASS STORAGE (beside some EEPROM from which to boot a new clean OS in case of power loss or in case of hard-reset requested by the user).
THEY DID 100% RELY EXCLUSIVELY ON RAM. Everything was stored in RAM back then.
(And RAM was battery backed for persisting).

The issue is not that NV-RAM is a faster storage medium, just like SSDs were faster than HDs.

Yes, I know. The main point is that NV-RAM is RAM, but special RAM that survives between boots (like SSDs), just like Battery-backed RAM used to some decades ago.
Back then, that used to enable developers to completely drop the concept of files, and use persisting in-RAM databases.

{...}

The third thing it means is that, over time, for at least some purposes, people will consider alternative storage models to files. Some devices did this in the past --- Newton "soups" were a kinda non-filed-based storage system, and various cloud storage schemes are likewise not file based (Apple offers key-value stores for apps using iCloud, and I'm sure so do various other APIs).

And like I said, PalmOS did do it too (just like the newton) a couple of decades ago, except that they relied on batteries and plain simple RAM, instead of some cool advanced technology (the thing that you could export as PDB files on your desktop computer. They were the internal databases used by everything inside a Palm).
It worked already back then.

Non-file-oriented storage, had been already solved back then, at least for some types of databases.

And given that nowadays tons of code (among which, a huge chunk of the modern web) doesn't not rely on direct file access, but on some form of database access (be it SQLite embed into most modern browser, or special no-SQL databases like all the popular key-value stores) that's tons of code that can directly benefit from non-volatile RAM.

No need to invent "new forms of software that can go file-less". Code currently running in the wide exsits that already has no concept of file, and instead leaves this kind of low-level handling to some databse engine.

But ALL these cool new ideas require something to enforce the ordering of writes out to the NV-RAM. Without that in place, you're playing amateur hour. That's the part that's missing right now from the actual CPUs in the field. And that's the part that requires careful new additions to every OS, whether Linux, Apple, Google, or MS.

Or you know, I you don't want to milk the last nano-second out of your NV-RAM, you could also simply use some form of representation (copy-on-write, log-structure, journaling, etc.) that means that your in-RAM database doesn't get b0rked in case of crashes mid-writing, or power-loss in the middle of an update, or simply if your memory access got re-ordered due to some cache optimizing mumbo jumbo.

If 20 years ago PalmOS PDAs worked perfectly well with their battery powered RAM, it shouldn't be that much complicated to do in-RAM databases nowadays.

(Yes, the simplistic techniques used to avoid corruption are probably performing much worse than inventing new forms of CPUs that can control the order of writes-to-NV-RAM. Still, they are usable now).

You can't just treat this stuff like a RAM-disk, because RAM-disks didn't persist if the CPU crashed/lost power half-way through a storage operation.

Battery backed RAM disk did exist and used to be the best solution before SSDs started to take over that marked segment.
So yes, there used to be RAM disk that did persist between reboots.

NV-RAM, if it sees the stores in the wrong order and power is lost part way through, will be basically a scrambled file system, like we used to see back in the 80s before the PC OS vendors took write ordering seriously.

Multi-session ISO9660 (basically a *VERY CRUDE* form of log-structured filesystem) did already exist and started getting widespread in the 80s.
And I'm sure the FS buffs here will probably point you to various copy-on-write and log-structured filesystems that can handle such kind of scrambling.
(Sorry I only know the modern ones).

There are various ways to mitigate write re-ordering.

[DrYak]

Persistent memory in this context means that you don't have anymore a distinction between RAM and permanent storage, which is a bit different from just battery-backing a tiny amount of RAM while most data is still on a slow permanent storage medium.

There was no slow permanent storage back then. (I mean except a small EEPROM from which to reboot a blank new system if you requested a hard reset).
Legally it was NOT even possible : putting flash in a PDA was patented.
PalmOS from 1.0 all the way to somewhere in the 5.x series was entirely in-RAM databases that were kept alive by battery.

The only time these data would be on a slow permanent storage is if you exported it as .PDB file using the sync cable for backup purpose and store it on the desktop's harddisk.
(Even the actual normal everyday garden-variety sync doesn't exchange files. It does a database merge between the desktop application and the in-device database. .PDB files are used for backup purpose exclusively).

[starshipeleven]

There was no slow permanent storage back then. (I mean except a small EEPROM from which to reboot a blank new system if you requested a hard reset).
Legally it was NOT even possible : putting flash in a PDA was patented.
PalmOS from 1.0 all the way to somewhere in the 5.x series was entirely in-RAM databases that were kept alive by battery.

That's pretty normal for most embedded devices. They boot from a storage device, then live in RAM for as long as the energy lasts.
That is still not the same as here, embedded devices have programs designed to run 24/7 from RAM.

In this case, we are talking of implementing a bunch of tricks to get normal applications designed for the current PC paradigm to work in a different paradigm.

[DrYak]

That's pretty normal for most embedded devices. They boot from a storage device, then live in RAM for as long as the energy lasts.

There's a big difference between a *router* which is basically a single-purpose machine that does constantly the same stuff (filtering packets and routing them around).
And a PDA/tablet/smartphone which is supposed to be a "computer in your pocket", complete with numerous applications, including calendar, notes taking, contacts, etc. but also games, etc.

And nope the Palm doesn't *boot from ROM then only lives in RAM for the session*.
Palm PDA boot from RAM (that was persisted thanks to batteries).
ROM is only used in case the user asks for a hard-reset, or if the RAM content was lost.

But if you shut down your PDA and then start it again, as long as the battery is alive - the RAM content will be kept, including all the databases that are persisted between restarts.

That is still not the same as here, embedded devices have programs designed to run 24/7 from RAM.

In this case, we are talking of implementing a bunch of tricks to get normal applications designed for the current PC paradigm to work in a different paradigm.

The thing is that nowadays, most user-level applications (of which most nowadays are webapps) don't give a damn fuck about what the storage paradigm is. They usually don't handle files directly (and in case of web-apps, they don't even have access to files by design).
As long as their database requests (SQL, key/value, whatever is trendy now) get served, modern apps are happy.

And this situation has been the case since already the days of the first PDAs.

In fact the situation is quite opposite with embedded devices : lots of them (like routers, etc.) actually use (volatile) RAM disks, so they are still designed to run with files paradigm.

[starshipeleven]

There's a big difference between a *router* which is basically a single-purpose machine that does constantly the same stuff (filtering packets and routing them around).
And a PDA/tablet/smartphone which is supposed to be a "computer in your pocket", complete with numerous applications, including calendar, notes taking, contacts, etc. but also games, etc.

Function is irrelevant, I'm just saying that when you design a device from scratch and its whole application ecosystem from scratch you can have everything designed to run from RAM and never shut down, and it's not hard.

The talks here are about migrating the current PC ecosystem (designed with a RAM/disk paradigm in mind), to a future where you have "persistent memory", without rewriting everything.

And nope the Palm doesn't *boot from ROM then only lives in RAM for the session*.
Palm PDA boot from RAM (that was persisted thanks to batteries).
ROM is only used in case the user asks for a hard-reset, or if the RAM content was lost.

But if you shut down your PDA and then start it again, as long as the battery is alive - the RAM content will be kept, including all the databases that are persisted between restarts.

Technically speaking, the Palm never shuts down unless you are hard resetting it. Its "shutdown" is actually a "suspend to RAM" mode.

The thing is that nowadays, most user-level applications (of which most nowadays are webapps) don't give a damn fuck about what the storage paradigm is. They usually don't handle files directly (and in case of web-apps, they don't even have access to files by design).
As long as their database requests (SQL, key/value, whatever is trendy now) get served, modern apps are happy.

Most applications in my PC (windows/linux), and in my smartphone (Android) aren't webapps working with remote content, so I'm not sure what you mean here.

In fact the situation is quite opposite with embedded devices : lots of them (like routers, etc.) actually use (volatile) RAM disks, so they are still designed to run with files paradigm.

That's because the software they use (usually some Linux kernel and some linux userspace coming from server/desktop) is not designed for true Persistent memory, and none feels like rewriting such software from scratch.

[DrYak]

The talks here are about migrating the current PC ecosystem (designed with a RAM/disk paradigm in mind), to a future where you have "persistent memory", without rewriting everything.

And again, tons of user-facing application aren't directly dealing with files anymore.
They rely more on libsqlite3 still replying no matter what the underlying storage paradigm, rather than needing extensive rewrites because we've shifted from files on block-accessed mass storage to peristant RAM

Technically speaking, the Palm never shuts down unless you are hard resetting it. Its "shutdown" is actually a "suspend to RAM" mode.

In theory. And maybe for the few users that did only use the base applications.

For power users that tried to push the limit, Palm PDAs could crash and reboot.
(The fact that the first m68k processors used in these lacked an MMU and thus the first generations of PalmOS lacked memory protection didn't help)

(or you could simply push the reset switch to reboot it)

And did reboot from the current in-RAM state.

[starshipeleven]

And again, tons of user-facing application aren't directly dealing with files anymore.

Most applications in my PC (windows/linux), and in my smartphone (Android) aren't webapps working with remote content, so I'm not sure what you mean here.

In theory. And maybe for the few users that did only use the base applications.

For power users that tried to push the limit, Palm PDAs could crash and reboot.
(The fact that the first m68k processors used in these lacked an MMU and thus the first generations of PalmOS lacked memory protection didn't help)

(or you could simply push the reset switch to reboot it)

And did reboot from the current in-RAM state.

I'm talking of technical definitions, not of how the buttons are labeled.
"shutdown" means you cut power to EVERYTHING, and when talking of "reboot" people usually means a "hard reboot" which implies a shutdown followed by a power up.

If the device isn't cutting power to everything it is just "soft-rebooting".

http://digital.ni.com/public.nsf/allkb/C268F401137B3D14862573A600771CB4

In this case the operating system does not screw up the current RAM image when starting up as it has no BIOS or anything, so when soft-rebooting it can use the same RAM contents without ill effects.
A soft-reboot for most other devices means that the hardware itself is already initialized so the hardware initialization phase is skipped, but the RAM contents are usually scratched as the OS starts up normally.