Announcement

Collapse
No announcement yet.

Persistent Memory Was A Popular Topic At This Week's LinuxCon Europe

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • starshipeleven
    replied
    Originally posted by DrYak View Post
    And again, tons of user-facing application aren't directly dealing with files anymore.
    Most applications in my PC (windows/linux), and in my smartphone (Android) aren't webapps working with remote content, so I'm not sure what you mean here.

    In theory. And maybe for the few users that did only use the base applications.

    For power users that tried to push the limit, Palm PDAs could crash and reboot.
    (The fact that the first m68k processors used in these lacked an MMU and thus the first generations of PalmOS lacked memory protection didn't help)

    (or you could simply push the reset switch to reboot it)

    And did reboot from the current in-RAM state.
    I'm talking of technical definitions, not of how the buttons are labeled.
    "shutdown" means you cut power to EVERYTHING, and when talking of "reboot" people usually means a "hard reboot" which implies a shutdown followed by a power up.

    If the device isn't cutting power to everything it is just "soft-rebooting".
    http://digital.ni.com/public.nsf/all...2573A600771CB4

    In this case the operating system does not screw up the current RAM image when starting up as it has no BIOS or anything, so when soft-rebooting it can use the same RAM contents without ill effects.
    A soft-reboot for most other devices means that the hardware itself is already initialized so the hardware initialization phase is skipped, but the RAM contents are usually scratched as the OS starts up normally.

    Leave a comment:


  • DrYak
    replied
    Originally posted by starshipeleven View Post
    The talks here are about migrating the current PC ecosystem (designed with a RAM/disk paradigm in mind), to a future where you have "persistent memory", without rewriting everything.
    And again, tons of user-facing application aren't directly dealing with files anymore.
    They rely more on libsqlite3 still replying no matter what the underlying storage paradigm, rather than needing extensive rewrites because we've shifted from files on block-accessed mass storage to peristant RAM

    Originally posted by starshipeleven View Post
    Technically speaking, the Palm never shuts down unless you are hard resetting it. Its "shutdown" is actually a "suspend to RAM" mode.
    In theory. And maybe for the few users that did only use the base applications.

    For power users that tried to push the limit, Palm PDAs could crash and reboot.
    (The fact that the first m68k processors used in these lacked an MMU and thus the first generations of PalmOS lacked memory protection didn't help)

    (or you could simply push the reset switch to reboot it)

    And did reboot from the current in-RAM state.

    Leave a comment:


  • starshipeleven
    replied
    Originally posted by DrYak View Post
    There's a big difference between a *router* which is basically a single-purpose machine that does constantly the same stuff (filtering packets and routing them around).
    And a PDA/tablet/smartphone which is supposed to be a "computer in your pocket", complete with numerous applications, including calendar, notes taking, contacts, etc. but also games, etc.
    Function is irrelevant, I'm just saying that when you design a device from scratch and its whole application ecosystem from scratch you can have everything designed to run from RAM and never shut down, and it's not hard.

    The talks here are about migrating the current PC ecosystem (designed with a RAM/disk paradigm in mind), to a future where you have "persistent memory", without rewriting everything.

    And nope the Palm doesn't *boot from ROM then only lives in RAM for the session*.
    Palm PDA boot from RAM (that was persisted thanks to batteries).
    ROM is only used in case the user asks for a hard-reset, or if the RAM content was lost.

    But if you shut down your PDA and then start it again, as long as the battery is alive - the RAM content will be kept, including all the databases that are persisted between restarts.
    Technically speaking, the Palm never shuts down unless you are hard resetting it. Its "shutdown" is actually a "suspend to RAM" mode.

    The thing is that nowadays, most user-level applications (of which most nowadays are webapps) don't give a damn fuck about what the storage paradigm is. They usually don't handle files directly (and in case of web-apps, they don't even have access to files by design).
    As long as their database requests (SQL, key/value, whatever is trendy now) get served, modern apps are happy.
    Most applications in my PC (windows/linux), and in my smartphone (Android) aren't webapps working with remote content, so I'm not sure what you mean here.

    In fact the situation is quite opposite with embedded devices : lots of them (like routers, etc.) actually use (volatile) RAM disks, so they are still designed to run with files paradigm.
    That's because the software they use (usually some Linux kernel and some linux userspace coming from server/desktop) is not designed for true Persistent memory, and none feels like rewriting such software from scratch.

    Leave a comment:


  • DrYak
    replied
    Originally posted by starshipeleven View Post
    That's pretty normal for most embedded devices. They boot from a storage device, then live in RAM for as long as the energy lasts.
    There's a big difference between a *router* which is basically a single-purpose machine that does constantly the same stuff (filtering packets and routing them around).
    And a PDA/tablet/smartphone which is supposed to be a "computer in your pocket", complete with numerous applications, including calendar, notes taking, contacts, etc. but also games, etc.

    And nope the Palm doesn't *boot from ROM then only lives in RAM for the session*.
    Palm PDA boot from RAM (that was persisted thanks to batteries).
    ROM is only used in case the user asks for a hard-reset, or if the RAM content was lost.

    But if you shut down your PDA and then start it again, as long as the battery is alive - the RAM content will be kept, including all the databases that are persisted between restarts.

    Originally posted by starshipeleven View Post
    That is still not the same as here, embedded devices have programs designed to run 24/7 from RAM.

    In this case, we are talking of implementing a bunch of tricks to get normal applications designed for the current PC paradigm to work in a different paradigm.
    The thing is that nowadays, most user-level applications (of which most nowadays are webapps) don't give a damn fuck about what the storage paradigm is. They usually don't handle files directly (and in case of web-apps, they don't even have access to files by design).
    As long as their database requests (SQL, key/value, whatever is trendy now) get served, modern apps are happy.

    And this situation has been the case since already the days of the first PDAs.

    In fact the situation is quite opposite with embedded devices : lots of them (like routers, etc.) actually use (volatile) RAM disks, so they are still designed to run with files paradigm.

    Leave a comment:


  • starshipeleven
    replied
    Originally posted by DrYak View Post
    There was no slow permanent storage back then. (I mean except a small EEPROM from which to reboot a blank new system if you requested a hard reset).
    Legally it was NOT even possible : putting flash in a PDA was patented.
    PalmOS from 1.0 all the way to somewhere in the 5.x series was entirely in-RAM databases that were kept alive by battery.
    That's pretty normal for most embedded devices. They boot from a storage device, then live in RAM for as long as the energy lasts.
    That is still not the same as here, embedded devices have programs designed to run 24/7 from RAM.

    In this case, we are talking of implementing a bunch of tricks to get normal applications designed for the current PC paradigm to work in a different paradigm.

    Leave a comment:


  • DrYak
    replied
    Originally posted by starshipeleven View Post
    Persistent memory in this context means that you don't have anymore a distinction between RAM and permanent storage, which is a bit different from just battery-backing a tiny amount of RAM while most data is still on a slow permanent storage medium.
    There was no slow permanent storage back then. (I mean except a small EEPROM from which to reboot a blank new system if you requested a hard reset).
    Legally it was NOT even possible : putting flash in a PDA was patented.
    PalmOS from 1.0 all the way to somewhere in the 5.x series was entirely in-RAM databases that were kept alive by battery.

    The only time these data would be on a slow permanent storage is if you exported it as .PDB file using the sync cable for backup purpose and store it on the desktop's harddisk.
    (Even the actual normal everyday garden-variety sync doesn't exchange files. It does a database merge between the desktop application and the in-device database. .PDB files are used for backup purpose exclusively).
    Last edited by DrYak; 26 October 2016, 03:14 PM. Reason: (well yeah, there's a corner case where data is persisted on a slow permanent storage medium : on the desktop's harddisk when doing a backup)

    Leave a comment:


  • DrYak
    replied
    Originally posted by name99 View Post
    I have no idea what Palm were doing. But most of the commenters here (including you) don't seem to appreciate the technical points that are stake here.
    I know I tend to make long post, and that this length tends to discourage people. So here's the TL;DR:
    - All PalmOS-powered PDA, from the first Pilot, all the way to somewhere during the PalmOS 5.x ARM era :
    DID NOT HAVE ANY MASS STORAGE (beside some EEPROM from which to boot a new clean OS in case of power loss or in case of hard-reset requested by the user).
    THEY DID 100% RELY EXCLUSIVELY ON RAM. Everything was stored in RAM back then.
    (And RAM was battery backed for persisting).

    Originally posted by name99 View Post
    The issue is not that NV-RAM is a faster storage medium, just like SSDs were faster than HDs.
    Yes, I know. The main point is that NV-RAM is RAM, but special RAM that survives between boots (like SSDs), just like Battery-backed RAM used to some decades ago.
    Back then, that used to enable developers to completely drop the concept of files, and use persisting in-RAM databases.

    {...}

    Originally posted by name99 View Post
    The third thing it means is that, over time, for at least some purposes, people will consider alternative storage models to files. Some devices did this in the past --- Newton "soups" were a kinda non-filed-based storage system, and various cloud storage schemes are likewise not file based (Apple offers key-value stores for apps using iCloud, and I'm sure so do various other APIs).
    And like I said, PalmOS did do it too (just like the newton) a couple of decades ago, except that they relied on batteries and plain simple RAM, instead of some cool advanced technology (the thing that you could export as PDB files on your desktop computer. They were the internal databases used by everything inside a Palm).
    It worked already back then.

    Non-file-oriented storage, had been already solved back then, at least for some types of databases.

    And given that nowadays tons of code (among which, a huge chunk of the modern web) doesn't not rely on direct file access, but on some form of database access (be it SQLite embed into most modern browser, or special no-SQL databases like all the popular key-value stores) that's tons of code that can directly benefit from non-volatile RAM.

    No need to invent "new forms of software that can go file-less". Code currently running in the wide exsits that already has no concept of file, and instead leaves this kind of low-level handling to some databse engine.

    Originally posted by name99 View Post
    But ALL these cool new ideas require something to enforce the ordering of writes out to the NV-RAM. Without that in place, you're playing amateur hour. That's the part that's missing right now from the actual CPUs in the field. And that's the part that requires careful new additions to every OS, whether Linux, Apple, Google, or MS.
    Or you know, I you don't want to milk the last nano-second out of your NV-RAM, you could also simply use some form of representation (copy-on-write, log-structure, journaling, etc.) that means that your in-RAM database doesn't get b0rked in case of crashes mid-writing, or power-loss in the middle of an update, or simply if your memory access got re-ordered due to some cache optimizing mumbo jumbo.

    If 20 years ago PalmOS PDAs worked perfectly well with their battery powered RAM, it shouldn't be that much complicated to do in-RAM databases nowadays.

    (Yes, the simplistic techniques used to avoid corruption are probably performing much worse than inventing new forms of CPUs that can control the order of writes-to-NV-RAM. Still, they are usable now).

    You can't just treat this stuff like a RAM-disk, because RAM-disks didn't persist if the CPU crashed/lost power half-way through a storage operation.
    Battery backed RAM disk did exist and used to be the best solution before SSDs started to take over that marked segment.
    So yes, there used to be RAM disk that did persist between reboots.

    NV-RAM, if it sees the stores in the wrong order and power is lost part way through, will be basically a scrambled file system, like we used to see back in the 80s before the PC OS vendors took write ordering seriously.
    Multi-session ISO9660 (basically a *VERY CRUDE* form of log-structured filesystem) did already exist and started getting widespread in the 80s.
    And I'm sure the FS buffs here will probably point you to various copy-on-write and log-structured filesystems that can handle such kind of scrambling.
    (Sorry I only know the modern ones).

    There are various ways to mitigate write re-ordering.

    Leave a comment:


  • lejeczek
    replied
    Intel team looked like all-Polish team?

    Leave a comment:


  • starshipeleven
    replied
    Originally posted by name99 View Post
    I have no idea what Palm were doing. But most of the commenters here (including you) don't seem to appreciate the technical points that are stake here.
    Finally someone that grasps what "persistent memory" is about, in this context.

    Leave a comment:


  • name99
    replied
    Originally posted by DrYak View Post

    They early 2000s' Palm PDAs would like to say "hi" to you.

    Note: Back then some other company had managed to land a patent on using flash memory as storage medium on PDAs. Instead of paying the patent, Palm went for using only RAM + batteries in their PDAs. Part of this memory is used as actual RAM for the programms, the rest is used as a form of persistent storage (Thanks to the battery) on which databases where used to store the data.

    In other words: it's been 20 years since mass produced computers have been using persistent RAM as storage.

    Even back then, this permitted extremely fast application swtiching, searching of storage, etc.
    I have no idea what Palm were doing. But most of the commenters here (including you) don't seem to appreciate the technical points that are stake here.
    The issue is not that NV-RAM is a faster storage medium, just like SSDs were faster than HDs.
    The issue is that because NV-RAM is
    (a) byte addressable (or perhaps cache-line addressable)
    (b) RAM (duh) as opposed to IO
    you can read and write to it without going through the OS mechanisms of files, an IO stack, interrupts, etc.

    So what does that mean? The first, trivial, thing it means is that, if you still want to use a file system (and most people will, at least initially) your file system will now be a user-space library, and protection will be implemented by the standard TLB page protection mechanisms (with all that may or may not mean for the finer granularity security that some file systems provide). But more important, robust file systems rely on data being persisted to storage in a very specific order. For IO that order is imposed by sticking fsync()s or something similar between specific writes. With current instruction sets, the equivalent instructions don't really exist or are not optimal. What you need is additional instructions that can be inserted between writes that will flush out previous writes to the persistent RAM. (Or provide other, smarter, ordering mechanism; but flushing is the simplest scheme so will probably be used first). Intel only recently added such instructions, likewise ARM in the ARMv8.2A ISA.

    The second thing it means is that existing file systems really should be altered to exploit the new capabilities NV-RAM offers. In particular not having to funnel everything through chunk-sized granules (chunks being 4 or 16kB or so) allows for a different set of tradeoffs in the design of the OS, buying you some percentage higher performance. MS has done substantial work in this, though, being MS, one expects that research will probably stay buried in Microsoft Research for twenty years before it finally, maybe, makes it into a replacement for NTFS.

    The third thing it means is that, over time, for at least some purposes, people will consider alternative storage models to files. Some devices did this in the past --- Newton "soups" were a kinda non-filed-based storage system, and various cloud storage schemes are likewise not file based (Apple offers key-value stores for apps using iCloud, and I'm sure so do various other APIs).

    But ALL these cool new ideas require something to enforce the ordering of writes out to the NV-RAM. Without that in place, you're playing amateur hour. That's the part that's missing right now from the actual CPUs in the field. And that's the part that requires careful new additions to every OS, whether Linux, Apple, Google, or MS.
    You can't just treat this stuff like a RAM-disk, because RAM-disks didn't persist if the CPU crashed/lost power half-way through a storage operation. NV-RAM, if it sees the stores in the wrong order and power is lost part way through, will be basically a scrambled file system, like we used to see back in the 80s before the PC OS vendors took write ordering seriously.

    Leave a comment:

Working...
X