Announcement

Collapse
No announcement yet.

IBM Proposing A CPU Namespace For The Linux Kernel

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • IBM Proposing A CPU Namespace For The Linux Kernel

    Phoronix: IBM Proposing A CPU Namespace For The Linux Kernel

    IBM engineer Pratik Sampat published an early prototype of a CPU namespace interface for the Linux kernel. This CPU namespace was devised to address coherency issues with current means of viewing available CPU resources as well as addressing possible security issues stemming from understanding resource access/positioning on the system...

    https://www.phoronix.com/scan.php?pa...-CPU-Namespace

  • #2
    In principle I love the idea about virtual mapping, but I fail to see how this differentiate itself from the recent core scheduling work in practice. Would it not achieve mostly the same thing? ... That being said I think that CPU namespace sound like the correct way forward.

    http://www.dirtcellar.net

    Comment


    • #3
      Fantastic idea. This issue can become a pretty significant pain point in highly heterogeneous compute environments. However part of me feels like if they are going to redesign this interface from scratch why not take it a step further and solve another big issue in this space...

      Scheduling tasks in cgroups with XXXX millicores becomes incredibly difficult when you have significantly different cpu types deployed in production. For example, maybe your nginx frontend needs 4 cores on a 4ghz system, but you still have 2ghz core nodes in service on which nginx needs 8. Adopting some sort of standard compute unit makes it possible to abstract this away in such a manner that you can wind up with consistent performance across your fleet.

      I know for a fact from first hand experience that at least one major tech company has rolled their own custom implementation of this to great effect, but an upstream standard would benefit everyone.

      Comment


      • #4
        Wow, I always thought that docker really restrics CPU cores..
        Well, it seems not
        Code:
        docker run --cpuset-cpus="0" -it python bash
        [email protected]:/# python -c "import os; print(os.cpu_count())"
        16
        I always used `os.cpu_count()` in my code when multiprocessing is required.
        And other libraries do the same
        Last edited by RedEyed; 10 October 2021, 08:08 AM.

        Comment


        • #5
          Originally posted by RedEyed View Post
          I always used `os.cpu_count()` in my code when multiprocessing is required.
          And other libraries do the same
          Correct, which is exactly what this idea would solve.

          Comment


          • #6
            Well, now I will always use
            Code:
            docker run --cpuset-cpus="0" -it python:3.9 bash
            [email protected]:/# python -c "import os; print(len(os.sched_getaffinity(0)))"
            1

            Comment


            • #7
              Originally posted by RedEyed View Post
              Wow, I always thought that docker really restrics CPU cores..
              Well, it seems not
              Code:
              docker run --cpuset-cpus="0" -it python bash
              [email protected]:/# python -c "import os; print(os.cpu_count())"
              16
              I always used `os.cpu_count()` in my code when multiprocessing is required.
              And other libraries do the same
              A bit archaic these days, most things are adopting async libraries now. Let the library/runtime worry about multiprocessing, like that old saying of let the libraries worry about data structures and algorithms. I see people saying you flat-out should never make a thread manually anymore (unless you need a long-running thread for a dedicated task as to not hog the threadpool or block the async runtime, but that's more of an architectural concern and not a multiprocessing one, but I digress)

              Comment


              • #8
                Originally posted by Ironmask View Post

                A bit archaic these days, most things are adopting async libraries now. Let the library/runtime worry about multiprocessing, like that old saying of let the libraries worry about data structures and algorithms. I see people saying you flat-out should never make a thread manually anymore (unless you need a long-running thread for a dedicated task as to not hog the threadpool or block the async runtime, but that's more of an architectural concern and not a multiprocessing one, but I digress)
                I know what I do.
                I use multiprocessing for computing tasks, not IO.

                I don't know, why you proposed async without even figuring out the task?

                Comment


                • #9
                  Originally posted by RedEyed View Post

                  I know what I do.
                  I use multiprocessing for computing tasks, not IO.

                  I don't know, why you proposed async without even figuring out the task?
                  Async is for both compute and I/O

                  Comment


                  • #10
                    Originally posted by Ironmask View Post

                    Async is for both compute and I/O
                    async can't be used for compute in python at least
                    Last edited by RedEyed; 10 October 2021, 01:43 PM.

                    Comment

                    Working...
                    X