Announcement

Collapse
No announcement yet.

systemd Clocks In At More Than 1.2 Million Lines

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • aht0
    replied
    No "fireworks" here. Somewhat informative rather.

    Leave a comment:


  • r08z
    replied
    Originally posted by grumbert View Post
    and here was me thinking this thread would not deliver fireworks since hreindl got banned
    This is the best thing that has happened to this forum in a while. I think a premium member was the one who reported it since Michael only listens to premium members about concerns.

    Leave a comment:


  • oiaohm
    replied
    Originally posted by aht0 View Post
    You tested it when? Year-wise? Lots have changed over the years (ca 20 FreeBSD point-releases since systemd was conceived).
    I have been monitoring this init issue since 1994 so so every Freebsd release from then to now.

    Originally posted by aht0 View Post
    Let's say I employ FreeBSD but using rctl. Leaving jails out of it for now.
    Does not in fact work to block leaks you need to use jails to set up a subject-id that then rctl can work with to track down and kill leaking processes.

    Originally posted by aht0 View Post
    How come Solaris has pidfile issue? It derives from BSD and you remarked on Page 9
    This is one of those historic goofs that happens when Solaris was implementing zones and SMF. They effectively lowed the security that low on zones that you could not open file handle on PID to make pidfile process unstable. So for a short time Solaris managed to create the same level of screw up Linux kernel had from day one. Yes they attempted to fix it with zones at first before fixing the goof up with signalling.

    So systemd developers and solaris SMF developers basically attempted to fix a broken pidfile problem the same way with cgroups/zones. Solaris SMF developers were able to get kernel level cooperation a lot sooner to fix zones and the pidfile break caused by the implementation of zones.

    Yes there are lot of events that happened with SMF at it start that were fixed inside the first 12 months for SMF on Solaris that have taken a decade with Systemd on Linux. So I am not highly impressed with the Systemd or Linux kernel rate of progress on these problems.

    This is why it would be a good idea to have a pidfile test case on BSD so a goof like this does not ever get though.

    Leave a comment:


  • oiaohm
    replied
    Originally posted by aht0 View Post
    There are design issues, though it seems to be more subjective matter.
    Stuff process leaks and not sending signals right and not handling hotplug and other things not right. Are not subjective problems. These a items you can create test cases for and test init systems against and watch them break or not break.

    Originally posted by aht0 View Post
    It's opponents would be MUCH happier if using systemd was truly optional, not 'you don't want systemd? HOW dare you. Prepare for huge extra effort getting past it, working out shims..". Systemd is consciously designed in a way that makes using alternatives hard. That could have been avoided and would have make opposition to it much lighter.
    You get it wrong the systemd response is why the hell are you using a broken init system. Not how dare you use alternative. You see the difference when you start talking about openrc with systemd people they are like that not ready yet but it could interesting. Openrc has design things where it could work.

    This is kind of a bias point of view. The lead init system has always changed things. sysvinit particular things when it was dominate made it way into consolekit and udev and other parts making alternative hard. Systemd is not really making alternatives init work any harder than sysvinit use to.

    Originally posted by aht0 View Post
    Whatever benefits Linux kernel would eventually reap, it's going to end up with a sort of Lindows - all distros are akin to others. Distro-hopping used to be interesting, always something new. Now, sigh.. hundreds of clone-distros and it's only going to worsen. Prediction: amount of distros is going to decrease, eventually leaving only handful of big ones. Plus handful systemd-less.
    This is rose coloured glasses. Go back and look closely at all those distributions you use to distribution hop between and notice how many back then were sysvinit.
    https://distrowatch.com/table.php?distribution=debian
    You can look though distrowatch. You will fairly much see that all that has happen is sysvinit distributions have become systemd distributions.

    No one was complaining when Redhat was lead developer on all the sysvinit distribution core parts from 1998 to 2013. Yes this is sysvinit, consolekit, udev.... basically all the stuff systemd replaced. Reality dominate force in Linux since 1998. Yet people have only worked this out after systemd was released. It been well and truly over due for people to get there head out the sand on this one. I was trying alternative init systems back in 1998 so I really do know how much sysvinit particular built in quirks got in way.

    There are more distributions than we have man power to maintain. So you use to distro hop to get away from one failure and move to the next distribution and hit another pack of failure. Why because you were hopping between distributions without enough resources to maintain themselves. Always something new is being rose coloured glasses there was always something new that works better as well as something new that was busted. So if the number of distributions in fact reduce this will be a good thing. Instead of making massive number of distributions we need to focus on making the core ones quality. This is truly the rescission we need to have.

    aht0 really why people backing systemd don't take your point serousally is what you are saying is not backed by historical fact. If you hated redhat/ibm controlling init development you really should have been complaining and working on alternatives since 1998 and have them fairly much good quality with a decent pool of developers behind you. Or will you simple admit that you had your head in the sand completely ignoring how much control over init development Redhat had.

    Of course to fight for alternative init will truly mean having some developer resources. Not just throw a init system out there that don't work work and con users into use it then wonder why you never get many distributions using it.

    Most of the systemd-less distributions are using init solutions that are known to be broken. Lets keep on using old broken instead of making something that in fact works and can truly go one to one with systemd in service management and win.

    Simple point stop going boo hiss.... and put something that when a person like me looks at it we cannot rip holes in it.

    "this is windows like" is commonly uses against systemd. Systemd design is not windows like. Systemd in fact Solaris like so it should be stop making Linux like Slowaris. Yes Slowaris does party come from it detecting process leaks and being slow to respond at times.

    "sort of Lindows" As soon as you use this you basically say you don't know the history of where systemd comes from. Systemd is a mix of launchd from OS X and SMF from Solaris. Windows init system is broken in lot of the same way historic BSD and Linux init systems are plus extras.

    Leave a comment:


  • aht0
    replied
    Originally posted by oiaohm View Post
    (a) I have personally tested on all of those. Solaris with SMF can also deal with process leak. (b)Cgroup and Zones deal with process leak. (c)Attempting to use cgroup/zones to deal with pidfile issues does not work out. Sun documented process leak in Solaris as one of the justification for having SMF use Zones. The first platform to deal with the process leak problem is Solaris.
    (a)You tested it when? Year-wise? Lots have changed over the years (ca 20 FreeBSD point-releases since systemd was conceived).
    (b)Let's say I employ FreeBSD but using rctl. Leaving jails out of it for now.
    (c)How come Solaris has pidfile issue? It derives from BSD and you remarked on Page 9
    Freebsd you have always been able to open a process as a file handle so locking that PID from change from the point of view of your application. In fact the first BSD kernel had this feature and the Linux kernel only starts getting that in kernel 5.1.
    Last edited by aht0; 05-26-2019, 05:16 AM.

    Leave a comment:


  • aht0
    replied
    Originally posted by oiaohm View Post
    I will give you systemd had way more bugs than it should be this is implementation not design issues. Items like depinit on paper implemented ideally with no defects fails on the design table because you don't have enough information in particular places to make sane choices. Fairly much all the init options designs for Linux before systemd falls into the camp of even if ideally implemented is still broken.
    There are design issues, though it seems to be more subjective matter. It's opponents would be MUCH happier if using systemd was truly optional, not 'you don't want systemd? HOW dare you. Prepare for huge extra effort getting past it, working out shims..". Systemd is consciously designed in a way that makes using alternatives hard. That could have been avoided and would have make opposition to it much lighter.

    Whatever benefits Linux kernel would eventually reap, it's going to end up with a sort of Lindows - all distros are akin to others. Distro-hopping used to be interesting, always something new. Now, sigh.. hundreds of clone-distros and it's only going to worsen. Prediction: amount of distros is going to decrease, eventually leaving only handful of big ones. Plus handful systemd-less.

    Leave a comment:


  • oiaohm
    replied
    Originally posted by lkcl View Post
    depinit (written by richard lightman) tackled and solved this issue in a much cleaner way. firstly, daemons spawned as background tasks were treated as "legacy": the default preferred option was (is) to run them as foreground processes, and depinit had the option to farm the stdin, stdout *and* stderr separately to *separate* (dependent) services. safe_mysqld, as with all daemon-spawning, was therefore entirely unnecessary and redundant.
    Sorry you have missed that deamons like apache, cups, postgresql and oracle db do spawn background tasks themselves. So this alteration does nothing major to fix process leaking.

    Originally posted by lkcl View Post
    secondly, depinit captured *all* signals, and i do mean *all*. nothing escaped, except unfortunately if testing depinit recursively (yes, this was possible), the linux kernel rather unsportingly sends all daemon-spawned signals (of orphaned processes) down to PID1, unconditionally.
    That message to PID1 does not tell you want service that orphaned process in fact came from.

    Originally posted by lkcl View Post
    depinit was (when run as PID1) capable of hunting down even fork-bombs and malicious viruses, by deploying ever-increasingly-aggressive hunt-and-kill strategies that were SLOWLY escalated.
    And killing process that have been children of background processes started by the like of cups/apache before they have completely their job.

    Originally posted by lkcl View Post
    basically, it was extremely well-engineered, and was only a few K-lines of code.
    The different true attempts to fix the PIDFILE signal and process leak problem on Linux that kind of worked as long as everything happened to go the solutions way.
    1) Upstart attempts ptrace solution this turned out to work but it also hit performance hard and made programs with anti-debugging fail and when you wanted to debug way harder.
    2) Systemd attempted cgroup + PID namespace solution to the PIDFILE problem where you cannot send signals to the right process this turns out to be ponies all the way down(this is a wacky developer quote of the problem).

    The systemd PIDFILE solution ponies all the way down.
    1) attempt to kill process start termination process inside PID namespace of target.
    2)termination process send signal.
    3) target process fails to die.
    4) termination process locks up
    5) go 1 now you have 2 processes to send signals to to kill
    Yes a well and truly growing forkbomb the PID namespace stopped from sending signal to the wrong process but ends up when things go wrong with a growing forkbomb.

    Mind you the cgroup solution works perfectly for tracking what has started what. Remember when a process is tagged with a cgroup and it loses it parent and gets connected to the PID1 you can query what service this is from. Once you know what service you can look up if that service should be running or not running and make a way smarter move than depinit blind guess.

    Originally posted by lkcl View Post
    basically, it was extremely well-engineered, and was only a few K-lines of code.
    Depinit not extremely well-engineered.
    List of problems
    1) PIDFILE problem hits this thing quite hard. Before 5.1 Linux kernel delivering a signal to correct process is kind of pot luck. This does not implement the BSD process protection either. So this broken be you using a Linux or BSD kernel or any other unix/posix kernel made in the past 2 decades..
    2) It does not in fact address the process leak problem correctly due to being overly aggressive to make up for not knowing enough.

    Depinit along with many other init need to go into the bin of historically broken. Either never used again or someone rewrites to at least deliver signals correctly on bsd kernels and on 5.1 and newer linux kernels. Basically depinit a few K-lines of code that does not work right.

    Originally posted by lkcl View Post
    systemd is unconscionable, end of discussion, and its carte-blanche deployment, IN DIRECT VIOLATION OF A DEBIAN VOTE, will continue to haunt us until distros give *all* users the FULL right not to be forced into using it.
    The big thing to remember maintainer in distributions have the right not to be constantly getting bug reports about issues caused by init under many different services. Like having a report that cups is randomally eating print jobs or after restarting cups not being able to print. They should not have to put up with this. Dropping supporting all broken init solutions fixes this problem.

    Basically I want init options that when you look at process leak and signal handling in fact stand a chance of working correctly. When you look at init systems for Linux with this requirement the list comes very short very quickly. Systemd and openrc in what you would call init systems. You would also have like docker and other container management solutions.

    https://www.debian.org/vote/2014/vote_003
    You really need to read the vote carefully.
    Maintainers are encouraged to accept technically sound patches to enable improved interoperation with various init systems.
    Key words here is "technically sound" a init system that lacks proper process leak handling is not technically sound so the patch to add support for that init system is automatically not technically sound so maintainer can opt out of including it. Carte-blanche deployment with no competition with systemd is a direct result of no one getting a quality init system up yet. The idea that systemd is unconscionable forgets giving users broken by design init systems is really unconscionable as well.

    Its about time we get really strict on the requirements init systems have to pass. After 5.1 Linux kernel correct signal delivery should be expected. Currently correct tracking and handling of process leaks should be expected. Any init system not above this bar should either be scrapped or rewritten.

    Yes bsd with jails and correct signal delivery should be expecting super good init system ideas.

    I will give you systemd had way more bugs than it should be this is implementation not design issues. Items like depinit on paper implemented ideally with no defects fails on the design table because you don't have enough information in particular places to make sane choices. Fairly much all the init options designs for Linux before systemd falls into the camp of even if ideally implemented is still broken.

    Leave a comment:


  • lkcl
    replied
    Originally posted by oiaohm View Post
    5) check if any processes the service started are still running if there are still running you have a process leak. This is when systemd starts printing the timeout error and SMF prints Invalid service shutdown waiting X before terminating Y.
    depinit (written by richard lightman) tackled and solved this issue in a much cleaner way. firstly, daemons spawned as background tasks were treated as "legacy": the default preferred option was (is) to run them as foreground processes, and depinit had the option to farm the stdin, stdout *and* stderr separately to *separate* (dependent) services. safe_mysqld, as with all daemon-spawning, was therefore entirely unnecessary and redundant.

    secondly, depinit captured *all* signals, and i do mean *all*. nothing escaped, except unfortunately if testing depinit recursively (yes, this was possible), the linux kernel rather unsportingly sends all daemon-spawned signals (of orphaned processes) down to PID1, unconditionally.

    depinit was (when run as PID1) capable of hunting down even fork-bombs and malicious viruses, by deploying ever-increasingly-aggressive hunt-and-kill strategies that were SLOWLY escalated.

    basically, it was extremely well-engineered, and was only a few K-lines of code. systemd is unconscionable, end of discussion, and its carte-blanche deployment, IN DIRECT VIOLATION OF A DEBIAN VOTE, will continue to haunt us until distros give *all* users the FULL right not to be forced into using it.


    Leave a comment:


  • oiaohm
    replied
    Originally posted by aht0 View Post
    (a)Hold your horses for a second! You are basing this claim on what precisely? Have you actually tested it against OS X, Windows XP - 10 RS5, all 4 BSD's, Solaris..?
    I have personally tested on all of those. Solaris with SMF can also deal with process leak. Cgroup and Zones deal with process leak. Attempting to use cgroup/zones to deal with pidfile issues does not work out. Sun documented process leak in Solaris as one of the justification for having SMF use Zones. The first platform to deal with the process leak problem is Solaris.

    Process leak is quite simple. Its some process the service starts that when restart/stop on the service is run is not shutdown. Since this service was not shutdown it can be holding resources that are required that the service can work correctly.

    Windows XP to 10 the most common form of process leak issue happens with winspool the print server and it drivers.
    https://windowsreport.com/print-spoo...ce-windows-10/
    Person can run through this complete list and still be having random printer problems. The process leak problem temporary goes away when ever they reboot the machine. Rebooting windows to fix problems is classed as normal behaviour and they don't bother bug reporting it any more.

    Interesting enough different third party printer drivers for cups with OS X and the BSD also process leak. If you don't mind percentage of your users having random printer issues where they have to reboot their systems to fix don't deal with this problem. BSD based solutions could move to always putting services in individual jails so become able to detect process leak.

    Originally posted by aht0 View Post
    First, I don't really think there is actually such an issue present. I tried searching information about it but came up blank. Just no information. Problem either does not exist, expresses itself extremely rarely && people never bothered writing about it.Kindly provide me with pointers, please.
    This is basically wrong. The problem expresses is self a lot. People understand how to detect a memory leak. People do not normally understand the processes to detect process leak.

    Every one scripts that you saw systemd timing out on had a process leak. You will find the makers of those programs would have been using the same service shutdown process on BSD and had a process leak over there as well.

    Process leak is fairly straight forward to check for. cgroup/zones make it lighter.

    Process to find a process leak.
    1) Set up means to keep track of all processes a service/s start. Systemd cgroup and Solaris Zones do this role. Ubuntu upstart attempted ptrace for this role and it did not work out.
    2) start the service.
    3) Expose to normal workload.
    4) stop service.
    5) check if any processes the service started are still running if there are still running you have a process leak. This is when systemd starts printing the timeout error and SMF prints Invalid service shutdown waiting X before terminating Y.

    Please note point 3 just straight up starting and stopping the service may not show the process leak even that the service will process leak. Running though everything a normal workload will do cannot be done by the developer. So this test need to be performed by end users.

    You get benign process leaks that all they do is consume a PID number and some memory so basically a different form of memory leak.
    But you also get harmful process leaks that have locked resources that stop you from restarting service correctly this is like the winspool and cups that result in not being able to print even after restarting print server. But its not only print services this has happen with database services, web services and so on. Anything that executes sub programs as a service can lose track of what it started.

    Process leaks is one of these background problem that for longer than Linux has existed people have not been properly running diagnostics for. Remember SMF in Solaris using Zones with SMF was the first to go after this problem. We do need to learn from this and improve our service management systems not to ignore this problem.

    The systemd timeout message could be a lot more bunt and clear what the problem is.

    Leave a comment:


  • aht0
    replied
    Originally posted by oiaohm View Post
    LOL this is such a joke of answer. (a)Process leak issues with background services effects OS X, Windows, BSD, Linux ...
    Unless you are using SMF from Solaris or Systemd init at this stage you are lacking the tools to deal detect the defect.
    Really we cannot ever 100 percent ditch the diagnostic software. With (b)cgroup and pidfile stuff fixed up items like openrc could be built and have the required diagnostics to detect this issues early.
    (a)Hold your horses for a second! You are basing this claim on what precisely? Have you actually tested it against OS X, Windows XP - 10 RS5, all 4 BSD's, Solaris..? Or it's just an extrapolation based on what init/service manager any of the OS'es in question happens to have. In latter case, it's speculative. Fact that Linux happens to have particular issue does definitely not mean everything else has. None of OSes in question has any historic relation to Linux nor share any relevant code historically.

    (b)Cgroup is purely Linux-specific implementation. None of the others share this particular way of managing resources.

    Originally posted by oiaohm View Post
    (a)Also the 1.2 million lines of code in systemd is way smaller than freebsd min selfhost. If you are suggesting BSD as option you are a hypocrite because (b)it has a large block of diagnostic software..
    You know, replace your term "self-host" with a "core system". More easily understandable and it's more fitting. You won't see FreeBSD in "pieces" anyway.

    (a) Are you very sure in your claim? Because there's drastic difference to "FreeBSD installed by default" and "FreeBSD updated by compiling". I'll explain, it's bit lengthy but nothing to do about it..
    -RELEASE is updated by binary packages - lots like your average Linux distro. What components are included by FreeBSD team you'll get.
    -STABLE is something like rolling release but wholly compiled. You'd pull in latest code revision, build new world and kernel, do mergemaster -Ui, reboot and voila, upgrade done. Trick is, based on what's written in your /etc/src.conf - your system build may exclude lots of core system components you deem unnecessary.

    https://www.freebsd.org/cgi/man.cgi?query=src.conf shows you possible options for building upgrades. Based on this, I suspect, IF I wanted to build truly minimal FreeBSD core system, LOC ratio 'FreeBSD vs systemd' would take serious adjustment and not for latter's favor. Current FreeBSD standard install source loc should be around 10 million somewhere. IF I stripped out everything, leaving just FreeBSD kernel, libc and init, I am pretty sure, code base left would be smaller than systemd loc stand-alone

    But, why are you comparing systemd (stand-alone) vs FreeBSD (complete OS) loc? systemd itself is not stand-alone. It, at minimum, needs kernel as well. And, oh boy, kernel is 25+m loc. Let's be fair. systemd+kernel vs freeBSD base install - latter's loc is almost three times less.

    When we take comparison further, though it's pointless... lets compare Linux distro vs FreeBSD. Recent Debian distro installed is something like half a billion loc.

    (b) Partially correct. It has. But are testing/debugging bits installed? Read what I wrote above. It's much like Gentoo or Arch. I can 'massage' the OS in au quite few ways. You don't have to guess what I run. Besides, excluding non-used components from core system reduces build time spent on system upgrades, there's incentive to exclude bits.


    Originally posted by oiaohm View Post
    This is the laugh this is the hard reality of systemd its find (a) issues that have been silent faults in background that random-ally hit people. Systemd is also working on the frameworks required to address these silent faults.
    (a) Issues for Linux kernel, mind you. Which has been developed on it's own. You seem to think that almost all OSes have identical flaws hidden inside them and only systemd can find them..

    Do you know that BSD pkill and Linux's pkill aren't even strictly same program? Share name, do similar thing, true but authors differ, code both are based on, differs.. All BSD's differ internally. Bold claims.


    Originally posted by oiaohm View Post
    aht0 serous-ally look closely at your current solution how would you find a process leak event and know you are going to need to fix it. Under freebsd you will have to put services in jails to find out. Out the box you will not be informed you have a problem.
    First, I don't really think there is actually such an issue present. I tried searching information about it but came up blank. Just no information. Problem either does not exist, expresses itself extremely rarely && people never bothered writing about it.Kindly provide me with pointers, please.

    Originally posted by oiaohm View Post
    So since aht0 is happy to live with random hard to diagnose problems I cannot stop him/her from being that foolish.
    IF I had problems, I'd look for alternatives
    And FIY, I am ~40y male.
    Last edited by aht0; 05-24-2019, 04:25 PM.

    Leave a comment:

Working...
X