Systemd In Ten Years Has Redefined The Linux Landscape

[fsfhfc2018]

Finally being spiteful over the fact we have a corporate monoculture problem does not help anything. Its not good grounds to be anti-systemd as it really not offering anything better/more useful.

You know Oiaohm, I used to have a fair amount of respect for you-- elsewhere, as well as here. This is a purely circular argument you're making, and a lot of twaddle, and in the future I will just consider a shill and a troll.

You also accused me of not caring about such things before systemd existed, when we didn't know each other at the time. So on top of calling your post twaddle I'm going to say it's dishonest twaddle, and my respect for you has evaporated entirely. Of course I don't expect you to care. I've more or less said you're full of crap, and if not for shills like yourself, systemd wouldn't have come along this far.

Maybe you think it's reasonable to use something based on some touted (circular) technical merit despite being a complete horror supported by shills, but for me-- that's the exact opposite of what I signed up for.

I bid you and the other sophists on this horrifically designed forum adieu. There are some good posters here, I can't stand the lying and drivel and won't abide the trolling from people like yourself. Even someone who sculpts Disney characters out of turds for a living has got better things to do.

[yoshi314]

Not wanting a corporate monoculture to take over has nothing to do with "spite."

what i meant was when Lennart proposed introduction of /run directory people yelled "FHS violation. NAK!". that is the kind of resistance i was referring to.

i am not a fan of systemd being a kitchen sink project that it is nowadays, but they introduced some improvements that straightened out issues that nearly every distro swept under the rug:

- where do we keep volatile files?
- how does one provide an universal distro-agnostic init script for a software project?
- how to manage resources for services?
- how to check service status along with its most recent logs (if you have no idea where it is logging to)?

there are many better and lesser known issues that systemd inspired others to solve in the meantime. and not just that project benefited from it. i still think the authors pushed their weight around a bit too much (especially with kdbus attempt), but i think - all things considered - it turned out for the best.

[oiaohm]

You also accused me of not caring about such things before systemd existed, when we didn't know each other at the time. So on top of calling your post twaddle I'm going to say it's dishonest twaddle, and my respect for you has evaporated entirely. Of course I don't expect you to care. I've more or less said you're full of crap, and if not for shills like yourself, systemd wouldn't have come along this far.

I am not a shill pushing systemd. I am a person who has had to custom build distributions for embedded usage.

I have been around the Linux online world from 2000 before systemd basically no one really talked about the problem or if they did really did not understand the problem.

I might be wrong about you but most people who bring up that arguement are not aware of the realities.

There were people before systemd who did argue against the sysvinit related projects being put under 1 project on the false believe this would give redhat more corporate control than they already had.

i am not a fan of systemd being a kitchen sink project that it is nowadays, but they introduced some improvements that straightened out issues that nearly every distro swept under the rug:

This is also the problem people think the systemd kitchen sink is new. Reality its not new compared to the old kitchen sync operationally its better. sysvinit, udev, consolekit, glibc and many other parts have been developed under redhat roof. This use to lead to a horrible problem. What version of consolekit goes what what version of udev?? Remember there were under kitchen sink development as a single product by redhat. So distributions who were not redhat there maintainers were wasting hours/days/months basically reverse engineering what version of what should be with each other so it somewhat works. This kind of explains why so many distributions quickly changed to systemd it was instant drop in man hours required to get their distributions ready.

Reality here is systemd being kitchen sink has cleaned up that by in fact making the kitchen sink nicely clearly visible. Reality here is sysvinit was really like a kitchen sink over filled with plates what are projects random-ally stacked and the kitchen sink barely visable to the outside observer who is not the kitchen sink redhat but appears perfectly ordered to the redhat side and 100 percent in the kitchen sink.

Basically systemd single project move in fact gets everyone on a level footing.

- how does one provide an universal distro-agnostic init script for a software project?

Linux standard base attempted to-do this but the failure was in fact caused by the messy kitchen sink. Its one of those funny ones LSB init scripts for sysvinit now in fact work perfectly under systemd based distributions where they were pot luck to work under sysvinit distributions before systemd existence. LSB universal init script is 2002.

So its not how do we provide universal distro-agnostic init scripts as this was in fact kind of worked out before systemd it was more how do we stop distributions adding unique one off bugs to their init and service management process. Yes distributions having their maintainers having to reverse what was the right mix of parts to put together was the biggest cause of unique one off distributions bugs in the init and service management process. Fixing this in fact freed up resources to go after some other problems.

The systemd kitchen sink is in fact one of the true big improvements. Systemd kitchen sink makes it clear where the kitchensink is. This also shows how much you really need to implement to make a init/service management solution that work correctly of course this is a lot larger than a lot of people before systemd attempt to implement to replace sysvinit and failed.

Yes this is another one of the biggest complaint about systemd as well that to make a replacement init/service management solution is going to be harder since systemd is now a kitchen sink instead of the prior multi plates. Reality is no harder now if you want working. There were a lot of attempts that just produced not practically usable init and service management options because they were not implementing anywhere near enough.

[S.Pam]

systemd is no saviour. For those that claim that is just a collection of separate applications that work together should take another look how difficult it was to get logind to work stand-alone from systemd in order to use Gnome with OpenRC.

On the topic of systemd is also a post here https://forums.gentoo.org/viewtopic-...ighlight-.html

----------------------------------------------------------------------------------------------------------------
|                          |systemd                                   |s6/s6-rc                                |
|--------------------------------------------------------------------------------------------------------------|
|Simple service definition |Perhaps yes, but with lots of gotchas [b] |Yes, by chainloading [a]                |
|Support for cgroups       |Yes, hardcoded in init                    |Doable, in better ways [d]              |
|"Socket activation"       |Yes                                       |Yes, by fd holding [a]                  |
|Service logging [c]       |Inefficient, SPOF, binary logs            |Efficient, reliable, textual logs       |
|Separate /usr             |Unsupported, for questionable reasons [1] |Supported                               |
|Portability               |Only Linux with glibc                     |Linux (with glibc, musl, ...), BSD, ... |
|--------------------------------------------------------------------------------------------------------------|
|Speed ([b] for all below) |Hangs that are hard to predict            |Consistently fast                       |
|Simplicity                |Big, very complex due to tight coupling   |Tiny, loosely coupled                   |
|Reliability               |Easy to begin, but with lots of gotchas   |Easy to learn and predictable           |
|Maintainability           |Number of unresolved bugs growing quickly |Very few bugs, resolved quickly         |
|Flexibility               |Hardly extensible, components unreusable  |Easily extensible, components reusable  |
|Security                  |Many vulnerabilities                      |Few (if any) vulnerabilities            |
|--------------------------------------------------------------------------------------------------------------|
|[a-d] See the "Basic background", "Quality", "Logging" and "Linux cgroups" parts below, respectively.         |
|[1] <https://forums.gentoo.org/viewtopic-p-8354032.html#8354032>.                                             |
----------------------------------------------------------------------------------------------------------------

[oiaohm]

systemd is no saviour. For those that claim that is just a collection of separate applications that work together should take another look how difficult it was to get logind to work stand-alone from systemd in order to use Gnome with OpenRC.

Key problems there is elogind went for absolutely stand alone this does cause issues when you attempt to mix systemd and elogind. It also created a lot more work thinking elogind directory tree layout does not match the parent systemd logind so you cannot port out fixes either. Taking an application out of a collection can be very hard. Its like taking a tool out of gnu coreutils. That is a poor example as they have created trouble for themselves how they have done it.

On the topic of systemd is also a post here

Bias post and very factually incorrect post.

Service logging [c] |Inefficient, SPOF, binary logs |Efficient, reliable, textual logs

Lets bring in a few horrible facts here. This was written in 2019. The writer is still clueless.

https://elinux.org/images/6/69/Demystifying_Systemd.pdf

This is from 2016 when someone decide to see how much of a diet systemd can be put on if you are serous about it.

Funny enough journald is a optional part if you are building up from source. So if systemd got a few patches to add a runtime option to disable journald it could run without it on normal distributions as well.

https://lists.fedoraproject.org/archives/list/[email protected]/thread/3YRVHE4G47B6GSVVLZ6L75IVUHSEBLYW/

Also 2016 people also worked out that it was possible to turn journald off when it was built in. You had to setup replacement right so everything was collected as it need to be instead of disappearing into the never to be seen again void or triggering journald to socket activate..

To collect from all the logging sockets and get to textual logs be it s6/s6-rc sysvinit or systemd you have to run something that is a SPOF be it a program like rsyslog or systemd journald. Yet here in 2019 referencing that this is a difference. It was documented in 2016 that this is not in fact a difference. Documentation to setup replacement to journald is kind of not well known.

So the logging difference is myth. Systemd gets little upset if you have not set your logging right where s6/s6-rc will ignore the problem and losing your logging information to the void of no return.

Yes that write up referenced 2013 bug to make point things do change over time. Basically it was 2016 when embedded developer look close and found out the journald requirement of systemd was basically myth. Lot of ways the way Logging has been done on Unix and Linux these days really need to go completely back to the drawing board to reduce the SPOF risk.

Support for cgroups |Yes, hardcoded in init |Doable, in better ways

Hmm problem this is not understanding the problem space.

A more important use of cgroups in systemd is to track the process trees of services, in order to deal with orphan processes of badly written service programs after the main service process exits, and to deal with services that do not support non-fork()ing execution.

This is one use of cgroups. But not the only reason why systemd has implemented cgroups the way they have. Turns out the scheduler in the Linux kernel to work effectively needs groups. Be them cgroups or autogroup created extra. Yes autogroup is have the kernel guess what need to be grouped this does not work out that great it gets it wrong badly effecting performance. Since you have to have cgroups so the kernel can allocate out processes to cpu cores and memory locations correctly for efficiency it only makes sense to proceed on to use cgroups to assist with your badly written unruly services.

To handle unruly services, a cgroup-based babysitter program can be written to fork()/exec() the given service process in the specified cgroup. If the service supports non-fork()ing execution, the babysitter then waits for its child (the main service process) to exit, kills processes remaining in the cgroup, and then exits; if not, the babysitter then waits for the cgroup to be empty, and then exits. As far as I know such a program is not yet implemented, but I believe that I have outlined enough above for a practical implementation of the requirement, and that such a program will be available soon if enough people seriously demand it.

This is bonus feature of systemd that helps with problems as it comes up. Notice bonus the day in day out feature of cgroups with systemd is more stable performance of services due to the kernel scheduler having more useful information to make it selections with. Also notice how the feature is not in s6/s6-rc and is like hey it will appear when there is people seriously demand it. Reality here the people who want this feature are using systemd and are not going to waste time duplicating it. This is basically a checkbox you need to check. Not say someone could write the later. Notice completely missed that all services need to be grouped so that scheduler in these massive multi core systems knows what should be grouped with each other because there could be information transfer between them.

cgroup started of with one idea and with cgroupv2 and its adding features it being expand to-do the unruly services stuff better. Including apply bpf filter over a cgroup so everything inside cannot get out even if it has root privilege. Cgroups did not start designed as an item to handle unruly services but is very quickly coming the best tool for the job.

Simple service definition |Perhaps yes, but with lots of gotchas [b] |Yes, by chainloading [a]

Except that chainloading has a very big gotcha for service management. How do you restart the minder process to load in new libraries without restarting the service that it minding. This was always the downfall of deamontools. So chainloading for service management is something you wish to avoid if you can because you are in fact increasing every services dependency tree by chain-loading. One way to avoid this nightmare is cgroups so having the kernel track the service instead of a program you start.

Separate /usr |Unsupported, for questionable reasons [1] |Supported

Early systemd did have this feature. It been clear if people wanted to work on the separate /usr stuff upstream systemd is not past taking patches on this. Does the reason why we had split /usr and boot stuff really make sense any more.

Portability |Only Linux with glibc |Linux (with glibc, musl, ...), BSD, ...

Portability problems are true but s6 is not giving the Linux kernel the information it in fact requires to function correctly. If a kernel has cgroups you really need to use them. Portability is a common excuse not to use cgroups.

Speed ([b] for all below) |Hangs that are hard to predict |Consistently fast

This is false statement about s6. s6 has performance problems on massive multi core systems with Linux so not Consistently fast and this is all due to lack of cgroup usage so resulting in screwing up on massive multi core or complex NUMA setups.

Simplicity |Big, very complex due to tight coupling |Tiny, loosely coupled

Tiny is not excuse for being feature poor.

Reliability |Easy to begin, but with lots of gotchas |Easy to learn and predictable

Some truth here but s6 is not a predictable at times due to again not using cgroups(scheduler issues caused by not using it) and not having any form of proper process tracking so depending on services todo the right thing. Reality large percentage of services including highly popular databases don't do the right thing all the time.

Maintainability |Number of unresolved bugs growing quickly |Very few bugs, resolved quickly
Security |Many vulnerabilities |Few (if any) vulnerabilities

Not really sure this is valid to draw either of these results when you are comparing to something that is feature poor.

Flexibility |Hardly extensible, components unreusable |Easily extensible, components reusable

Kind of true.

Notice out of all those point only 2 turned out to be somewhat true.

I would like to see decent competitor to systemd but s6 at this stage absolutely is not it. Yes we have a big problem of people not understanding that cgroup usage is basically not an optional feature if you want everything to work the best it can with the Linux kernel.

[cybertraveler]

systemd is no saviour. For those that claim that is just a collection of separate applications that work together should take another look how difficult it was to get logind to work stand-alone from systemd in order to use Gnome with OpenRC.

On the topic of systemd is also a post here https://forums.gentoo.org/viewtopic-...ighlight-.html

----------------------------------------------------------------------------------------------------------------
| |systemd |s6/s6-rc |
|--------------------------------------------------------------------------------------------------------------|
|Simple service definition |Perhaps yes, but with lots of gotchas [b] |Yes, by chainloading [a] |
|Support for cgroups |Yes, hardcoded in init |Doable, in better ways [d] |
|"Socket activation" |Yes |Yes, by fd holding [a] |
|Service logging [c] |Inefficient, SPOF, binary logs |Efficient, reliable, textual logs |
|Separate /usr |Unsupported, for questionable reasons [1] |Supported |
|Portability |Only Linux with glibc |Linux (with glibc, musl, ...), BSD, ... |
|--------------------------------------------------------------------------------------------------------------|
|Speed ([b] for all below) |Hangs that are hard to predict |Consistently fast |
|Simplicity |Big, very complex due to tight coupling |Tiny, loosely coupled |
|Reliability |Easy to begin, but with lots of gotchas |Easy to learn and predictable |
|Maintainability |Number of unresolved bugs growing quickly |Very few bugs, resolved quickly |
|Flexibility |Hardly extensible, components unreusable |Easily extensible, components reusable |
|Security |Many vulnerabilities |Few (if any) vulnerabilities |
|--------------------------------------------------------------------------------------------------------------|
|[a-d] See the "Basic background", "Quality", "Logging" and "Linux cgroups" parts below, respectively. |
|[1] <https://forums.gentoo.org/viewtopic-p-8354032.html#8354032>. |
----------------------------------------------------------------------------------------------------------------

Interesting post, thanks.

[arokh]

I don't have spite.

Every anti-systemd nutjob.

[rodne1379]

I liked service status more than systemctl status, service scripts have less boilerplate now, the new security bugs should have been avoided with rust. otherwise systemd is not changed anything for me.

wait i will check it and let you know exact