GNU Shepherd 1.0 Service Manager Released As "Solid Tool" Alternative To systemd

[oiaohm]

One server could be down for 20 hours and no-one would care because the other 200+ would pick up the slack. 200 ~90% servers are still cheaper and *more reliable* than a single 99.999999% server making compromises for boot-up performance.

That matters in Ultra high availability like share trading. Split brain issues with data. If server is done for 20 hours it could be holding a transaction that the other 200 don't know about. You are right that what you suggested is cheaper. But when you issues are split brain with data its not more reliable stopping a server for 20 hours. These split brain issues is why 8 nines mainframe hardware hardware exists. Yes the hardware where you can hot plug everything.

Besides, neither SMF or systemd are particularly fast. Mainly because they come with terrible hacks that could be solved by a "slow and steady" shell script.

https://elinux.org/images/6/69/Demystifying_Systemd.pdf

Above shows that we use a lot more systemd parts than we need to in most cases.

Yes the move over to varlink instead of dbus for more things in systemd will remove the dbus requirement todo fancy things.

kpedersen the biggest thing that makes it tricky is the Linux kernel recycling of PID and breakaway support in process tree. shell scripts are not designed to work with pidfd or cgroups. You really do need to use both of these on Linux.

pidfd to make sure that when you send a instruction to a PID it goes to the process you in fact intended it so that another process had not started in it place.
cgroupsv2 is to counter the breakaway in the PID tree. X application starts Y application under Linux Y application can come its own thing not linked to X at all this is the break away.

Slow and steady shell script cannot solve the problems coming from Linux kernel design that comes from Linus originally miss reading a book on posix.

Yes we do need a new kill command and a few other things for Linux. Something like

safekill PID "/bin/this" startime

Where safekill command uses pidfd to connect to the PID thenchecks that PID is the right executable and has the right start time then sends the terminate message so counting Linux PID recycling.

Yes then tooling to operate cgroupv2 effectively. There have been syscalls added to the Linux kernel to make starting processes in a new cgroup clean and simple and as single call to kernel but you cannot use this from a shell script.

Shell script on Linux starts running into being a round peg in a square hole so not working exactly right.

Yes this could be fixed by making a new shell script language with a few Linux particular addons to deal with Linux PID recylcing and PID tree breakway..

Its not like SMF solves the whole "slowaris" image did it? Arguably Solaris 9 booted faster too.

Solaris 9 is when SMF was introduced it booted faster than Solaris 8 that was using shell scripts. So that not argument there are benchmarks of it.

Core systemd and core SMF is not that slow either.

https://www.youtube.com/watch?v=yjf1Ft9fS_Y

You have people who go and benchmark openrc vs systemd and find opps systemd is faster.

Yes a systemd system starting slow is more a product of I need to clean out what I don't in fact use. Others have done this with busybox and runit and quite a few other init systems. Systemd itself is not that slow compared to shell scripts.

Shell scripts are slow due to how often you are having to call the kernel to start another process then wait for that process to end before you can go forwards this adds up. Cost of starting the process and cleaning up the process when it ends is free.

A scripting language design perfectly for service management where starting process that will have to quickly terminate are minimized could possible be faster than systemd but this is not your shell scripts.

Yes the idea that shell script init/service management is fast is not in fact backed by benchmarks there is a fundamental issue that makes shell script init/service management slow..

[kpedersen]

99.999999% ("eight nines") 315.58 milliseconds per year 26.30 milliseconds per month

The number of nines per month is just marketing masturbation. Not even NASA or any other project that does something useful needs that. Simply because we don't design for single points of failure anymore.

One server could be down for 20 hours and no-one would care because the other 200+ would pick up the slack. 200 ~90% servers are still cheaper and *more reliable* than a single 99.999999% server making compromises for boot-up performance.

Besides, neither SMF or systemd are particularly fast. Mainly because they come with terrible hacks that could be solved by a "slow and steady" shell script.

Its not like SMF solves the whole "slowaris" image did it? Arguably Solaris 9 booted faster too.

[oiaohm]

So you test in production
I meant testing before applying the change not when something is wrong. If something went wrong, you roll back to the last known good state.

99.999% ("five nines") 5.26 minutes per year 26.30 seconds per month.

This is where thing start getting hard.
99.999999% ("eight nines") 315.58 milliseconds per year 26.30 milliseconds per month
This is where thing are insanely hard where you are needing automated repair.

If you try get 5 9s on a single host, you play with fire.

Yes these are not single host systems. You may test a configuration on the +1 or more systems yes the hot spares. Yes common N+1 but in 5 9 it common to be N+3.

At 8 nines just doing lots of transfers from production to hot spares can in fact consume you yearly allowance of downtime.

Guix is a kind of an immutable system so stuff has to change in chunks and this also means, that you can't just go, login via SSH and start to monkey around.

This is where you run into problems. High 9 uptime as in over 5 nines having configuration management solution log into production systems and update service settings and have services restart in a controlled order can be a requirement. What you called monkey around has it place.

If you're interested in Guix in production and managing uptimes, then I would advice you to get in contact with the folks at https://hpc.guix.info/about/ that run Guix HPC clusters.

HPC is not high uptime. 4 nines is normally enough for HPC clusters.
99.99% ("four nines") Downtime per year 52.60 minutes Downtime per month 4.38 minutes

The workflow you are talking about works fine in 4 nines. Starts coming problem at 5 nines and is absolutely unworkable at 8 nines. Systemd and SMF design is about working at the horrible levels of 8 nines and above.

https://www.redhat.com/en/blog/kernel-live-patch-support-cadence-update

Remember you are live patching the kernel so you don't have restart and take the downtime hit at 5 nines and above..

The way operate true High-availability cluster and High performance computing clusters is in fact different. Guix is more design for the high performance computing clusters.

Systemd and SMF design is more how to meet the absolutely maxed out High-availability cluster. Solaris/Sun OS just like Redhat supports running OS kernel patching. System designed for very high nines will have this feature. You want to avoid service disruption as much as possible,

HPC can have lot more disruptions its that the processing completes at some point.

Guix currently is not ultra high availability cluster OS and shepherd is currently not designed for it. Yes the bit you call monkey around this does come need to keep your downtime as low as possible..

[slalomsk8er]

BTW, I'm playing with Guix and learning Guile on my old ThinkPads and not on Servers yet but the immutable nature of Guix and the tooling for building containers is tempting me to also put it into production, when I'm comfortable enough with it.

The problem comes when you are trying to get 99.999 and above uptime. You are trying to have a mess up as issue target recoverable. Like you screw up ssh and you only want its uptime counter effected and so on.
...
Yes testing in vm when something wrong is consuming time that high nines uptime does not always give you.

So you test in production
I meant testing before applying the change not when something is wrong. If something went wrong, you roll back to the last known good state.

The problem is having this paradigm work for the 99.999 and above uptimes. This is where it gets horrible because sections of the system will be classed in 99.999 systems are nice to have but in failed at the moment not critical because them not working is not part of what the server must provide at 99.999+ uptime.

If you try get 5 9s on a single host, you play with fire.

Yes the guix design you alter system.scm and you are effectively playing with everything is a problem.

Guix is a kind of an immutable system so stuff has to change in chunks and this also means, that you can't just go, login via SSH and start to monkey around.

Yes if you do more than one scm where do you define the lines that they don't clash with each other.

Are you a programmer? Guile is a programming language and supports modules.
As in all projects, you have to test and debug you code before you deploy it

If you're interested in Guix in production and managing uptimes, then I would advice you to get in contact with the folks at https://hpc.guix.info/about/ that run Guix HPC clusters.

[WorBlux]

Technically speaking, systemd's ini-style files are not the only way to create units. They can be created programatically via API, and indeed much of the internal stuff is created that way.

For what it's worth, I have often been unhappy with the limitations of ini files and their tepid text templating. Doing anything sophisticated is hard if not impossible. It could be nice to be able to use a domain-specific language instead, like Shepherd does (and unfortunately requires).

I personally would not pick Guile (a dialect of Scheme, which is a dialect of LISP). But imagine using Lua or JavaScript to write your own generator for systemd units.

There is a good idea here.

Guile is the official GNU scripting language, and the Guix package manager also uses guile. You aren't going to get too deep into modifying a GUIX system without running into Guile elsewhere.

[oiaohm]

If you really messed up, you can just go back to a good generation in the boot menu.

The problem comes when you are trying to get 99.999 and above uptime. You are trying to have a mess up as issue target recoverable. Like you screw up ssh and you only want its uptime counter effected and so on.

This is a different workflow. Systemd design and SMF design both have going after this uptime and performance as objectives.

Remember uptime is not always being measured on all services as well. So sshd being down might not be important as long as the base management controller is working.

Yes testing in vm when something wrong is consuming time that high nines uptime does not always give you.

.Compartmentalization of systemd and smf configuration with the multi files thing starts making sense.

As I said, it's a completely different paradigm as you create new system generations as you go and creating them as containers and images for testing, before your deploy to your actual running system, is easy.

The problem is having this paradigm work for the 99.999 and above uptimes. This is where it gets horrible because sections of the system will be classed in 99.999 systems are nice to have but in failed at the moment not critical because them not working is not part of what the server must provide at 99.999+ uptime.

Yes the guix design you alter system.scm and you are effectively playing with everything is a problem. Yes if you do more than one scm where do you define the lines that they don't clash with each other.

The multi file configuration system per service starts making sense in 99.999 where different services have different repair requirements and that configurations may be being altered temporary to restore some functionality that has to function for 99.999.

Making a highly reliable service management system that flexible enough for all use cases is not easy.

[slalomsk8er]

The empty value is so its not append.

[Socket]
​ListenStream=127.0.0.1:8080

That would leave the existing Listen-stream in place

This did cost me some headscratching and documentation searching. From my programming experience I expected var=value to overwrite and would have opted to use var+=value to append but oh well.

That has a downside what happens if your edit program on system.scm decides delete the file and crash when it done that.

Yes there are a few edit programs that have the following workflow.

1) create a working file
2) do a move command to replace original.

Power failure or crash at wrong point no file. What happens with Guix if the system.scm disappears. There is a durability reasons and authorization reasons to the many files method of systemd. Of course having a single file to back up to have all the system configuration in one file has advantage for backing up and duplicating configuration.

Nothing happens to a Guix system if the file disappears as it's the recipe to build a generation of the system.
I bet, most users leverage Git to version control their changes and even if not, every generation of a Guix system includes the file that defined it in the guix store.

Scheme does not lend it self the way ini or xml files do to configuration stacking with existing tools.

Please also note this configuration stack thing SMF of Solaris does the same with XML files and .d directories. Lot of XML and INI configuration services support the .d directory for stacking and appending to configuration. The fact it can be done with xml means it would be somewhat possible with scheme but it would require some different processing.

Also think about it slalomsk8er the .d where you can do a small change see if it works or not and if it does not work delete the file you added to the ,d directory so putting yourself back where you started is useful.

Systemd has brought in a few good ideas from SMF.

The .d directory thing came about to reduce cases of editing some bit of software configuration file and completely breaking the thing. This is something I find bad that shepherd is missing.

Not sure about that as Scheme is somehow quite able to allow me overwriting the defaults in my system.scm file.

As I said, it's a completely different paradigm as you create new system generations as you go and creating them as containers and images for testing, before your deploy to your actual running system, is easy.

https://guix.gnu.org/manual/en/html_node/Invoking-guix-system.html#index-virtual-machine

If you really messed up, you can just go back to a good generation in the boot menu.

[oiaohm]

It's clunky to first need to empty the value but nice to be able to overwrite a value in the configuration from the distribution package.

The empty value is so its not append.

[Socket]
​ListenStream=127.0.0.1:8080

That would leave the existing Listen-stream in place

Shepherd in practice, at least under Guix, is configured with a completely different paradigm as you define all that in your system.scm file and don't need to define this stuff all over the place in overwrite files.

That has a downside what happens if your edit program on system.scm decides delete the file and crash when it done that.

Yes there are a few edit programs that have the following workflow.

1) create a working file
2) do a move command to replace original.

Power failure or crash at wrong point no file. What happens with Guix if the system.scm disappears. There is a durability reasons and authorization reasons to the many files method of systemd. Of course having a single file to back up to have all the system configuration in one file has advantage for backing up and duplicating configuration.

Scheme does not lend it self the way ini or xml files do to configuration stacking with existing tools.

Please also note this configuration stack thing SMF of Solaris does the same with XML files and .d directories. Lot of XML and INI configuration services support the .d directory for stacking and appending to configuration. The fact it can be done with xml means it would be somewhat possible with scheme but it would require some different processing.

Also think about it slalomsk8er the .d where you can do a small change see if it works or not and if it does not work delete the file you added to the ,d directory so putting yourself back where you started is useful.

Systemd has brought in a few good ideas from SMF.

The .d directory thing came about to reduce cases of editing some bit of software configuration file and completely breaking the thing. This is something I find bad that shepherd is missing.

[slalomsk8er]

now final exmaple of override with systemd lets place a alter.conf ssh.socket.d .

[Socket]
ListenStream=
​ListenStream=127.0.0.1:22

Changed listerning to ipv4 localhost only. notice again have not changed ssh.socket file.

This kind of configuration modification without in fact altering the package provided files is where systemd comes into its own.

It's clunky to first need to empty the value but nice to be able to overwrite a value in the configuration from the distribution package.

Shepherd in practice, at least under Guix, is configured with a completely different paradigm as you define all that in your system.scm file and don't need to define this stuff all over the place in overwrite files.

For example, the shepherd service for opensshd is defined in https://git.savannah.gnu.org/cgit/gu...s/ssh.scm#n557 and configured by putting an openssh-service-type into your system.scm.

[oiaohm]

[QUOTE=slalomsk8er;n1511741]
Yes, the keywords ('#:') vs comments (';') are something one needs to get used to while learning Guile.

(define sshd
(service
'(sshd ssh-daemon) ;the secure shell daemon
#:start (make-inetd-constructor ;start on demand
'("/usr/sbin/sshd" "-D" "-i")
(list (endpoint
(make-socket-address AF_INET INADDR_ANY 22))
(endpoint
(make-socket-address AF_INET6 IN6ADDR_ANY 22)))
#:max-connections 10)
#:stop (make-inetd-destructor)
#:respawn? #t))

(register-services (list sshd))
(start-in-the-background '(sshd))​​​

vs.

From openssh-server_9.7p1-7ubuntu4_amd64.deb and could be cleaned up for sure to only provide the same features but it's at minimum 2 files and more verbose anyway even without the boiler plate of Guile code around it.

ssh.socket
ssh.service
[email protected]

https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

slalomsk8er it would be all three. [email protected] results in each connection getting it own cgroup with it own name with systemd and this can have been hardened.

Do note you said min of 3 files. Lets say you want to change one line in those files of systemd.

ssh.socket.d ssh.service.d [email protected] directories can be created then a file placed in one of those directories depend on the line you wish to replace 2 to 3 lines.

Yes lets say you have placed the following alter.conf in the ssh.service.d directory.

[Service]
ExecStart=
ExecStart=/usr/sbin/sshd -D

ExecStart is something in a systemd service file that can be lists more than once so to clear the page you have to do a null line. Yes this has disabled using the environmental value for how sshd starts up.

Now that would require a change in [email protected] so a alter.conf in the [email protected] to match up.

[Service]
ExecStart=-
ExecStart=-/usr/sbin/sshd -i

Something should now stand out.

/usr/sbin/sshd" "-D" "-i" from the shepherd is not a 1 to 1 match to /usr/sbin/sshd -i and /usr/sbin/sshd -D

Then you open up the manual.

https://linux.die.net/man/8/sshd

-D' When this option is specified, sshd will not detach and does not become a daemon. This allows easy monitoring of sshd.

And

-i' Specifies that sshd is being run from inetd(8). sshd is normally not run from inetd because it needs to generate the server key before it can respond to the client, and this may take tens of seconds. Clients would have to wait too long if the key was regenerated every time. However, with small key sizes (e.g. 512) using sshd from inetd may be feasible.

Yes I like how opensshd documentation tells you not to run in -i mode. 3 files with systemd files you pulled with openssh is that it can be run in two different ways and the shepherd has 1 only one way.

Yes /usr/sbin/sshd" "-D" "-i" could be called a 1 to 1 match /usr/sbin/sshd -i with the -D being added to get around a shepherd limitation. Remember this is a mode you are told not to use by default by opensshd documentation.

This is not a new problem where a init alternative end up not reading the documentation of the software they do their examples with and do there example against recommend from upstream settings.

The thing about systemd ini is that it was chosen so you can use .d directories to override items in service files while still allowing update to change those files. The ini choice of systemd is not just about readability.

now final exmaple of override with systemd lets place a alter.conf ssh.socket.d .

[Socket]
ListenStream=
​ListenStream=127.0.0.1:22

Changed listerning to ipv4 localhost only. notice again have not changed ssh.socket file.

This kind of configuration modification without in fact altering the package provided files is where systemd comes into its own.