Systemd Continued Commanding Linux Systems In 2015

[cjcox]

Once again I failed to notice the apocalypse. None of the machines under my supervision experienced these stability issues or "anomalies", in fact managing them became easier. Maybe if I survived it, I can help others. Can you point to the bug reports that discuss the issues you're having? I guess there are quite a lot of them, maybe other people here can help you fix the issues. Also, can you elaborate on this "event driven -> lot of things get hung during shutdown" issue? Haven't encountered it yet.

What system do you use to manage your deployments? I have to use puppet and it's very painful. Puppet (for example) assumes that all service startup go through systemctl, which messes up legacy things done through init.d instead (in puppet these will fail). Of course, we can blame puppet (heaven knows we can't blame systemd without being crucified).... In the Red Hat case the easiest way to deal with most (but not all) of the system resource problems is by telling all of puppet to handle service resources the old way... why? Because the old way in this transition period handles both cases. The other cases in your puppet modules you have to edit case by case.

Ok... I know, you systemd guys are thinking "we are right" (can they think any other way?)... and therefore it's a puppet problem. I'm just pointing out that systemd (to use systemd speak) "has exposed many problems in other peoples software". And we, the sysadmins and devops of the world, have to pay the price. I know that all systemd proponents have convered their ears at this point... I fully understand it's my problem to contend with, but to say "it isn't because of systemd" is just incorrect. It's just life and life got a whole lot more painful during this transition.

I am glad you haven't seen the lockup on shutdown... if you ever do see this, will you be honest enough to post? Just because I'm not afraid to point systemd failings doesn't make me your enemy. Or does it?

Again, these are full lockups, so difficult to diagnose (especially with systemd...sorry, but must be said).

[Candy]

... but, why in the world you ever go as ineffective as loading log in text editor? the only thing i could imagine being even more ineffective is probably someone loading those logs in libreoffice, printing them to pdf and then inspecting in adobe acrobat

Don't get me wrong. I am no system administrator.

I simply use the desktop edition of Fedora (and previously other distributions targeted for the same audience). The log files I deal with are basicly the ones on my own desktop installation. They had always been around ~10mb (and even far less than that ... say 4-5mb) (considering the fact that I often delete the contents after a while or simply re-install a backup to keep the system clean).

Therefore loading up a file of that size inside an editor was a quite common and quick task. Can it be done differently ? Yes clearly. Of course things might look differently if you need to deal with servers and hundrets of user accounts that keep spitting out huge amounts of log contents. So we may end up talking about different use cases here.

I do see that journald may be an interesting thing for servers and for administrators that need to wade through a lot of content. But for the ordinary workstation installation (aka desktop installation) using journald (which easily generates a couple of hundret megabytes of binary blobs) is plain overkill.

[SystemCrasher]

You might be aware of this, so this is just an addition in case someone isn't: It is not only less convenient but actually impossible (where impossible =~ requires breaking modern cryptography) because the journal applies recent research results by Dr. Poettering

That's what I referred to, exactly. I've been taking a look some ages ago and I remembered overall idea. Yet, thanks for referring to pdf with details, good thing to read for sure

@Nille_kungen: this is a technical forum. If someone publicly states technically inaccurate criticism of the other peoples work, he should expect be called out for that.

I've noticed systemd haters aren't very good in technical aspects, they are good at personally attacking Poettering or those who think he haves a point. Or at very most such people behave like a kids who failed to get idea: nobody promised to keep them happy by doing what pleases them. Poettering is perfectly free to come with some new conception (which isnt actually new, Solaris had it way before, Canonical did it for Linux in half-assy way, etc). System maintainers and everybody else are free to give it a try and decide what they like, and there is no reasons why it should please someone, nobody haves such a silly goal. If it happens to be systemd... okay, there is no way to prevent maintainers going for it anyway. Except by forking and doing all stuff on their own, which means a lot of work. That's where we come to key idea. The only reliable way to get something done in way it pleases you is to do it yourself. Everything else is very optional, relies on free will of other individuals and do not have to be taken as granted, to say the least. Seems some people are bad at understanding it. Else I do not get why they behave like most systemd haters do.

Hey, haters, Linux is all about choice. And nobody takes away your freedom. But you do not have freedom to order others what to do, because it violates their very basic freedoms. So if it happens they are doing not what you want, feel free to take your freedom and do it yourself. You are perfectly free to do a shitload of unpaid routine work to back your preferences & ways, if you think this crap really matters that much. Others may or may not join you. This is really up to them to decide and it is invalid to yell on them just because they made different choice. You have a choice, as long as you can come and take it. But sometimes it takes doing heck a lot of crap yourself. We did it when there was upstart and sysv init in debian, so if I preferred upstart, I had to struggle hard to get rid of sysv init. Now it is your turn to face the same fate. Fair enough, isn't it?

[tinko]

What system do you use to manage your deployments? I have to use puppet and it's very painful. Puppet (for example) assumes that all service startup go through systemctl, which messes up legacy things done through init.d instead (in puppet these will fail). Of course, we can blame puppet (heaven knows we can't blame systemd without being crucified).... In the Red Hat case the easiest way to deal with most (but not all) of the system resource problems is by telling all of puppet to handle service resources the old way... why? Because the old way in this transition period handles both cases. The other cases in your puppet modules you have to edit case by case.

Ok... I know, you systemd guys are thinking "we are right" (can they think any other way?)... and therefore it's a puppet problem. I'm just pointing out that systemd (to use systemd speak) "has exposed many problems in other peoples software". And we, the sysadmins and devops of the world, have to pay the price. I know that all systemd proponents have convered their ears at this point... I fully understand it's my problem to contend with, but to say "it isn't because of systemd" is just incorrect. It's just life and life got a whole lot more painful during this transition.

I am glad you haven't seen the lockup on shutdown... if you ever do see this, will you be honest enough to post? Just because I'm not afraid to point systemd failings doesn't make me your enemy. Or does it?

Again, these are full lockups, so difficult to diagnose (especially with systemd...sorry, but must be said).

I don't have enemies on Internet boards. I have never experienced a lockup. I do have experience of service that failed to start or stop properly, but those are killed after a timeout of 90 seconds which can easily look like a lockup. This value is configurable through /etc/systemd/system.conf. If a hanging service does not get killed, you might have indeed found a bug in the core of systemd, which should be filed.

As for your legacy init scripts, there might be an easy way around this issue. You can create simple "wrapper" unit files that define services that are started and stopped via scripts inside init.d . Unit files can be very short, have an easy syntax and are very readable. You might be interested in the following link http://blog.ls-al.com/systemd-using-...-init-scripts/

I would of course prefer to remove the init.d-script entirely and do the whole thing via an unit file, in that case this might be your link: http://0pointer.de/blog/projects/sys...-admins-3.html . Writing a good unit file could also be a chance to make use of many useful systemd features and maybe you can commit it back to the upstream of your service..

[Candy]

Linux is all about choice. And nobody takes away your freedom.

No offense but this sentence is utterly old and should be printed out, framed by a picture frame and nailed to a wall. The times where Linux is all about choice are long gone. Even the second sentence telling people to code their own stuff (to say it simply with my own words) is also over. Linux turned out to be far too complex and people can only work on subsets of it. Some do kernel development, other people focus on driver development, others code tools and applications and again others focus on the whole picture of desktop development. There is a lot of controverse stuff going on and people will continue disagreeing and having controversal conversations. The amount of times people told me to code my own XYZ implementation of XYZ would probably turned out to become an own individual operating system by now. People need to find a consense (?) with their controversal conversations and find a middle way to get things done. Sadly - and this is really hurting - there are a bunch of people who abuse their positions by enforcing their own view of things.

[SystemCrasher]

Therefore loading up a file of that size inside an editor was a quite common and quick task.

Sure. And so we remember Linux and other *nix like are generally multi-tasked, multi-user systems. Everyone can launch editor and edit logs. At which point they aren't logs anymore, but piece of science fiction. With no way to check whether event took its place or if it has been edited to conceal something, etc. This is especially rough if there is break-in. If program writes some file, it means it can access it. That's where attacker can hijack log entries, e.g. hiding entry point of attack, so we neither have idea there is breakin, nor we can trace entry point of attack. What you refer to is more like debugging output, tracing and so on rather than system logging. Sure, line is quite blurry, but it does not excuses poor state of security. And your way of thinking reminds me Win95, when there was no these "stupid" access rights, no stupid multiple users, no way to separate resources, and everybody can pwn whole system at will. The difference? WinCIH was able to erase flash ROM in Win95, but failed to do it in NT, thanks to decent kernel-user separation and access rights. So it seems "stupid" things can sometimes save butt quite a lot, aren't they? And you can be sure *nix systems were aware of it even before NT has appeared. So I'm sorry to inform you, but win95 ways of thinking are considered obsolete even by MS for something like 16 years, and *nixes were well ahead of NTs. In fact many concepts of NT based systems were borrowed from *nix systems. E.g. UAC is nothing more than weird implementation of sudo, etc.

[tinko]

What system do you use to manage your deployments? I have to use puppet and it's very painful. Puppet (for example) assumes that all service startup go through systemctl, which messes up legacy things done through init.d instead (in puppet these will fail). Of course, we can blame puppet (heaven knows we can't blame systemd without being crucified).... In the Red Hat case the easiest way to deal with most (but not all) of the system resource problems is by telling all of puppet to handle service resources the old way... why? Because the old way in this transition period handles both cases. The other cases in your puppet modules you have to edit case by case.

Ok... I know, you systemd guys are thinking "we are right" (can they think any other way?)... and therefore it's a puppet problem. I'm just pointing out that systemd (to use systemd speak) "has exposed many problems in other peoples software". And we, the sysadmins and devops of the world, have to pay the price. I know that all systemd proponents have convered their ears at this point... I fully understand it's my problem to contend with, but to say "it isn't because of systemd" is just incorrect. It's just life and life got a whole lot more painful during this transition.

I am glad you haven't seen the lockup on shutdown... if you ever do see this, will you be honest enough to post? Just because I'm not afraid to point systemd failings doesn't make me your enemy. Or does it?

Again, these are full lockups, so difficult to diagnose (especially with systemd...sorry, but must be said).

Somehow my last post got lost, so I'll be brief.
No, I've never hat a lock up during shutdown except for one case long ago on a desktop machine that was related to some nvidia driver issue with the machine dieing the moment Xorg exited.
I did have services hanging during shutdown, systemd will kill them after 90 seconds, which might look like a lock up if one is unaware of that. This timeout can be changed via /etc/systemd/system.conf iirc. I cannot rule out that you have found a bug that causes systemd to fail to kill the processes, but in that case it should be filed, because it does sound serious.

Your puppet issue may be easy to resolve. (I think I've heard about systemd looking into init.d if no service file was found but maybe this has changed or I'm wrong)
Here is my suggestion:

a) write a short "wrapper" unit file, that starts and stops the service by calling the init.d-script, check out this easy example http://blog.ls-al.com/systemd-using-...-init-scripts/
b) write a proper unit file, and while you're at it you can benefit from some systemd features that are hard to do via shellscripts, like private tmp directories and stuff like that. In this case, this is the link to follow: http://0pointer.de/blog/projects/sys...-admins-3.html

[Candy]

This is especially rough if there is break-in.

If someone compromised the system, then the binary *non changeable* logs won't help anymore. Once broken in (and root access granted), the person may (and mostlikely will) cause far more damage to the system than you might extrapolate from the logs. Not to mention that the person could simply rm -rf the entire journald directory or simply issue some journactl --vacuum-size=10 magic...

[tinko]

If someone compromised the system, then the binary *non changeable* logs won't help anymore. Once broken in (and root access granted), the person may (and mostlikely will) cause far more damage to the system than you might extrapolate from the logs. Not to mention that the person could simply rm -rf the entire journald directory or simply issue some journactl --vacuum-size=10 magic...

The point of this has been explained in this thread already. The person that breaks in will not be able to hide the fact. If your journal is flushed, you know something is wrong and that is realy the best any logging system can do for you in the case of a server break in.

If someone has broken into your system would you like your logs to be devoid of traces of the break in or not? Because I would like to know that someone has broken into the system rather than have his hidden spy daemon send him all my data or corrupt my further work while I'm cluelessly using the machine, unaware that anything has happened.

[pal666]

So if someone doesn't come to the same conclusion as you he must be an imbecile.

if someone wants to open database in text editor he must be an imbecile
if someone is raving about non-unixy monolithic systemd, when systemd is base system and freebsd's base is much bigger and much more monolithic, he must be an imbecile

People that doesn't like systemd mustn't be stupid nor incompetent.

they are stupid not because they don't like systemd, but because they don't like it due to their incompetence

I can't understand why this hate against people that doesn't like systemd.

because they are imbeciles who also happen to dislike systemd