Month: January 2015

The Systemd debacle

I’m late to write this, but perhaps better late than never (and truth be told, I’ve been neglecting this blog, largely because I prefer to be writing software than complaining about it, though I recently seem to have precious little time for either). If you’re reading this then you most likely already know about Systemd, the init-system-replacement-cum-kitchen-sink brainchild of Lennart Poettering and others (yes, they want me to call it “systemd”, but I’m averse, for some reason, to proper nouns beginning with lower-case letters; something to do with having had a moderately good education, I guess). Since its inception Systemd has gone on to become the primary, if not the only, choice of init system on a number of Linux distributions, and has more-or-less become a dependency of the Gnome desktop environment. You’ll also already be aware that not everyone is happy with this state of affairs.

Amongst other examples:

On the other hand, Systemd has its advocates:

  • Lennart Poettering, in his initial announcement of Systemd and how good its boot times are.
  • Lennart Poettering, trying to debunk some “myths” regarding Systemd (and apparently failing to recognize the mild irony of “debunking” both Myths #2 and #3, and of the fact that Lennart himself is largely responsible for Myth #2 due to the announcement linked in the previous point)
  • This blog post entitled “Why systemd?” by, err, Lennart Poettering.
  • This guy (from Arch Linux?), who I assume is not actually Lennart Poettering, though I can’t tell for sure.
  • LWN editor Jonathan Corbet basically saying that Systemd isn’t really that bad. Specifically: … The systemd wars will wind down as users realize that their systems still work and that Linux as a whole has not been taken over by some sort of alien menace.
  • Various linux distributions that are now using Systemd as their primary init system

There’s a fair amount of hyperbole on both sides, so who’s really right? And is the question a technical one or is it purely political?

On Lennart

Ol’ Lennart has received his fair share of criticism over the years. Here’s one which made me laugh, from one “HackerCracker” commenting on a Kuro5hin article:

That said, it seems there’s a putsch on to make Linux into Windows, complete with an inscrutable binary log, called SystemD. And $DEITY help you if you go criticizing it for its many faults, you will be pilloried as a Luddite, a moron, an emotional weenie without an ounce of brains as it is being written by a very, very, VERY intelligent man by the name of Lennart Poettering. The very same Lennart Poettering that brought the horror that is PulseAudio. And Avahi. And a host of other code abortions masquerading as THE NEXT BEST THING EVAR! to hit Linux. But the smiling fascist cheerleaders for this new SystemD paradise fail to see the incredible irony in their belligerently stupid marketing campaign. I guess it makes sense in a way, after all, it was Vladimir Ilyich Lenin who said ‘We shall win by slogans’ (paraphrasing).

(A mildly amusing aside: Googling for “+pulseaudio +horror” returns over 80,000 results).

Comparing Lennart to Lenin is ridiculous, but it’s fair to say that Poettering’s reputation precedes him. PulseAudio was widely criticized when it first arrived on the scene, for being both overly complex and, well, buggy as all fuck. I still don’t use PulseAudio on my system, mainly because I’ve never seen the need (although some software is starting to depend on it a bit, and to be fair I suspect the vast majority of bugs have, by this stage, been ironed out), and because the one time I looked at installing it, it became a dependency nightmare (I’m really flummoxed as to why it requires ConsoleKit [or, of course, Systemd] as a hard dependency, but that’s fodder for a future discussion, perhaps).

Lennart doesn’t help himself, I think, by being a bit of an arse. Not a huge arse, just a bit of an arse, but even being a bit of an arse is going to piss people off. He made a huge amount of noise about receiving death threats, at one point, which were apparently related to Systemd, claiming that “the Open Source community is full of assholes” [sic] and railing against Linus Torvalds and those associated with him. A ZDNet article covers the story pretty well, but I think this choice quote from Bruce Bryfield really sums it up:

… the complaints coming from Poettering amount to a new definition of chutzpah. Poettering, you may remember, is fond of sweeping critiques of huge bases of code, and of releasing half-finished replacements like PulseAudio, systemd, and Avahi that are radical departures from what they replace. He is a person as much known for expecting other people to tidy up after him as for his innovations. For many people, this high-handed behavior makes Poettering an example of the same abusive behavior that he denounces — and his critique more than slightly problematic.

Reading Poettering’s “Why systemd?” article is also revealing of character. The article essentially consists of a long list of Systemd features, with crosses marked for other init systems which don’t have those same features, which masquerades as an unbiased comparison. The only pretence of humility is at the end, where there’s a red mark against Systemd for lack of maturity. The whole thing reads a bit like a propaganda pamphlet, and the concluding remark – I believe that momentum is clearly with systemd. We invite you to join our community and be part of that momentum – helps to cement this perception. It’s a little bit creepy, really.

On the other hand, Poettering is by no means stupid. The initial announcement and discussion of Systemd is well worth a read, as it highlights many of the fundamental ideas behind the software, and some of it is in fact quite clever. Which brings us to the technical side of things.

Technical side

I don’t believe there’s necessarily anything wrong with Systemd on a technical level (but hold that thought, because the key word here is necessarily – I’ll elaborate on that a bit later). There are certainly some real problems that it seeks to address. The old “runlevel” system from Sys V init systems never made any real sense, and there were always issues managing service dependencies with such a system (if you want to run A and B, and B requires A to be started before it is itself started, how do you arrange that?). Although I don’t personally have much experience with Upstart, Poettering’s initial Systemd announcement gives a reasonable critique (if it is indeed correct). So there was space, I think, for an init system which provided proper service management; i.e. which allows starting and stopping individual services, and will automatically start their depencies / stop their dependents if required.

On the other hand, Systemd does a lot more than just provide service management. Some of these things are, I personally think, unarguably worthwhile or at least “not harmful”. Allowing parallel startup of services falls into this category, as does socket-based activation, where the service manager opens a socket on behalf of some service, and only starts the service when a client connects to the service (for one thing, this means the service isn’t actually running and consuming resources when it’s not actually needed, and for another thing, this simplifies handling of service dependencies so that they do not, in many cases, need to be explicitly configured).

There are other things Systemd can do that I consider might be going a little too far. Its use of autofs to allow filesystems to “appear” mounted before the filesystem checks and so forth have actually been run, for example, to me seems like excessive parallelization. On the other hand, it doesn’t hurt and I suppose that you can always just ignore the feature if you do not want to use it.

The use of cgroups to prevent processes from escaping their parent’s supervision (by daemonizing themselves) is probably a good idea, though this problem is I suppose as old as unix itself and one of the main reasons that it has had no solution up to this point is because there hasn’t, in general, been a pressing need. The only real benefit [edit Jan 2022: I neglected to mention resource control, which is pretty huge] I see is that it can be used to prevent users from leaving processes running on a machine after they’ve logged out, which is of course only an issue on true multi-user machines. For such a limited scenario, I have a lot of trouble understanding why Systemd has a hard dependency on cgroups.

In fact, most of the Linux features required by Systemd as listed in Poettering’s “Debunking myths” document (Myth #15: SystemD could be ported to other kernels if the maintainers just wanted to and Myth #16: systemd is not portable for no reason) are obscure enough that it remains unclear why these features are actually required. The “debunk” therefore completely fails – the myths remain undebunked (if that is actually a word). (Also, are there really two separate myths? These seem to be identical to me).

As well as the use of obscure Linux kernel features, Systemd requires DBus, and this is obviously unnecessary. So one valid critique of Systemd is that it has unnecessary dependencies. However, this is a not necessarily a strong argument against using Systemd.

[Edit 3/1/2016: The most important point that I missed when I first wrote this article is that Systemd crams a lot of stuff (including their D-Bus implementation) into the PID 1 process, a process which brings the whole system down if it crashes. This is a valid concern, and probably is the most legitimate technical concern raised against Systemd to this point].

The Human side

In fact, I had a lot of trouble actually putting my finger on what it was about Systemd that really bothered me so much. I generally disregard political arguments about software because I feel that technical merit should be the main focus. So, regardless of how much I might dislike Lennart Poettering’s manner, his habit of using hyperbole, and his apparent tendency of failing to provide rational and logical arguments, I’d normally be inclined to say that we should just swallow the bile, install Systemd on our systems and get on with it. Why does that seem so hard to do in this case?

I’ve read many complaints about Systemd; some of them are listed above, although they generally fail to provide a compelling technical argument against Systemd. If the reasons for wanting to avoid Systemd aren’t technical, can they still be valid? I’ve struggled with this question for some time. Here’s a few pieces which helped me to finally clear it up in my mind:

“Has modern linux lost its way?” (John Goerzen) – choice quote:

This is, in my mind, orthogonal to the systemd question. I used to be able to say Linux was clean, logical, well put-together, and organized. I can’t really say this anymore. Users and groups are not really determinitive for permissions, now that we have things like polkit running around. (Yes, by the way, I am a member of plugdev.) Error messages are unhelpful (WHY was I not authorized?) and logs are nowhere to be found. Traditionally, one could twiddle who could mount devices via /etc/fstab lines and perhaps some sudo rules. Granted, you had to know where to look, but when you did, it was simple; only two pieces to fit together. I’ve even spent time figuring out where to look and STILL have no idea what to do.

[Edit: ok, I realise that the above quote states that the issues are orthogonal to “the systemd question”, in the eyes of the author, and it’s true that the issues raised are not specifically about Systemd; nonetheless the general concepts very much also explain my concerns with Systemd; I don’t think that they actually are completely orthogonal.]

This resonates a bit with me. I’m worried about how using Systemd could render me unable to solve issues with my own system, without first having to delve deep into the internals of Systemd – which is something I’d rather not have to do, especially because the damn thing seems so hard to pin down, with new releases happening on a frequent basis. I feel like my init system should not do that. I want stability – both system stability, but stability in the sense that I want to have some assurance that I understand how the system works, and that I don’t have to follow every commit on this arrogant developer’s pet project just to keep that understanding valid. So much seems to have been crammed into Systemd in such a short space of time – it’s replacing init, udev, syslogd, inetd, cron, ConsoleKit, login/getty, network configuration, and recently even an EFI boot manager.

Fear of Change?

It could be argued that this argument against the adoption of Systemd is driven by fear of change. Fear of change is generally viewed with negative connotation, but of course it’s not an uncommon occurrence and is grounded in our past experiences. I’m not against change, and I think that Systemd is clearly a step forward in certain directions where I have, for some time, thought that some improvement would be nice. The problem is not the change, but that there is too much change, and too fast. Here’s a quote from Theodore Ts’o:

A realization that I recently came to while discussing the whole systemd controversy with some friends at the Collab Summit is that a lot of the fear and uncertainty over systemd may not be so much about systemd, but the fear and loathing over radical changes that have been coming down the pike over the past few years, many of which have been not well documented, and worse, had some truly catastrophic design flaws that were extremely hard to fix.

And yes, that’s definitely part of it. This isn’t the first time that some upstart has told us they had the solution to all our problems. Remember HAL? Remember Devfs? These were kind of a mess, but at some point or another the distributions were all using them. These eventually were superceded by Udev, which was at least fairly easy to understand, but now udev has been eaten by Systemd. And I mean, sure, you don’t have to use Systemd; as Poettering is so fond of pointing out, he’s not forcing anybody to do anything. The problem is that the distributions are jumping on the bandwagon and as a result we’re going to see hard dependencies emerge in future versions of software that we want to use.

In conclusion

The vitriol against Systemd is probably not warranted; the fault doesn’t lie with Systemd itself. But the dislike that people have for having lots of changes rammed down their proverbial throat is reasonable. Things can be improved, but it doesn’t have to happen all at once, and the choice shouldn’t be between accepting it all at once or get left behind. That distributions are adopting SystemD in its entirety is disconcerting. That some Debian folks were so strongly against this is, in fact, reasonable. But it’s not because there’s anything in particular wrong with Systemd; rather, it’s because we don’t really know what might be wrong with it. And we don’t know how much effort we’re going to have to make to contort our systems to work around Systemd which will later be made redundant when the next greatest whizz-bang system component comes along.