Let’s Talk about Service Dependencies

(aka: Escape from System D, part IV).

First: anyone who’s been keeping tabs will have noticed that there hasn’t been a lot of progress on Dinit recently; this has been due to multiple factors, one being the hard disk drive in my laptop dying and this impeding my ability to work on the train to and from work, which is when I usually found time to work on Dinit. However, I’ve by no means abandoned the project, will hopefully have a replacement laptop soon, and expect the commits to resume in due course (there have been a small number made recently, in fact).

In this post I wanted to discuss service dependencies and pros and cons of managing them in slightly different ways. In an earlier post I touch on the basics of service management with dependencies:

if one service needs another, then starting the first should also start the other, and stopping the second should also require the first to stop.

It’s clear that there are two reasons that a service could be running:

  1. It has been explicitly started, or
  2. It has been started because another service which depends on it has been started.

This is all very well, but in the 2nd case, there’s an open question about what to do when the dependency service stops. There are two choices in this regard:

  1. A started service remains running when its dependencies stop, even if the service has not itself been explicitly started, or
  2. A started service automatically stops when its dependencies stop (unless it has itself been explicitly started).

Which is the better option? The first option is probably simpler to implement (it doesn’t require tracking whether a service was explicitly started, for instance); the second option, though, has the nice properties that (a) it doesn’t keep unneeded services running and (b) explicitly starting and then stopping a service will return the system to the original state (in terms of which services are running). Also, if you want to emulate the concept of run levels (which essentially describe a set of services to run exclusively), you can do so easily enough; switching run level is equivalent to explicitly starting the appropriate run level service and stopping the current one.

(Systemd makes a distinction between service units, which describe a process to run, and target units, which group services. However, I’m not sure there’s a real need for this distinction; services can depend on other services anyway, so the main difference is that one has an individual associated process and the other doesn’t. Indeed Systemd’s systemctl isolate command can accept a service unit, although it expects a target unit by default. Dinit on the other hand makes no real distinction between services and targets at this higher level.)

There are some complications, though, which necessarily add complexity to the service model described above. Mainly, we want some flexibility in how dependency termination is handled. The initial “boot” service, for instance, probably shouldn’t stop (and release all its dependencies as a result) if a single dependency (let’s say the sshd server, for example) terminates unexpectedly; similarly, we wouldn’t necessarily want boot to be considered failed if any of a number of certain dependency services failed to start. On the other hand, for other service/dependency combinations, we might want exactly that: if the dependency fails then the dependent also fails, and if the dependency stops then the dependent also stops.

Other problems we need to solve:

  • It may be convenient to have persistent services that remain started after they are started (due to a dependent starting, even when the dependent stops. For instance, if we have a service which mounts the filesystem read/write (from read-only) it’s probably convenient to leave it “running” after it starts, since undoing this is complicated and may be error-prone.
  • Boot failure needs a contingency; it should be possible to configure what happens if some service essential for boot fails (whether it be to start a single-user shell, reboot, power off, or simply stop with an error message).

With all the above in mind, I’ve narrowed down the necessary dependency types as follows:

  • regular – the dependency must start before the dependent starts, and if the dependency stops then the dependent stops.
  • soft – the dependency starts (in parallel) with the dependent, but if it fails or stops this does not affect the dependent. It’s not precisely clear that this dependency type is necessary in its own right, but it forms the basis for the following two dependency types.
  • waits-for – as for soft, but the dependent waits until the dependency starts (or fails) before it starts itself.
  • “milestone” – The dependency must start before the dependent starts, but once the dependent has started, the dependency link becomes soft. This is different from “waits-for” in that if the dependency fails, the dependent will not start.

This is what I’m currently implementing (up until now, only “regular” and “waits-for” dependencies have been supported by Dinit).

For the boot failure case, Dinit currently starts the service named “single” (i.e. the single-user service); however, some flexibility / configurability might be added at a later date.

For next time

There are a lot of things that I want write about and implement, and though finding the time has been increasingly difficult lately I’m hoping things will calm down a little over the next few months.

One thing I really need to do is look again, properly, at some of the other supervision/init systems out there. There are two motivations for this: one, determining whether Dinit is really necessary in its own right –  that is, can any of the existing systems do everything that I’m hoping Dinit will be able to, and would it make sense to collaborate with / contribute to one of them? In particular s6 and Nosh are two suites which seem like they are well-designed and capable. (Note that I don’t envisage stopping work on Dinit altogether, and don’t feel like availability of another quality init system is going to be a bad thing).

There’s still a lot more work that needs to be done with Dinit, too. Presently it’s not possible to modify loaded service definitions (including changing dependencies) which is certainly a must-have-for-1.0 feature, but that’s really just the tip of the iceberg. At some point I’d like to create a formal list of what is needed to truly supplant Systemd in the common Linux software ecosystem. Completing the basic Dinit functionality remains a priority for now, however.

Thanks for reading and, as always, constructive comments are welcome.


Be Careful with Simplistic Locking

(or: Re-entrant mutexes considered harmful)

A common method of protecting data from race conditions in an object-oriented system is to use a simple mutex. The pattern goes something like this:

  1. Call some access or modification method on an object
  2. On entry to the method, mutex (associated with the object) is obtained
  3. … method does its stuff
  4. Mutex is released
  5. Method returns

Java, for instance, provides built-in support for this particular sequence with the “synchronized” keyword. A method tagged as “synchronized” automatically obtains the mutex on entry and releases it on return, which removes a lot of the burden from the programmer who otherwise has to make sure that the mutex is properly released on every code path (including exceptions). This works fine in a lot of cases. Java’s mutexes are also re-entrant meaning that one synchronized method can call another one in the same object without causing instant deadlock, even though they share the same mutex (the mutex is associated with the object, not the method). However, “synchronized” is not suitable as a general purpose mutex, and in fact its re-entrancy can lead to bad design.

Consider a design which uses the listener pattern. That is, some class has a modifier method which modifies data and notifies a number of listeners of the change (via some listener interface). How could such a design be made thread-safe? it might be tempting to mark the modifier method as synchronized, and in many cases this would work fine. If a listener needs to access or modify data, it can do so, and the re-entrant nature of “synchronized” means no deadlock will occur. However, this approach is a bad design, for subtle reasons, which basically boil down to this: a mutex should not necessarily prevent data access or modification by threads which do not hold the mutex.

Huh, you may be thinking, what’s this guy smoking? That’s precisely what a mutex is for. But that’s not exactly what a mutex is for; a mutex is a mechanism to allow threads to co-ordinate access to data (or some resource) in order to avoid race conditions, but it is not specifically meant to limit resource access to the single thread holding the mutex. In particular, if the thread holding the mutex is waiting on a mutex held by a second thread, and the second thread is operating under the assumption that this is the case, then it should be fine for that thread to access the resource that is protected by the mutex – clearly there can be no race condition, since the only other thread that otherwise has any right to access the resource (i.e., the thread that holds the mutex) is blocked.

If you think this argument is absurd, consider two cases in Java where this situation can come about:

  1. An arbitrary thread needs to modify the data (the “model”) of a Swing component, and needs to do so synchronously, from within a listener callback which is called with a mutex protecting an object “O” held. To modify the model safely the listener needs to use EventQueue.invokeAndWait() or equivalent. The code invoked on the dispatch thread also attempts to access the object “O”. Bang – deadlock.
  2. Again, a listener callback is called with a mutex held. This time the listener calls a method on a remote object (in a second Java VM) via RMI. The remote invocation calls back in to the first VM and attempts to obtain the (supposedly re-entrant) mutex, however, due to the RMI implementation, is now running on a different thread in the first VM. Bang – deadlock.

The obvious workaround – invoking the listener callback(s) outside of a synchronized block, i.e. without the mutex held, does work; but it leaves the possibility that the data is modified between the modification event and the listener being notified, which can be undesirable. The best solution from a design point-of-view is to move the acquisition and release of the mutex outside of the responsibility of the object being protected by it. Thus the sequence at the start of the would be changed to:

  1. Acquire mutex
  2. Invoke method
  3. … method does its thing
  4. Method returns
  5. Release mutex

Also, it should be assumed (even enforced) that the mutex is not re-entrant, to allow for the case where the executing thread is not the one which actually acquired the mutex. This is more complicated, and places more burden on the programmer – which is not desirable, especially when dealing with concurrency – but it is a better overall design. For one thing, it allows a solution for the two problems outlined above; secondly, it separates concerns – why should an object worry about synchronising if it may or may not be used in a multi-threaded system?

Ideally there’d be some language support for this sort of design too. It would be nice if there was a way to tag that a specific object should only be touched when an associated mutex was held, and have static analysis to determine when such a rule was being broken. Unfortunately such things are not yet readily available/usable, at least not as far as I’m aware.

In conclusion: be careful when using “synchronized” or re-entrant mutexes; consider using non-re-entrant mutexes instead.