Re: Thoughts on "First Class Services"

From: Avery Payne <avery.p.payne_at_gmail.com>
Date: Wed, 29 Apr 2015 09:46:39 -0700

Note: this re-post is due to an error I made earlier today. I've gutted
out a bunch of stuff as well. My apologies for the duplication.

On 4/28/2015 11:34 AM, Laurent Bercot wrote:
>
> I'm also interested in Avery's experience with dependency handling.

Hm. Today isn't the best day to write this (having been up since 4am)
but I'll try to digest all the little bits and pieces into something.
Here we go...

First, I will qualify a few things. The project's scope is, compared to
a lot of the discussion on the mailing list, very narrow. There are
several goals but the primary thrust of the project is to create a
generic, universal set of service definitions that could be plugged into
many init, distribution, and supervision framework arrangements. That's
a tall order in itself, but there are ways around a lot of this. So
while the next three paragraphs are off-topic, they are there to address
those three concerns mentioned.

With regard to init work, I don't touch it. Trying to describe a proper
init sequence is already beyond the scope of the project. I'm leaving
that to other implementers.

With regard to distributions, well, I'm trying to make it as generic as
possible. Development is done on a Debian 7 box but I have made efforts
to avoid any Debian-isms in the actual project itself. In theory, you
should be able to use the scripts on any distribution.

With regard to the supervision programs used, the difference in command
names have been abstracted away. I'm not entirely wild about how it is
currently done, but creating definitions is a higher priority than
revisiting this at the moment. In the future, I will probably
restructure it.

~ ~ ~ ~ ~ ~ ~ ~

What

---
The dependency handling in supervision-scripts is meant to be used in 
installations that don't have access to it.  Put another way, it's a 
"Poor Man's Solution" to the problem and functions as a convenience.  
The feature is turned off by default, and this will cause any service 
definition that requires other services to run-loop repeatedly until 
someone starts them manually.  This could be said to be the default 
behavior of most installations that don't have dependency handling, so 
I'm not introducing a disruptive behavior with this feature.
Why
---
I could have hard-coded many of the dependencies into the various run 
scripts, but this would have created a number of problems for other areas.
1. Hard-coding prevents switching from shell to execline in the future, 
by necessitating a re-write.  There will be an estimated 1,000+ scripts 
when the project is complete, so this is a major concern.
2. We are already using the filesystem as an ad-hoc database, so it 
makes sense to continue with this concept.  The dependencies should be 
stored on the filesystem and not inside of the script.
With this in mind, I picked sv/(service)/needs as a directory to hold 
the definitions to be used.  Because I can't envision what every init 
and future dependency management framework would look like, I'll simply 
make it as generic as I can, leaving things as open as possible to 
additional changes.    A side note: it is by fortuitous circumstance 
that anopa uses a ./needs directory that has the same functionality and 
behavior.  I use soft links "just because".  Anopa uses named files.  
The net effect is the same.
Each dependency is simply a named soft link that points to a service 
that needs to be started, typically something like 
"sv/(service)/needs/foobar points to /service/foobar".  In this case, a 
soft link is made with the name of the service, pointing to the service 
definition in /service.  This also allows me to ensure that the 
dependency is actually available, and not just assume that it is there.
A single rule determines what goes into ./needs, "you can only have the 
names of other services that are explicitly needed".  You can say "foo 
needs baz" and "baz needs bar" but NEVER would you say "foo needs baz, 
foo needs bar".  This is intentional because it's not the job of the 
starting service to handle the entire chain.  It simplifies the list of 
dependencies because a service will only worry about its immediate 
needs, and not the needs of dependent services it launches.  It also has 
the desirable property of making dependency chains self-organizing, 
which is an important decision with hundreds of services having 
potentially hundreds of dependencies.   Setup is straightforward and you 
can easily extend a service need by adding one soft link to the new 
dependency.  This also fits with my current use of a single launch 
script; I don't have to change the script, just the parameters that the 
script uses.  The new soft link becomes just another parameter.  You 
could call this "peer-level dependency resolution" if you like.
How
---
Enabling this behavior requires that you set sv/.env/NEEDS_ENABLED to 
the single character "1".   It is normally set to "0".  With the setting 
disabled (zero), the entire set of code for dependency handling is 
simply ignored; it is inside of a large if-then wrapper that looks at 
this flag and skips over it when needed.  When the setting is enabled 
(one), the code will trigger.  There has been discussion elsewhere about 
re-writing this code, and while I like the idea of cleaning it up, for 
the time being, it is functional as-is.
The script is meant to walk through a list of services in no order, 
asking each service to start, and once started ensuring that it is 
indeed running by calling its ./check script (if it has one). Should a 
requested service fail to start, a log entry is issued with the name of 
the offending service, and the ./run script is aborted, leading to a 
run-loop.  On the next loop, the service start will quickly succeed for 
all of the services that are already running, which causes the script to 
rapidly advance to the one service that had the issue.
Note that, because the same launch script is used for both the original 
service and its dependency, it is possible for the dependency to start 
its own dependencies.  This is essentially a self-organizing recursive 
decent.
Once all of the needs are started, the behavior is the same as if 
dependency handling is disabled; a run directory is created if needed, 
and the process launched.
I don't attempt to shut down any dependent services because of the 
complexity involved.  While I'm sure I could write some kind of 
reverse-tree-walking script, it's just not something easily done in 
shell.  The behavior of "service goes down but dependencies stay up" is 
not ideal, but keep in mind, this is an optional feature of last resort, 
a freebie.  You get what you pay for sometimes. :)
Experiences
----------
I am currently using this feature on my home server, and it works fine.  
I have yet to have it fail to start a dependency chain, unless I am 
testing a definition and the settings were bad to begin with.  Start up 
time for a test chain, forked-daapd>avahi-daemon>dbus, is very quick on 
a single core Sempron 140, less than a second, so I don't see speed 
being an issue.  Disabling the feature brings back the "old behavior" 
and forked-daapd will not start until you manually start dbus and 
avahi-daemon.
If you have questions I'll try to answer them.  Hopefully I've 
adequately covered the design decisions and the net effect of them. 
Otherwise, that's my 0.05 cents on my experiences with dependency 
management.

Received on Wed Apr 29 2015 - 16:46:39 UTC