The best way to get god is via rubygems:

God currently only works on Linux (kernel 2.6.15+), BSD, and Darwin systems. No support for Windows is planned. Event based conditions on Linux systems require the cn (connector) kernel module loaded or compiled into the kernel and god must be run as root.

The following systems have been tested. Help us test it on others!

Note: this quick start guide requires god 0.12.0 or above. You can check your version by running:

The easiest way to understand how god will make your life better is by trying out a simple example. To get you up and running quickly, I'll show you how to keep a trivial server running.

Open up a new directory and write a simple server. Let's call it simple.rb:

Now we'll write a god config file that tells god about our process. Place it in the same directory and call it simple.god:

This is the simplest possible god configuration. We start by declaring a God.watch block. A watch in god represents a process that we want to watch and control. Each watch must have, at minimum, a unique name and a command that tells god how to start the process. The keepalive declaration tells god to keep this process alive. If the process is not running when god starts, it will be started. If the process dies, it will be restarted.

In this example the simple process runs foreground, so god will take care of daemonizing it and keeping track of the PID for us. When possible, it's best to let god daemonize processes for us, that way we don't have to worry about specifying and keeping track of PID files. Later on we'll see how to manage processes that can't run foreground or that require PID files to be specified.

To run god, we give it the configuration file we wrote with -c. To see what's going on, we can ask it to run foreground with -D:

There are two ways that god can monitor your process. The first and better way is with process events. Not every system supports it, but those that do will automatically use it. With events, god will know immediately when a process exits. For those systems without process event support, god will use a polling mechanism. The output you see throughout this section will show both ways.

After starting god, you should see some output like the following:

Here you can see god starting up, noticing that the simple process isn't running, starting it, and then checking every five seconds to make sure it's up. If you'd like to see god work its magic, go ahead and kill the simple process. You should then see something like this:

While keeping a process up is useful, it would be even better if we could make sure our process was behaving well and restart it when resource utilization exceeds our specifications. With a few additions, we can easily have our process restarted when memory usage or CPU goes above certain limits. Edit your sample.god config file to look like this:

Here I've specified a :memory_max option to the keepalive command. Now if the process memory usage goes above 150 megabytes, god will restart it. Similarly, by setting the :cpu_max, god will restart my process if its CPU usage goes over 50%. By default these properties will be checked every 30 seconds and will be acted upon if there is an overage for three out of any five checks. This prevents the process from getting restarted for temporary resource spikes.

To test this out, modify your simple.rb server script to introduce a memory leak:

Ctrl-C out of the foregrounded god instance. Notice that your current simple server will continue to run. Start god again with the same command as before. Now instead of starting the simple process, it will notice that one is already running and simply switch to the up state.

In order to get our new simple server running, we can issue a command to god to have our process restarted:

From the logs you can see god killing and restarting the process:

God will now start reporting on memory and CPU utilization of your process:

On the last line of the above log you can see that CPU usage has gone above 50% for three cycles and god will issue a restart operation. God will continue to monitor the simple process for as long as god is running and the process is set to be monitored.

Now, before you kill the god process, let's kill the simple server by asking god to stop it for us. In a new terminal, issue the command:

You should see the following output:

And in the foregrounded god terminal window, you'll see the log of what happened:

Now feel free to Ctrl-C out of god. Congratulations! You've just taken god for a test ride and seen how easy it is to keep your processes running.

This is just the beginning of what god can do, and in reality, the keepalive command is a convenience method written using more advanced transitional and condition constructs that may be used directly. You can configure many different kinds of conditions to have your process restarted when memory or CPU are too high, when disk usage is above a threshold, when a process returns an HTTP error code on a specific URL, and many more. In addition you can write your own custom conditions and use them in your configuration files. Many different lifecycle controls are available alongside a sophisticated and extensible notifications system. Keep reading to find out what makes god different from other monitoring systems and how it can help you solve many of your process monitoring and control problems.

Now that you've seen how to get started quickly, let's see how to use the more powerful aspects of god. Once again, the best way to learn will be through an example. The following configuration file is what I once used at gravatar.com to keep the mongrels running:

That's a lot to take in at once, so I'll break it down by section and explain what's going on in each.

Here I've set a constant that is used throughout the file. Keeping the RAILS_ROOT value in a constant makes it easy to adapt this script to other applications. Because the config file is Ruby code, I can set whatever variables or constants I want that make the configuration more concise and easier to work with.

Because the config file is written in actual Ruby code, we can construct loops and do other intelligent things that are impossible in your every day, run of the mill config file. I need to watch three mongrels, so I simply loop over their port numbers, eliminating duplication and making my life a whole lot easier.

A watch represents a single process that has concrete start, stop, and/or restart operations. You can define as many watches as you like. In the example above, I've got some Rails instances running in Mongrels that I need to keep alive. Every watch must have a unique name so that it can be identified later on. The start and stop attributes specify the commands to start and stop the process. If no restart attribute is set, restart will be represented by a call to stop followed by a call to start. The optional grace attribute sets the amount of time following a start/stop/restart command to wait before resuming normal monitoring operations. If the process you're watching runs as a daemon (as mine does), you'll need to set the pid_file attribute.

Behaviors allow you to execute additional commands around start/stop/restart commands. In our case, if the process dies it will leave a PID file behind. The next time a start command is issued, it will fail, complaining about the leftover PID file. We'd like the PID file cleaned up before a start command is issued. The built-in behavior clean_pid_file will do just that.

Watches contain conditions grouped by the action to execute should they return true. I start with a start_if block that contains a single condition. Conditions are specified by calling condition with an identifier, in this case :process_running. Each condition can specify a poll interval that will override the default watch interval. In this case, I want to check that the process is still running every 5 seconds instead of the 30 second interval that other conditions will inherit. The ability to set condition specific poll intervals makes it possible to run critical tests (such as :process_running) more often than less critical tests (such as :memory_usage and :cpu_usage).

Similar to start_if there is a restart_if command that groups conditions that should trigger a restart. The memory_usage condition will fail if the specified process is using too much memory. The maximum allowable amount of memory is specified with the above attribute (you can use the kilobytes, megabytes, or gigabytes helpers). The number of times the test needs to fail in order to trigger a restart is set with times. This can be either an integer or an array. An integer means it must fail that many times in a row while an array [x, y] means it must fail x times out of the last y tests.

To keep an eye on CPU usage, I've employed the cpu_usage condition. When CPU usage for a Mongrel process is over 50% for 5 consecutive intervals, it will be restarted.

Conditions inside a lifecycle section are active as long as the process is being monitored (they live across state changes).

The :flapping condition guards against the edge case wherein god rapidly starts or restarts your application. Things like server configuration changes or the unavailability of external services could make it impossible for my process to start. In that case, god will try to start my process over and over to no avail. The :flapping condition provides two levels of giving up on flapping processes. If I were to translate the options of the code above, it would be something like: If this watch is started or restarted five times withing 5 minutes, then unmonitor it…then after ten minutes, monitor it again to see if it was just a temporary problem; if the process is seen to be flapping five times within two hours, then give up completely.

That's it!

To start the god monitoring process as a daemon simply run the god executable passing in the path to the config file (you need to sudo if you're using events on Linux or want to use the setuid/setgid functionality):

While you're writing your config file, it can be helpful to run god in the foreground so you can see the log messages. You can do that with:

You can start/restart/stop/monitor/unmonitor your Watches with the same utility like so:

Need to watch a script that doesn't have built in daemonization? No problem! God will daemonize and keep track of your process for you. If you don't specify a pid_file attribute for a watch, it will be auto-daemonized and a PID file will be stored for it in /var/run/god.

If you'd rather have the PID file stored in a different location, you can set it at the top of your config:

The directory you specify must be writable by god.

Watches can be assigned to groups. These groups can then be controlled together from the command line.

The above configuration now allows you to control the watch (and any others that are in the group) with a single command:

By default, the STDOUT stream for your process is redirected to /dev/null. To get access to this output, you can redirect the stream either to a file or to a command.

To redirect STDOUT to a file, set the log attribute to a file path. The file will be written in append mode and created if it does not exist.

To redirect STDOUT to a command that will be run for you, set the log_cmd attribute to a command.

By default, STDERR is redirected to STDOUT. You can redirect it to a file or a command just like STDOUT by setting the err_log or err_log_cmd attributes respectively.

It is possible to have god run your start/stop/restart commands as a specific user/group. This can be done by setting the uid and/or gid attributes of a watch.

This only works for commands specified as a string. Lambda commands are unaffected.

By default, God sets the working directory to / before running your process. You can change this by setting the dir attribute on the watch.

You can set any number of environment variables you wish via the env attribute of a watch.

If you want your process to run chrooted, simply use the chroot attribute on the watch. The specified directory must exist and have a /dev/null.

In addition to specifying start/stop/restart commands as strings (to be executed via the shell), you can specify a lambda that will be called.

If you do not provide a stop command, God will attempt to stop your process by first sending a SIGTERM. It will then wait for ten seconds for the process to exit. If after this time it still has not exited, it will be sent a SIGKILL. You can customize the stop signal and/or the time to wait for the process to exit by setting the stop_signal and stop_timeout attributes on the watch.

You should feel free to separate your god configs into separate files for easier organization. You can load in other configs using Ruby's normal load method, or use the convenience method God.load which allows for glob-style paths:

God won't start its monitoring operations until all configurations have been loaded.

God allows you to load or reload configurations into an already running instance. There are a few things to consider when doing this:

To load a config into a running god, issue the following command:

Config files that are loaded dynamically can contain anything that a normal config file contains, however, global options such as God.pid_file_directory blocks will be ignored (and produce a warning in the logs).

Sifting through the god logs for statements specific to a single Watch can be frustrating when you have many of them. You can get the realtime logs for a single Watch via the command line:

This will display log output for the local-3000 Watch and update every second with new log messages.

You can also supply a shorthand to the log command that will match one of your watches. If it happens to match several, the shortest match will be used:

God has an extensible notification framework built in that makes it easy to have notifications sent when conditions are triggered. Each notification type has a set of configuration parameters that must be set. These parameters may be set globally via Contact Defaults or individually via Contact Instances.

Contact Defaults - Some parameters are unlikely to change on a per-contact basis. You should set those parameters via the defaults mechanism.

Contact Instances - Each contact must have a unique name set. You may optionally assign each contact to a group.

Condition Attachment - To have a specific contact notified when a condition is triggered, simply set the condition's notify attribute to the name of the individual contact.

There are two ways to specify that a notification should be sent. The first, easier way is shown above. Every condition can take an optional notify attribute that specifies which contacts should be notified when the condition is triggered. The value can be a contact name or contact group or an array of contact names and/or contact groups.

The second way allows you to specify the priority and category in addition to the contacts. The extra attributes can be arbitrary integers or strings and will be passed as-is to the notification subsystem.

The above notification will arrive as an email similar to the following.

So far you've been introduced to a simple poll-based config file and seen how to run it. Poll-based monitoring works great for simple things, but falls short for highly critical tasks. God has native support for kqueue/netlink events on BSD/Darwin/Linux systems. For instance, instead of using the process_running condition to poll for the status of your process, you can use the process_exits condition that will be notified immediately upon the exit of your process. This means less load on your system and shorter downtime after a crash.

While the configuration syntax you saw in the previous example is very simple, it lacks the power that we need to deal with event based monitoring. In fact, the start_if and restart_if methods are really just calling out to a lower-level API. If we use the low-level API directly, we can harness the full power of god's event based lifecycle system. Let's look at another example config file.

A bit longer, I know, but very straighforward once you understand how the transition calls work. The name, interval, start, stop, and pid_file attributes should be familiar. We also specify the clean_pid_file behavior.

Before jumping into the code, it's important to understand the different states that a Watch can have, and how that state changes over time. At any given time, a Watch will be in one of the init, up, start, or restart states. As different conditions are satisfied, the Watch will progress from state to state, enabling and disabling conditions along the way.

When god first starts, each Watch is placed in the init state.

You'll use the transition method to tell god how to transition between states. It takes two arguments. The first argument may be either a symbol or an array of symbols representing the state or states during which the specified conditions should be enabled. The second argument may be either a symbol or a hash. If it is a symbol, then that is the state that will be transitioned to if any of the conditions return true. If it is a hash, then that hash must have both true and false keys, each of which point to a symbol that represents the state to transition to given the corresponding return from the single condition that must be specified.

The first transition block tells god what to do when the Watch is in the init state (first argument). This is where I tell god how to determine if my task is already running. Since I'm monitoring a process, I can use the process_running condition to determine whether the process is running. If the process is running, it will return true, otherwise it will return false. Since I sent a hash as the second argument to transition, the return from process_running will determine which of the two states will be transitioned to. If the process is running, the return is true and god will put the Watch into the up state. If the process is not running, the return is false and god will put the Watch into the start state.

If god has determined that my process isn't running, the Watch will be put into the start state. Upon entering this state, the start command that I specified on the Watch will be called. In addition, the above transition specifies a condition that should be enabled when in either the start or restart states. The condition is another process_running, however this time I'm only interested in moving to another state once it returns true. A true return from this condition means that the process is running and it's ok to transition to the up state (second argument to transition).

The other half of this transition uses the tries condition to ensure that god doesn't get stuck in this state. It's possible that the process could go down while the transition is being made, in which case god would end up polling forever to see if the process is up. Here I've specified that if this condition is called five times, god should override the normal transition destination and move to the start state instead. If you specify a transition attribute on any condition, that state will be transferred to instead of the normal transfer destination.

This is where the event based system comes into play. Once in the up state, I want to be notified when my process exits. The process_exits condition registers a callback that will trigger a transition change when it is fired off. Event conditions (like this one) cannot be used in transitions that have a hash for the second argument (as they do not return true or false).

Notice that I can have multiple transitions with the same start state. In this case, I want to have the memory_usage and cpu_usage poll conditions going at the same time that I listen for the process exit event. In the case of runaway CPU or memory usage, however, I want to transition to the restart state. When a Watch enters the restart state it will either call the restart command that you specified, or if none has been set, call the stop and then start commands.

God was designed from the start to allow you to easily write your own custom conditions, making it simple to add tests that are application specific.

If you'd like to hack on god itself or contribute fixes or new functionality, read this section.

The codebase can be found at https://github.com/mojombo/god. To get started, fork god on GitHub into your own account and then pull that down to your local machine. This way you can easily submit changes via Pull Requests later on.

We recommend using rbenv and ruby-build to manage multiple versions of Ruby and their separate gemsets. Any changes to god must work on both Ruby 1.8.7-p352 and 1.9.3-p0.

God uses bundler to deal with development dependencies. Once you have the code locally, you can pull in all the dependencies like so:

In order for process events to function during development you'll need to compile the C extensions:

Now you're ready to run the tests and make sure everything is configured properly. On Linux you'll need to run the tests as root in order for the events system to load. On MacOS there is no need to run the tests as root.

To run your development god to make sure config files and such still work properly, just run:

There are a bunch of example config files for various scenarios in test/configs that you can try out. For big new features, it's great to add a new test config showing off the usage of the feature.

If you intend to contribute your changes back to god core, make sure you create a new branch and do your work there. Then, when your changes are ready to be shared with the world, push them to your fork and issue a Pull Request against mojombo/god. Make sure to describe your changes in detail and add relevant tests.

Any feature additions or changes should be accompanied by corresponding updates to the documentation. It can be found in the docs directory. The documentation is done in Gollum format and then converted into the public site at http://godrb.com. To see the generated site locally you'll first need to commit your changes to git and then issue the following:

This will open the site in your browser so you can check for correctness.