What?

Nginxmgr is an application for managing pools of upstream servers in an Nginx deployment. It allows you to dynamically allocate, deallocate and monitor servers in an upstream pool via health checks and an XML-RPC interface.

Why?

I wrote this software because I use Nginx as a web application load balancer and have often found the need to monitor the upstream servers and have it react like a real load balancer would. I was also looking for a way to be able to manage the upstream servers programmatically with Python and monitor the health of the upstream pools by being aware of the number of servers available.

Use Case

Typical use case for Nginxmgr is in a large environment, with a large number of upstream pools and servers. Of course, that’s not a requirement. You could also use it on a pool of just a few servers.

Checks/Handlers

Out of the box, Nginxmgr comes with one check named url_ping. It’s job is to check the status of an upstream member every ‘n’ seconds and if it fails, call the appropriate handler. The handler will then remove the server from the upstream pool and reload Nginx. The url_ping check is also configurable, allowing checks on a port other than 80 and also status codes other than 200 (in case you want to get a 404 or something back).

Checks and handlers are Python modules and functions that allow for flexibility and ease of development were new checks/handlers needed.

Interoperability With Nginx

Currently Nginxmgr runs along side Nginx as a separate daemon, however it will not attempt to start if Nginx is not running. It also does a check in the main loop before the health checks are executed to determine if Nginx was shut down or died since the last check. When that check happensm, if Nginx isn’t running, Nginxmgr will shut itself down.

Installing Nginxmgr

First, you need to get the package. You can find links to the code below in the section titled “Get The Code”. Once you have the source tarball, you should extract it, then cd into the newly created nginxmgr directory. Once there you will then run: python setup.py install. This should install all the library code and the command line script. If you’re cloning the code (hg clone), all you will need to do is run the setup script as above in the root directory of your newly acquired repository.

Configuring Nginxmgr

There are two different configuration files for Nginxmgr. One is for the core code, and one is for the url_ping check command. You should only need to mess with the core config. Below are example configs with comments explaining what is going on:

/etc/nginxmgr/nginxmgr.cfg:

[core]
# Should I run in  the background?
daemon           = True
# Should I enable debug level logging?
debug            = True
# Path to handler scripts
handler_path     = '/etc/nginxmgr/handlers/'
# Path to check scripts
checks_path      = '/etc/nginxmgr/checks/'
# Health check config definitions
healthcheck_defs = '/etc/nginxmgr/health_check_defs.cfg'
# How do you want the log to look?
log_format       = '%(asctime)s %(levelname)s %(threadName)s %(filename)s LINE %(lineno)s: %(message)s'
# Where should I log to?
log_file         = '/var/log/nginxmgr.log'
# Where should I deposit my pid?
pid_file         = '/var/run/nginxmgr.pid'
# Location of nginx base config
ngx_base_config  = '/etc/nginx/nginx.conf'
# Location of nginx binary
ngx_bin          = '/usr/sbin/nginx'
# Location of nginx upstreams config
ngx_config       = '/etc/nginx/conf.d/upstreams.conf'
# Location of nginx pid file
ngx_pid_file     = '/var/run/nginx.pid'
# Interface for XML-RPC to listen on
xml_rpc_ip       = "0.0.0.0"
# Port for XML-RPC to bind to
xml_rpc_port     = 9000

    [healthcheck]
    # Enable health checks?
    enabled = True

        [[checks]]
        # Inidividual checks

            [[[url_ping_test_upstreams]]]
            # Type maps to a python module in etc/checks
            type     = url_ping
            # Upstream maps to an upstream pool in nginx upstream.conf
            upstream = test-upstreams 
            # Handler maps to a python module:function in etc/handlers
            handler  = url_ping_handler:badupstream
                # Custom stuff to define for custom checks (defined by check module)
                [[[[extras]]]]
                # port — If port is defined, any existing port in the upstream
                # will be stripped and replaced.
                # port     = 2000
                url      = "http://%s/"
                status   = 200
            [[[url_ping_test_upstreams-two]]]
            # Type maps to a python module in etc/checks
            type     = url_ping
            # Upstream maps to an upstream pool in nginx upstream.conf
            upstream = test-upstreams-two
            # Handler maps to a python module:function in etc/handlers
            handler  = url_ping_handler:badupstream
                # Custom stuff to define for custom checks (defined by check module)
                [[[[extras]]]]
                # port — If port is defined, any existing port in the upstream
                # will be stripped and replaced.
                # port     = 2000
                url      = "http://%s/setup"
                status   = 404

/etc/nginxmgr/health_check_defs.cfg:

[url_ping]
# Check a url ever little while to make sure it's up
# Corresponds to ./checks/check_name.py:function_name
check=url_ping:ping
# Timeout (in seconds)
schedule=2

Running, Stopping and Reloading Nginxmgr

To start Nginxmgr, you would run this:

nginxmgr -c /etc/nginxmgr/nginxmgr/cfg

It will then fork off into the background and write a pid file to wherever you specify in the config (default is /var/run/nginxmgr.pid).

To shut it down properly, get the pid from the pid file and send it a kill signal(SIGTERM, SIGQUIT or SIGABRT) . If not running in daemon mode, just hit CTRL-C.

To reload the config and the health checks send the pid a SIGHUP.

You could also use Nginxmgr in an init script with out difficulty.

XML-RPC Interface

The XML-RPC interface is ideal for managing upstream pools and nodes on the fly, without having to mess with Nginx itself. It is also nice because it provides a language agnostic way to automate operations on upstream pools and provide a way to monitor the health of the pools.

XML-RPC Exposed Methods

Methods exposed to the XML-RPC interface are:

• get_pools() — lists all of the pools available
• get_members('pool_name') — get all members available in specified pool
• disable_member('pool_name', 'IPADDR:PORT') — Temporarily disable member in pool
• remove_member('pool_name', 'IPADDR:PORT') — Remove member from pool
• enable_member('pool_name', 'IPADDR:PORT') — Enable disabled member in pool
• add_member('pool_name', 'IPADDR:PORT') — Add new member to pool

Things to Note

I made some assumptions when developing this and as such it should be noted that Nginxmgr has only been tested on Linux(Centos and Ubuntu) with Python > 2.5. Nginxmgr makes some calls to utilities common to these systems such as ‘ps’, ‘kill’ and ‘pgrep’. The paths are currently hard coded, but that could easily be changed and/or stuffed in the configuration file.

I also make some assumptions about your Nginx setup. Mainly I assume that you separate your Nginx upstreams in it’s own file. That’s how I prefer to manage it, but It’s not required.

I have a major rewrite planned as an exercise in Test Driven Development as well as taking advantage of practices outlined in Clean Code by Robert (Uncle Bob) Martin.

In this rewrite I will also be experimenting with the Python multiprocessing module rather than threads. I’ve also considered REST instead of XML-RPC.

Documentation is non-existent. Sorry, that is one of the things I would like to work on :)

Purpose of This Blog Post

The purpose of this post is to introduce the software see if there is even any interest for such a thing in the community. My hope is that someone else will have a use for it and wish to improve upon it.

Get The Code

I use mercurial for DVCS, so you can find the code on bitbucket here: https://bitbucket.org/benjaminws/nginxmgr/

From there you can either clone the repository or download a tarball of the source.

If for whatever reason there is a problem getting the code from bitbucket, I also have a copy of the code here: http://nginxmgr.code.just-another.net/

From there you can clone the repository.

Hope this software helps someone. Thanks for reading!