In Part 1 of the series we covered the basics of getting the development environment up and running. We also looked at how to get a really simple ErlyDTL template rendering. If you haven’t yet gone through Part 1, I suggest you do that now. If you have, read on!
There are a few reasons this series is targeting this technology stack. One of them is uptime. We’re aiming to build a site that stays up as much as possible. Given that, one of the things that I missed in the previous post was setting up a load balancer. Hence this post will attempt to fill that gap.
Any serious web-based application will have load-balancing in play somewhere. While not essential during development, it’s handy to have a similar set up in the hope that it exposes you to some potential issues you might face when the application reaches production.
There are many high-quality load-balancing solutions out there to choose from. For this series, we shall be using HAProxy, which is a common choice amongst developers building scalable web applications. The rest of this post will cover how to set up HAProxy, verifying that the configuration is correct and confirming that it behaves appropriately when nodes in our cluster go down.
Please note the goal is to demonstrate how HAProxy can be configured. When deploying to your production environments please make sure the configuration matches your needs.
Let’s start by pulling down the latest stable version of HAProxy’s source, extracting it and building it. Here’s a sample log of what you should expect:
oj@nix ~/blog $ wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.20.tar.gz ... snip ... oj@nix ~/blog $ tar -xzf haproxy-1.4.20.tar.gz ... snip ...
At this point we’ve got the source and we’re ready to make. HAProxy requires a parameter in order to build, and this parameter varies depending on your target system:
oj@nix ~/blog $ cd haproxy-1.4.20 oj@nix ~/blog/haproxy-1.4.20 $ make Due to too many reports of suboptimized setups, building without specifying the target is no longer supported. Please specify the target OS in the TARGET variable, in the following form: make TARGET=xxx Please choose the target among the following supported list : linux26, linux24, linux24e, linux22, solaris freebsd, openbsd, cygwin, custom, generic Use "generic" if you don't want any optimization, "custom" if you want to precisely tweak every option, or choose the target which matches your OS the most in order to gain the maximum performance out of it. Please check the Makefile in case of doubts. make: *** [all] Error 1
According to uname, I’m running Linux Kernel 2.6:
oj@nix ~/blog/haproxy-1.4.20 $ uname -r 2.6.31-21-generic
As a result, I’ll be passing in linux26. Make sure you specify the correct option depending on which system you are running. We’ll be building it and installing it so that it can be called from anywhere:
oj@nix ~/blog/haproxy-1.4.20 $ make TARGET=linux26 ... snip ... oj@nix ~/blog/haproxy-1.4.20 $ sudo make install ... snip ...
Simple! We now need to create a configuration for HAProxy which we can use during development. Not surprisingly, HAProxy can be run as a daemon, but it can also be invoked from the command line with a configuration passed as a parameter. For our development, we’ll be executing from the command line as this will give us feedback/output on what’s going on.
Note: If you’re using Mac OSX, you can easily use brew to install haproxy with the command
brew install haproxy
Let’s create a file called
dev.haproxy.conf inside our application directory so that it can be included in our source:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105
In the configuration above the
backend riaks section has four server nodes. Each one of them has the
check option specified. This enables health-checking on the same address and port that the server instance is bound to. If you decided that you didn’t want to do health-checking in this manner you easily enable health-checking over HTTP, as Riak has a built-in URI which can be used to validate the state of the node. Change the
backend riaks section in the configuration to look like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
To make sure this is functioning correctly, we need to open two consoles and change our working directory to our
csd application (for those who have forgotten,
csd is the application we’re building - it was mentioned in Part 1). In console 1:
oj@nix ~/blog/csd $ sudo haproxy -f dev.haproxy.conf -d Available polling systems : sepoll : pref=400, test result OK epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 4 (4 usable), will use sepoll. Using sepoll() as the polling mechanism.
This indicates that HAProxy is up and running and waiting for connections. Let’s get Webmachine fired up in console 2:
oj@nix ~/blog/csd $ ./start.sh ... snip ... =PROGRESS REPORT==== 11-Jul-2012::23:07:27 === application: csd started_at: nonode@nohost
Verification of HAProxy configuration
On the surface it appears that we haven’t broken anything. We also need to make sure that any communication with Riak that we have down the track is also functioning. So let’s validate that now.
First, we have to make sure that Riak is running. If you have followed Part 1 already and your Riak cluster is running then you’re good to go. If not, please read Part 1 for information on how to install Riak and configure it to run as a cluster of 4 nodes.
Next, let’s create 4 new connections and use the get_server_info/1 function to find out which node we are connected to. To do this, we’ll need to use an Erlang console which has all the Riak dependencies ready to go. It just so happens that when we fired up our Webmachine instance, we got an Erlang console for free. Simply hit the
enter key and you’ll be given a prompt. Notice that when we connect to Riak using the start_link/2 function, we are passing in the IP address and port of the load-balanced cluster instead of one of the running Riak nodes:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
So we can see that the load balancer has allocated four different connections, each to a different node in the cluster. This is a good sign. So let’s kill off one of the nodes:
In a very short period of time, you should see output in the HAProxy console which looks something like this:
1 2 3
The load balancer noticed that the node has died. Let’s make sure it no longer attempts to allocate connections to
dev3. Note that we call f() in our console before running the same script again, as this forces the shell to forget about any existing variable bindings:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
As we hoped,
dev3 is nowhere to be seen. Let’s fire it up again:
Note: It isn’t necessary to tell the node to rejoin the cluster. This happens automatically. Thanks to Siculars (see comment thread) for pointing that out.
HAProxy’s console will show you that it has re-established a connection to
As a final test, let’s make sure we see that node get connections when we attempt to connect:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
Excellent. Now that we’ve got our load-balancer set up in development, we’re ready to dive into connecting to Riak from our
csd application. That will be the topic for the next post in this series.
As always, comments and feedback are welcome and greatly appreciated. Suggestions on improvements and pointers on mistakes would be awesome. To anyone out there who has put HAProxy into production, we would love to hear your comments on your configuration!
Note: The code for Part 2 (this post) can be found on Github.