OJ’s rants What would OJ do?

Setting up Trac, Mercurial and SSH on Windows

WARNING - This blog post is long
This post has been edited since it was published. Please see the end of the article for any notes/modifications

Some Background Info

I had the need to do this for work recently. It was nothing short of a right royal pain in the butt. It was such a pain, in fact, that I have decided to document what I had to do to get it working so that other poor unfortunates will feel less pain if they have to do this themselves.

Almost regardless of the company and the software I'm working on, I use Mercurial for source code control. For the work I am doing at the moment, I was also using hg because the company I am involved with is relatively new and they hadn't yet sorted out a plan for version control or ALM. It was working quite well and I was pushing all my changes to my NAS box to make sure I had other copies backed up, etc. I was living the dream

The problem, though, was that this setup was working fine for me as a sole developer, but wasn't solving the problem for the other developers. There are two other people involved in development, and until now most of the work they have been doing has been in a different area to the bits I have been doing. Sharing and merging code was done by email patches. Those developers weren't using version control as far as I'm aware.

So given the volume of changes and the number of merges that were happening, it was well past time to get something proper in place that we could all use. The original plan was to go with an installation of TFS to handle all of these issues for us. Personally, I am not a huge fan of TFS. The experience I've had with it so far hasn't been great. It also has a shocking effect on the speed of my IDE (I seem to have no choice but to use integration with the IDE, which again isn't great).

To cut this part of the story short, we had issues getting TFS to work. We tried installing a few times, tweaking here and there, but we never managed to get it to work. I know I could have called a TFS guru, such as Snagy to pick his brains and perhaps ask him nicely to help out, but he's a busy man and there's no guarantee that we wouldn't have to start again from scratch. I wasn't really up for that, and neither was my boss.

So I told him that I thought it might just be easier to get a server running with Mercurial for the version control, and Trac for issue tracking and project "stuff". Not only would it mean that I could just push my local repository to the server and have the entire history transferred (TFS would have been latest version only) but it would save us a substantial cost in license fees, remove the need for VS integration, and everyone could be up and running with a DVCS in no time.

My boss, being the legend he is, gave me the thumbs up! Awesome.

So the rest of this post is dedicated to what was involved in getting things set up. If you're not interested, then go read something else

The need for a bit of security

Needless to say, the IP that we're building is something that my employer is rather protective of! And rightly so. Hence it won't come as a surprise to hear that security of this information is very important. I decided that having Mercurial run over plan old HTTP would be a bad idea. I was keen to have it tunnel via SSH, and have all clients authenticate using their own private keys. Hence, getting an SSH server was going to be part of the setup.

Web server

IIS is already installed on the server, and running another web server seemed like overkill. I didn't want to expose tracd directly over the web as it doesn't support SSL, so I wanted to get it running under IIS instead. This added another little bit of complexity to the install.

What to download

There were quite a few bits that I needed to download to get this working, they're listed below. At first it might not be obvious as to why these things are needed, just trust me

First, server-side components:

1. Python 2.5.4 - Get this version, not anything earlier or later if you want this guide to work! (download)
2. Setuptools (download)
3. ClearSilver (download)
4. PySqlite (download)
5. flup (download)
6. Trac installer v0.11.4 (download)
7. Mercurial (download)
8. Subversion (download)
9. Apache Tomcat AJP Isapi filter (download)
10. Junction (download)
11. CopSSH (download)
12. Windows Server 2003 Resource Kit (download)

Next, client-side:

1. Mercurial (download)
2. PuTTy (download)

Quite a lot isn't it

Server: Setting up the Python Environment

1. Execute the Python installer. This by default installs Python to C:\python25.
2. Modify the environment variables so that the Python install folder is included in the PATH.
3. Execute the Setuptools installer.
4. Execute the PySqlite installer.
5. Install the downloaded python eggs:
1. Open a command prompt
2. Navigate to the folder where your .egg files are downloaded to.
3. in the command line, type: easy_install [filename.egg].
4. Do this for ClearSilver and flup.

At this point, you should have a working installation of Python 2.5.4 with the required eggs and plug-ins.

Server: Setting up Mercurial

1. Execute the Mercurial installer. By default this installs to %programfiles%\Mercurial. Feel free to change this if you like.
2. During the installation, the installer will ask if you want to include the install folder in the system PATH. Make sure you click "yes", otherwise you'll have to do it manually afterwards.

The Mercurial client is now installed. Later on when installing Trac and getting it to work with Mercurial, I had issues because the Mercurial doesn't install the Mercurial bindings for Python. No only that, trying to use easy_install to install it didn't work because I didn't have Visual Studio 2003 installed on the server (nor did I want to install it!). It turns out that you can get round the issue by doing this:

1. Go to the Mercurial installation folder using Windows Explorer.
2. Locate the file Library.zip.
3. Open this file with an application like WinRAR or 7-zip (using the built-in Windows zip functionality doesn't work!).
4. Inside that archive there is a folder called mercurial. Extract this folder to a disk.
5. Open another Windows Explorer and navigate to your Python installation directory (eg. C:\Python25). Open up the Lib sub0folder (that's lib, not libs).
6. Copy or move the extracted mercurial folder to the lib folder.

Done! You've just "installed" the Mercurial bindings for Python. Later down the track, Trac will not crap out when you try and use it!

Server: Setting up TracMercurial

Given that we're using Mercurial for the back-end version control, we're going to need to set up and install the Mercurial plugin for Trac. This is where the need for SVN comes in. Yes, it's ironic that you need SVN to get something working with Mercurial! Just do it, ok!?

1. Execute the Subversion installer and let it go through with the default installation.
2. Make sure that the path to the SVN binaries is included in the system PATH environment variable.
3. Open up a command prompt and change directory to a temporary location.
4. run the command: svn co http://svn.edgewall.com/repos/trac/sandbox/mercurial-plugin-0.11 - this gets the right version of the plug-in from source.
5. run the command: cd mercurial-plugin-0.11
6. run the command: python setup.py bdist_egg - this creates an installable python egg.
7. run the command: python setup.py install - this installs the plug-in to the global location.

The TracMercurial plug-in is now installed. We'll need to configure it slightly when we've installed Trac. That information is listed in the next section.

Next we need to create a repository which will be used to house our source code. We'll pretend we're creating a project called "Slartibartfast".

1. Create a new folder on the file system somewhere meaningful, such as C:\Repos.
2. Open a command window, and change directory to that folder.
3. create a folder for your project: mkdir Slartibartfast.
4. Change to that folder: cd Slartibartfast.
5. Initialise the repository: hg init . (not the '.' at the end).

Your repository is now ready for code!

Server: Setting up Trac

We'll start with the basic install:

1. Execute the Trac installer.
2. Default installation settings are fine.
3. It will be installed to the scripts folder under the Python installation folder.

At this point it's a good idea to also add the scripts folder to the system PATH. This gives us the ability to run Python scripts from anywhere.

Next up, let's create a Trac project for our Slartitbartfast project.

1. Create a folder to house your Trac projects, say C:\Trac
2. Open a command window and change directory to your Trac projects folder.
3. Create a folder for your new project: mkdir Slartibartfast
4. Change to that folder: cd Slartibartfast.
5. Start the Trac administration application to start an interactive setup session: trac-admin . initenv

You need to fill out some values as it asks for them. Here's a sample interactive session that I did as a test run:

C:\Trac\Slartibartfast>trac-admin . initenv
Creating a new Trac environment at C:\Trac\Slartibartfast

Trac will first ask a few questions about your environment
in order to initialize and prepare the project database.

 Please enter the name of your project.
 This name will be used in page titles and descriptions.

Project Name [My Project]> Slartibartfast

 Please specify the connection string for the database to use.
 By default, a local SQLite database is created in the environment
 directory. It is also possible to use an already existing
 PostgreSQL database (check the Trac documentation for the exact
 connection string syntax).

Database connection string [sqlite:db/trac.db]>

 Please specify the type of version control system,
 By default, it will be svn.

 If you don't want to use Trac with version control integration,
 choose the default here and don't specify a repository directory.
 in the next question.

Repository type [svn]> hg

 Please specify the absolute path to the version control
 repository, or leave it blank to use Trac without a repository.
 You can also set the repository location later.

Path to repository [/path/to/repos]> C:\Repos\Slartibartfast

Creating and Initializing Project
 Installing default wiki pages

**
** craploads of import statements go here **
**

---------------------------------------------------------------------
Warning: couldn't index the repository.

This can happen for a variety of reasons: wrong repository type,
no appropriate third party library for this repository type,
no actual repository at the specified repository path...

You can nevertheless start using your Trac environment, but
you'll need to check again your trac.ini file and the [trac]
repository_type and repository_path settings in order to enable
the Trac repository browser.

---------------------------------------------------------------------
Project environment for 'Slartibartfast' created.

You may now configure the environment by editing the file:

  C:\TracProjects\Slartibartfast\conf\trac.ini

If you'd like to take this new project environment for a test drive,
try running the Trac standalone web server `tracd`:

  tracd --port 8000 C:\TracProjects\Slartibartfast

Then point your browser to http://localhost:8000/Slartibartfast.
There you can also browse the documentation for your installed
version of Trac, including information on further setup (such as
deploying Trac to a real web server).

The latest documentation can also always be found on the project
website:

http://trac.edgewall.org/

Congratulations!

You may notice the message: Warning: couldn't index the repository.
Don't be too concerned at this point, as we need to do a bit more to make sure that the Mercurial stuff is working.

Now we should test to make sure we're able to start the Trac daemon and see if the project site has been set up. We do that by executing the statement they suggested in the output:
tracd --port 8000 C:\TracProjects\Slartibartfast
Then we browse to http://localhost:8000 to see if it works! If all goes well you should see a project listing. Click on Slartibartfast and you should see something like this:

Trac is telling us why it's not able to index the repository, it's because we haven't enabled the plug-in yet. Let's do that now.

Open up the Trac.ini file which is located inside the conf folder under the main project folder (eg C:\Trac\Slartibartfast\conf\Trac.ini).

First, add the following section to the ini file:

[hg]
# -- Show revision number in addition to the changeset hash
show_rev = yes

# -- Changeset hash format
node_format = short
# hex:   Show the full SHA1 hash
# short: Show a shortened hash for the changesets

Then enable the Mercurial plugin by adding the following:

[components]
tracext.hg.backend.csetpropertyrenderer = enabled
tracext.hg.backend.hgdefaultpropertyrenderer = enabled
tracext.hg.backend.hgextpropertyrenderer = enabled
tracext.hg.backend.mercurialconnector = enabled

Note:Make sure that the [hg] and [components] sections don't already exist. If they do, then add the respective lines to the existing sections rather than creating new ones. For a fresh install, these sections shouldn't already exist, so you should be safe to add them.

Kill the tracd instance by press CTRL+C, then restart it (press the up arrow, then enter). Refresh your browser window and this time you should see this:

Excellent, our project is up and our repository is able to be read. We're done!

Server: Setting up the Trac Daemon as a Service

The tracd needs to run as a service. This is to make sure it's always running in a way that IIS can reference it. Given that we're later going to be hooking this up to IIS, we need to make sure it uses the AJP protocol. Here are the required steps:

1. Install the Windows 2003 resource kit.
2. Open a command window, and navigate to where the resource kit binaries are installed. On my system that was at C:\Program Files\Windows Resource Kits\Tools
3. Create a service called "Trac" by executing the following command: InstSrv Trac "C:\Program Files\Windows Resource Kits\Tools\SrvAny.exe" - (note the use of the full path is necessary for this to work).
4. Next we need to hack the registry a little to pass the right parameters and have the tracd started.
   1. Open regedit.
   2. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Trac
   3. Create a new sub-key called Parameters
   4. Inside this new sub-key, create two new values:
   1. Application of type String. Set the value to the full path of python.exe. Eg C:\python25\python.exe
   2. AppParameters of type String. Set the value to C:\python25\Scripts\tracd-script.py --port 8009 C:\Trac\Slartibartfast - (note the change of port number, it's there for a reason we'll cover shortly).
5. Open the services tool from the Control Panel, find the Trac service and start it up! Navigate to http://localhost:8009 to make sure it still works.

If you get the same site as you did before, then all is going well. If not, then you revisit the steps to make sure you haven't missed anything out and that your pointing to the right location of the trac project.

Now that we know this works, we need to change the AppParameters value in the registry so that Trac fires up using the appropriate protocol. Edit the value, and append the following to the end of the existing value: --protocol=ajp - this forces the use of AJP. AJP's default port is 8009, which is why we chose this value in the first place.

Restart the Trac service. If you attempt to browse to the site now you won't get anything meaningful, but it's ready to be talked to from IIS, which brings us to the next step.

Server: Setting up Apache Tomcat ISS ISAPI Filter in IIS 6.

To get IIS to talk to Trac, we need to add an ISAPI filter that's able to translate to the AJP protocol. This is what we'll be doing in this section.

First, we prepare the filter for use:

1. Create a folder to store the ISAPI file. I called mine C:\AJPConnector.
2. Copy the Apache Tomcat AJP ISAPI filter (isapi_redirect-1.2.28.dll) into the folder you created in the previous step, and rename it to isapi_redirect.dll.
3. Create a new text file in the same folder called isapi_redirect.properties. Open the file with a text editor and add the following content:

   # Configuration file for the ISAPI Redirector
   
   # The path to the ISAPI Redirector Extension, relative to the website
   # This must be in a virtual directory with execute privileges
   extension_uri=/AJPConnector/isapi_redirect.dll
   
   # Full path to the log file for the ISAPI Redirector
   log_file=C:\AJPConnector\isapi_redirect.log
   
   # Log level (debug, info, warn, error or trace)
   log_level=info
   
   # Full path to the workers.properties file
   worker_file=C:\AJPConnector\workers.properties
   
   # Full path to the uriworkermap.properties file
   worker_mount_file=C:\AJPConnector\uriworkermap.properties
   

4. Create another new text file in the same folder called workers.properties. Open it, and add this content:

   # Define 1 real worker
   worker.list=trac
   # Set properties for trac (ajp13)
   worker.trac.type=ajp13
   worker.trac.host=localhost
   worker.trac.port=8009
   worker.trac.socket_keepalive=0
   

5. Create another new text file in the same folder called uriworkermap.properties. Open it and add the following:

   /Slartibartfast*=trac
   

   You'll notice that the project name is what I've used here. Down the track if you want more projects, you need to add more lines to this file.

Next we need to install the ISAPI filter in IIS so that it can get used. Feel free to do this on the default website if you like. I created another website in IIS running on a different port, but you don't have to. Any time I refer to the website I'm referring to the site you have decided to install the filter on.

We need to enable the filter as a web service extension first. To do that, we follow these steps:

1. Open up the IIS manager.
2. Right-click on "Web Service Extensions" and select "Add a new web service extension".
3. In the resulting dialog box, click the "Add..." button and specify the file C:\AJPConnector\isapi_redirect.dll. Click "OK".
4. Specify the name "AJPConnector" and click "OK".

Next we need to add the filter to the website.

1. In IIS manager right-click on the website and view the properties.
2. Select the "ISAPI Filters" tab.
3. Click "Add".
4. Give the filter the name "AJPConnector".
5. Specify the full path to the dll: C:\AJPConnector\isapi_redirect.dll.
6. Click "OK".
7. Click on the "Home Directory" tab.
8. Make sure the "Execute Permissions" is set to "Scripts and Executables".
9. Make a note of the application pool that the web site is running under! You'll need this in a minute.
10. Click "OK" to close the dialog.

For the ISAPI filter to function, we need to make sure that IIS has access to the folder which the binary is stored in. To do this we need to know which account the website is running under.

1. In IIS manager open the list of Application Pools.
2. Choose the application pool that your website is running under. Rick-click and select properties.
3. Click on the "Identity" tab. This will show you the name of the account that your site is running under.
4. Browse to C:\AJPConnector, right-click and select "Properties", then choose the "Security" tab.
5. Click "Add" and type in the name of the account that you found listed in the "Identity" tab in the application pool properties. You can also use the search feature if you want to.
6. When the user has been recognised, click "OK".
7. Select the user in the list at the top of the dialog, and make sure that they have "Full Control" selected in the list at the bottom.
8. Click on the "Advanced" button.
9. Select the check box that says "Replace permission entries on all child objects with entries shown here that apply to child objects". Then press "OK". This will apply the permissions to all child objects in the folder.
10. Click "OK" to close the dialog box.

You may have noticed that in the isapi_redirect.properties file we specified an extension_uri property. This property specifies the path to use to get to the isapi_redirect.dll file via IIS. Notice how it's also a relative path from the root folder. Hence we need to create a virtual directory which maps to the same path.

1. In IIS manager right-click on the website and select "New" then "Virtual Directory...".
2. Fill out the wizard making sure you use the name "AJPConnector" for the directory name, and point it at C:\AJPConnector. The site also needs to be able to execute programs, not just scripts.

Restart IIS. The quickest way to do this is to type iisreset into a command prompt or into the "Run" dialog.

It's best at this point to make sure that the ISAPI filter has been loaded. To do this, simply open up the IIS manager again and browser to the sites "ISAPI Filters" tab. If all is working, then you should see something like this:

Notice the green arrow. If it's red and pointing downwards, then something is wrong. You need to make sure you have followed the above steps correctly. Double-check the security of the folders and make sure executables are enabled on your site.

If all is well then we can test it! Make sure your Trac service is running, then open a browser and navigate to your local IIS website. You should see the Trac Project website for Slartibartfast appear in your browswer. Yay!

It's now very easy to change IIS to function over HTTPS instead of HTTP. That's beyond the scope of this article as there are hundreds of blog entries and how-tos out there already.

Server: Setting up CopSSH

This is the final step! We need to give people access to the Mercurial repositories over SSH. For that we need to set up a functional SSH server and give them access to the repository path. On the surface these both seem like easy jobs. Unfortunately setting up SSH servers on Windows isn't pleasant and pointing everyone's home folders at a certain repository is not something that comes out of the box in Windows.

What we need to do is use the CopSSH installer to help with the "easy" installation of SSH. Then we need to use a program like Junction to provdie symbolic-link style functionality to the user's folders.

First up, run the CopSSH installer program. Follow all of the prompts and feel free to use the default installation. That was easy wasn't it! It should install an SSH server service for you. I found it quite hard to locate in the Service Management dialog, but eventually found it under the name "Openssh SSHD".

CopSSH comes with utilities which help you manage users. Those utilities are all in the programs menu and are easy to use. For me, those utilities weren't enough, because I knew that I was going to have to make sure that the folder links were set up on creation of user accounts and removed when those accounts were deleted.

It's a good move at this point to download Junction, if you haven't already, and put it in an easy to reach location. Feel free to add it to it's own folder or to system32. It's up to you. I'll assume you've put it in C:\tools.

I then created two scripts. The first lets you add users. It looks like this:

REM AddUser.bat
"C:\Program Files\ICW\Bin\copsshadm.exe" --command activateuser --user %1 --shell /bin/bash --passphrase %2
C:\tools\junction.exe "C:\Program Files\ICW\home\%1\Slartibartfast" C:\Repos\Slartibartfast

The first line of this file executes the user activation feature of CopSSH. It creates a new user from a local security login (so the users need to be accessible in Active Directory or in the local security set up). It creates a home folder for them and also creates a public/private key pair which it stores in their home folder as well. The keys have the passphrase that is passed to the script on the command line and will be used by the users to SSH in and commit changes to the repository. The public key for the user is added to the server as a recognised key which can be used to log in so we don't have to specify it manually.

The second line of the script executes the Junction program and links a folder called "Slartibartfast", located in the user's home folder, to the folder where the repository is stored.

To execute the script simply run: adduser.bat [username] [passphrase].
Please note that during my testing, I wasn't able to use a passphrase that had spaces. This isn't due to a bug in the script. Even if you replace --passphrase %2 with --passphrase "%2" it still doesn't work. It also doesn't work in the user interface tools. This appears to be a bug in CopSSH. This isn't a big issue though, just make sure you don't have spaces in your passphrase and you should be fine.

The second script is for removing users, it looks like this:

REM RemUser.bat
C:\tools\junction.exe -d "C:\Program Files\ICW\home\%1\Slartibartfast"
"C:\Program Files\ICW\Bin\copsshadm.exe" --command deactivateuser --user %1

It's obvious what this does. First it removes the symbolic link in the user's folder, then it uses CopSSH's command line tools to remove them as a valid user from the SSH server.

Before you assume that everything at this point is ready to rock, you may need to make sure that your firewall is updated so that the SSH port, 22, is open.

Congratulations, you are now finished with the server set up. Now let's get the clients working. Before you go to the next step, make sure you add a user!

Client: Setting up PuTTy

The first bit is easy, make sure that you've installed the PuTTy program and added the install path to the system's PATH environment variable.

Take a copy of the user's private key. This is generated by CopSSH and should be sitting in the user's home folder on the server. The key is usually called [username].key. Put the key on your local machine. We need to convert this key into a format that PuTTy can use.

1. Fire up the puttygen.exe utility. It comes with the PuTTy program. Simply type "puttygen" into the Run dialog box and it should come up if you've added the install folder to the system PATH.
2. Click "Load" and select the private key file you copied from the server.
3. Enter the user's passphrase into the dialog that appears and press "OK".
4. You should get a message stating ...

   Successfully imported foreign key
   (OpenSSH SSH-2 private key).
   To use this key with PuTTY, you need to
   use the "Save private key" command to
   save it in PuTTY's own format.

   Just press "OK".

5. Click "Save private key" and save the key in a safe location on your machine.

Your key can now be used by PuTTy.

When using SSH, you are going to have to have to specify the passphrase each time you connect to the server. This can get annoying quickly especially if you do a lot of commits. Instead of typing this in manually every time, I prefer to use the pageant.exe utility that also comes with PuTTy. This takes charge of handling the key passphrase for you.

1. From the command line or Run dialog, type pageant. You see a rather bland dialog appear.
2. Click "Add key". Browse to the folder where you have stored your new .ppk (the PuTTy version of your key) and select the key.
3. When prompted, specify the passphrase and click "OK".
4. You should now see an item appear in the list box. This is your private key. Pageant is now in memory and handling that key.
5. Close pageant, but you should still see it running in the system tray.

I also find it helpful to add pageant.exe to my start-up folder so that it is always running when I log in to my machine. You may choose to do the same.

Client: Setting up Mercurial

If you haven't already, make sure you install Mercurial on the client machine using the default settings. Make sure you tell the installer to add the Mercurial path to the system PATH.

The last step of configuration for the client is to tell Mercurial to use the PuTTy tools when using SSH. Mercurial can be configured by a user-specific configuration file called .hgrc. On Windows it can also be called Mercurial.ini. The file is located in your home folder. If you don't know what your home folder is, simply open a command prompt and type echo %USERPROFILE% - this will tell you the path.

If you haven't set up your configuration yet, then chances are the configuration file doesn't exist. So you'll have to create it. Create a file call either .hgrc or Mercurial.ini in your home folder manually, and open it in a text editor. Here is what part of mine looks like:

[ui]
username = OJ Reeves

editor = vim
ssh = plink -ssh -i "C:/path/to/key/id_rsa.ppk" -C -agent

The last line is the key and this is what you need to make sure it set properly. We are telling Mercurial to use the plink program. This also comes with PuTTy and is a command-line version of what the PuTTY program itself does behind the scenes. We also add a few parameters:

• -ssh : Indicates that we're using the SSH protocol.
• -i "file.ppk" : Specifies the location of the private key file we want to use to log in to the remote server. Change this to point to your local putty-compatible ppk private key. Make sure you user forward-slashes for the path separators as well!
• -C : This switch enables compression.
• -agent : This tells plink to talk to the pageant utility to get the passphrase for the key instead of asking you for it interactively.

The client is now ready to rock!

Finale

It took a while, but we got there in the end. Let's give our new-found Mercurial SSH server a spin. This should be easy as opening a command prompt at the appropriate location on disk and typing:

hg clone ssh://url.to.your.server.com/Slartibartfast

If everything has been configured properly, you should see Mercurial create a local folder called Slartibartfast which contains the repository. Of course, there won't be much in it because you've only created it recently! But you should be able to start using Mercurial to commit changes and push the committed changesets to the server just by running hg push.

The End

It took a while to figure most of this stuff out despite the documentation that exists on the web. It also took a bit of time to write this up! I hope that someone out there finds it useful.

The funny thing about all this, is that despite the pain, I found it easier to set up than TFS!

Feedback is always wanted and welcome. I hope it helps