Forge set-up

This page is under maintenance!
TODO: Set up sendmail. procmailrc for dynamic mailing lists

Passwords

…and other details to make note of, think about, or think up before starting.

adminPassword
This is the password of the privileged account you use to configure your system, the one you give when you sudo something. We'll refer to that account as me.
mysqlRootPassword
You will be asked to specify this when the MySQL is set up. Use it with the username root. You'll need it to set up and maintain the database that holds courtesy accounts.
mysqlForgePassword
You'll need to specify a less privileged user to access the database routinely. For these instructions, the username forge is used with this password.
forgePassword
You'll need a normal user to own the repositories and the web server. For these instructions, the username forge is used with this password.
bindUserPassword
You'll need an account to bind the LDAP client to AD. For these instructions, the username binduser is used with this password.

Standard software

This should get the lot:

sudo apt-get install mysql-server libaprutil1-dbd-mysql phpmyadmin libapache2-mod-php libaprutil1-ldap subversion mercurial git libapache2-mod-python libapache2-svn tidy python-subversion ldap-utils build-essential gawk cifs-utils xsltproc unoconv inkscape postfix procmail php-ldap php-mysql icoutils php-markdown librsvg2-bin texlive texlive-font-utils texlive-latex-base texlive-extra-utils texlive-latex-extra texlive-generic-extra texlive-science poppler-utils ps2eps

Installing mysql-server will require you to specify mysqlRootPassword, a root password for the MySQL database.

When asked about configuring phpmyadmin with dbconfig-common, say that you don't want to do anything yet.

SSH environment

The SSH environment file is not acted upon by default, so enable it for our repository types:

## At end of /etc/ssh/sshd_config:
Match user svn,git,hg
PermitUserEnvironment yes

PHP configuration

When used from the command line, PHP now doesn't pass environment into the $_ENV variable. You need to edit /etc/php/7.0/cli/php.ini, and set variables_order=EGPCS to allow hook scripts to receive environment when a Bash script calls PHP.

Enable Apache services

Enable built-in services, like HTTPS, DBD…:

sudo a2enmod ssl
sudo a2enmod dbd
sudo a2enmod authn
sudo a2enmod ldap
sudo a2enmod authnz_ldap
sudo a2enmod headers
sudo a2enmod authn_alias

Unprivileged account

Create a regular user to own repositories and the web server:

sudo adduser forge

Specify forgePassword as the password. You might have to specify the user id and group id to match the mount point that provides repositories via NFS, with --uid and --gid switches, or otherwise mask those differences in /etc/fstab. Don't give forge special permissions, e.g., making it a sudoer.

Create accounts svn, git and hg with the same UID and GID as forge, but with a different home directory:

sudo useradd -N -o -m -d /var/forge/service/svn-ssh -u $UID -g $GID svn
sudo useradd -N -o -m -d /var/forge/service/git-ssh -u $UID -g $GID git
sudo useradd -N -o -m -d /var/forge/service/hg-ssh -u $UID -g $GID hg

You can get the ids with:

id forge

These accounts must access the same data with the same permissions as forge, but allow programs to be invoked by SSH for different purposes, i.e., different repository types. ssh svn@example.org will employ files in ~svn/.ssh/ to run SVN-related commands, check SVN permissions, run SVN hooks, etc. Meanwhile, ssh git@example.org will employ files in ~git/.ssh/ to do corresponding Git-related operations.

Create a place to store state (repositories), generated resources (web pages, authorization files), and configuration:

sudo mkdir -p --mode=710 /etc/forge/secure/
sudo chown root.forge /etc/forge/secure
sudo mkdir -p /var/forge/
sudo chown -R forge.forge /var/forge

/var/forge/ is fully owned by forge, and is intended to simply be safely away from its home directory. Most of /etc/forge is to be owned by root, but readable by forge and all. Secure details are to be owned by root, but potentially readable only by forge.

Apache configuration

In /etc/apache2/envvars, set the web server to run as forge instead of the default www-data:

export APACHE_RUN_USER=forge
export APACHE_RUN_GROUP=forge

Static environment

All these files are root-owned, and forge should be able to read them, but not modify. There is, unfortunately, a lot of duplication among these files, but at least they are all in one directory.

Common shell environment

Create the file /etc/forge/profile.sh with this content:

export FORGE_SERVICE="/var/forge/service"
export FORGE_SVN_ROOT="/var/forge/repos/svn"
export FORGE_GIT_ROOT="/var/forge/repos/git"
export FORGE_HG_ROOT="/var/forge/repos/hg"
export FORGE_DOCROOT="${FORGE_SERVICE}/docroot"

This file can be sourced to find out where all the essential resources are. For example:

## In ~me/.bash_aliases

[ -r /etc/forge/profile.sh ] && . /etc/forge/profile.sh

SVN hook environment

User invocations of repository commands via SSH need to know where the RepoWebMan configuration is, and what charset to use. (TODO: Can we afford to allow the user to override the language, and force the charset to be what we want?) Create the file /etc/forge/ssh-environment with this content:

LANG=en_GB.UTF-8
REPOWEBMAN_CONFIG=/etc/forge/repowebman.ini

Create /etc/forge/svnserve.ini, used by SVN to check authorization and run hooks:

[general]
authz-db=/var/forge/service/svn-authz.conf
hooks-env=/etc/forge/svn-hooks-env.ini

Here is the SVN hook configuration to go in /etc/forge/svn-hooks-env.ini:

[default]
LANG=en_GB.UTF-8
PATH=/usr/bin:/bin:/var/forge/static/bin
REPOWEBMAN_CONFIG=/etc/forge/repowebman.ini
FORGE_HOME_LOCATION=https://forge.example.org/
FORGE_OPEN_HOME_LOCATION=http://forge.example.org/
FORGE_SVN_NEWS_DIR=/var/forge/service/svn-news
FORGE_SVN_SHADOW_DIR=/var/forge/service/svn-shadow

Virtual host configuration

You'll need several sections to deal with the multiple hostnames, and with having both open and SSL-protected hosts. Create /etc/forge/vhost.conf as the single point of reference for the site, and create /etc/forge/vhost-inner.conf for configuration that has to be repeated for multiple SSL hosts. Also set up /etc/forge/vhost-common.conf for configuration that can be used in both HTTPS and HTTP environments, but must be inside a <VirtualHost> block.

# In /etc/forge/vhost.conf

## By default, make all files inaccessible.  This is probably the
## default anyway, so it could be redundant.
<Directory />
  Options FollowSymLinks
  AllowOverride None
</Directory>

## Make a number of static resources publicly readable.
<Directory "/var/forge/service/docroot/">
  AllowOverride All
  Require all granted
</Directory>

## Make the imported static resources available without
## authentication.
<Directory "/var/forge/static/graphics/">
  Require all granted
</Directory>

## Make the hook-generated Atom feeds available without
## authentication.
<Directory "/var/forge/service/svn-news/">
  Require all granted
</Directory>

## Configure RepoWebMan and make it accessible.
<Directory "/usr/local/share/repowebman/">
  ## Tell RepoWebMan where to find its configuration.
  SetEnv REPOWEBMAN_CONFIG "/etc/forge/repowebman.ini"

  ## Run default configuration before every file.
  php_value auto_prepend_file "/usr/local/share/repowebman/defaults.inc.php"

  ## Make PHP files accessible without suffix.
  AllowOverride Options=Multiviews Indexes FileInfo

  ## RepoWebMan handles its own authorization.
  Require all granted
</Directory>

## Each HTTPS site alias needs its own certificate.
<VirtualHost *:443>
  SSLCertificateFile /etc/forge/server-org.crt
  ServerName forge.example.org
  Include /etc/forge/vhost-inner.conf
</VirtualHost>

<VirtualHost *:443>
  SSLCertificateFile /etc/forge/server-net.crt
  ServerName forge.example.net
  Include /etc/forge/vhost-inner.conf
</VirtualHost>

<VirtualHost *:443>
  SSLCertificateFile /etc/forge/server-com.crt
  ServerName forge.example.com
  Include /etc/forge/vhost-inner.conf
</VirtualHost>

## The HTTP sites share one configuration.
<VirtualHost *:80>
  ServerName forge.example.com
  ServerAlias forge.example.net
  ServerAlias forge.example.org

  SetEnv REPOWEBMAN_OPEN "yes"

  Include /etc/forge/vhost-common.conf

  ## Tell RepoWebMan that it's serving Git and SVN repositories
  ## entirely anonymously.
  SetEnv REPOWEBMAN_OPEN "yes"

  ## Some resources must be accessed via HTTPS.
  RewriteEngine On
  RewriteCond %{HTTPS} off
  RewriteRule (^/((dbadmin|manage|hosted)/.*|login))$ https://%{SERVER_NAME}$1 [R,L]

  ## Support some old URLs.
  RedirectPermanent /feeds/ "/svn-feeds/"

  ## Provide generated Atom feeds for SVN commits.
  Alias /svn-feeds/ "/var/forge/service/svn-news/"

  ## Set up an alternative stylesheet for the open site, so users have
  ## a visual hint about whether they are secure.
  Alias /styles/screen-secure.css /var/forge/service/docroot/styles/screen-open.css
</VirtualHost>
# In /etc/forge/vhost-common.conf

## Errors and logging are the same on all hosts.
ServerAdmin adminEmailAddress
ErrorLog /var/log/apache2/forge-error.log
CustomLog /var/log/apache2/forge-access.log common
ErrorDocument 401 /errors/401
ErrorDocument 403 /errors/403

## Make some resources available to both HTTP
## and HTTPS clients.
DocumentRoot /var/forge/service/docroot/
Alias /graphics/extras/ "/var/forge/static/graphics/extras/"
Alias /graphics/pixelmixer/ "/var/forge/static/graphics/pixelmixer/"

## Let RepoWebMan check authorization before passing
## on to Git's backend.
ScriptAliasMatch \
  "(?x)^/git-repos/.*\.git/(info/refs|git-(upload|receive)-pack)$" \
  "/usr/local/share/repowebman/cgi/git-backend"

## Direct browsing of Git to RepoWebMan.
ScriptAlias /git-repos "/usr/local/share/repowebman/git-web.php"

## Direct browsing of SVN to RepoWebMan.
RewriteEngine On
RewriteCond %{REQUEST_METHOD} =GET
RewriteCond %{REQUEST_URI} ^/svn-repos/
RewriteCond %{REQUEST_URI} !^/[^/]+/[^/]+/!svn/
RewriteRule ^/[^/]+(/.*)$ \
  "/usr/local/share/repowebman/svn-web.php$1" \
  [L,H=application/x-httpd-php]

## Let DAV handle other SVN requests.
<Location "/svn-repos/">
  DAV svn
  SVNParentPath /var/forge/repos/svn
  AuthzSVNAccessFile /var/forge/service/svn-authz.conf
  SVNHooksEnv /etc/forge/svn-hooks-env.ini
</Location>

SVNUseUTF8 On
# In /etc/forge/vhost-secure.conf

## Enable SSL on HTTPS sites.
SSLEngine on
SSLCertificateKeyFile /etc/forge/secure/privkey.pem
SSLCipherSuite \
      ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
<FilesMatch "\.(cgi|shtml|phtml|php)$">
  SSLOptions +StdEnvVars
</FilesMatch>

Include /etc/forge/vhost-common.conf

## This matches the [auth] login page setting of repowebman.ini.
## Links to log in or switch user go here, so that credentials you use
## will apply to the root of the site's file hierarchy.
ScriptAlias /login "/usr/local/share/repowebman/htdocs/login.php"
<Location "/login">
  Require valid-user
  Satisfy all
</Location>

## Enable the dynamic resources handled by PHPMyAdmin and RepoWebMan.
Alias /dbadmin/ "/usr/share/phpmyadmin/"
Alias /manage/ "/usr/local/share/repowebman/htdocs/"

## This matches the [auth] login page setting of repowebman.ini.
## Links to log in or switch user go here, so that credentials you use
## will apply to the root of the site's file hierarchy.
ScriptAlias /login "/usr/local/share/repowebman/htdocs/login.php"

## The old SVN Atom feeds are now only on the open site, and under an
## SVN-specific virtual path.
RewriteEngine On
RewriteRule ^/feeds/(.*) http://%{SERVER_NAME}/svn-feeds/HTML R,L 

## Specify the authentication mechanisms on
## the HTTPS sites.
<Location "/">
  AuthType Basic
  AuthName "Forge@Lancaster"
  AuthBasicAuthoritative on
  AuthBasicProvider mysql-email mysql-username ldap-email ldap-username
</Location>

## Redirect resources that only need to be available
## on the open site to the open site.
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/((svn-repos|svn-shadow|git-repos|hg-repos|manage|dbadmin|xslt|styles|hosted|graphics|errors|scripts)/|login)
RewriteRule ^/(.*)$ "http://%{SERVER_NAME}/$1" [R]

## Require authentication on SVN/HTTPS, while
## permitting anonymous access.
<Location "/svn-repos/">
  Require valid-user
  Satisfy any
</Location>

## Require authentication on Git/HTTPS.
<Location "/git-repos/">
  Require valid-user
  Satisfy all
</Location>

## Expose the database only to superusers.
<Location "/dbadmin/">
  DirectoryIndex index.php
  Options None
  AuthGroupFile /var/forge/service/groups.tab
  Require group super
</Location>

Enable this virtual host:

sudo ln -s /etc/forge/vhost.conf /etc/apache2/sites-enabled/050-forge

Apache authentication methods

Create two authentication aliases for MySQL:

# In /etc/forge/mysql.conf
DBDriver mysql
DBDPersist On
DBDMin  1
DBDKeep 2
DBDMax  10
DBDExptime 60
Include /etc/forge/secure/dbd-cred.conf

<AuthnProviderAlias dbd mysql-username>
  AuthDBDUserPWQuery "SELECT password FROM credentials WHERE username = %s"
</AuthnProviderAlias>

<AuthnProviderAlias dbd mysql-email>
  AuthDBDUserPWQuery "SELECT password FROM credentials WHERE email = %s"
</AuthnProviderAlias>

…and two for LDAP:

# In /etc/forge/ldap.conf
<AuthnProviderAlias ldap ldap-email>
  AuthLDAPURL ldap://your.domain.controller.example:389/OU=...,DC=...?mail?sub?(objectClass=user)
  Include /etc/forge/secure/ldap-cred.conf
</AuthnProviderAlias>

<AuthnProviderAlias ldap ldap-username>
  AuthLDAPURL ldap://your.domain.controller.example:389/OU=...,DC=...?sAMAccountname?sub?(objectClass=user)
  Include /etc/forge/secure/ldap-cred.conf
</AuthnProviderAlias>

…and enable them:

sudo ln -s /etc/forge/ldap.conf /etc/apache2/mods-enabled/forge-ldap.conf
sudo ln -s /etc/forge/mysql.conf /etc/apache2/mods-enabled/forge-mysql.conf

Secure details

You need to create several files to hold vital credentials. Create blank files and give them restrictive permissions first:

sudo touch /etc/forge/secure/{dbd,ldap}-cred.conf
sudo touch /etc/forge/secure/{mysql,ldap}.ini
sudo chmod 600 /etc/forge/secure/{dbd,ldap}-cred.conf
sudo chmod 640 /etc/forge/secure/{mysql,ldap}.ini
sudo chown root.forge /etc/forge/secure/{mysql,ldap}.ini

Then provide the content:

# To go in /etc/forge/secure/dbd-cred.conf
DBDParams "dbname=forge user=forge pass=mysqlForgePassword sock=/var/run/mysqld/mysqld.sock"
# To go in /etc/forge/secure/ldap-cred.conf
AuthLDAPBindDN "CN=binduser,OU=...,DC=..."
AuthLDAPBindPassword bindUserPassword

You should be able to check this way:

ldapsearch -x -D 'CN=binduser,OU=...,DC=...' -W \
 -H 'ldap://your.domain.controller.example:389' \
 -b 'OU=...,DC=...' \
 -s sub 'sAMAccountName=user'

If you haven't preserved the private key used for HTTPS, you'll have to create a new one:

sudo openssl genrsa -out /etc/forge/secure/privkey.pem 1024

If not already created, you will need to make an unsigned certificate request server.csr:

sudo openssl req -new \
    -key /etc/forge/secure/privkey.pem \
    -out /etc/forge/server.csr

To create or renew the self-signed certificate used here, server.crt, use:

sudo openssl x509 -req -days 999 \
    -in /etc/forge/server.csr \
    -signkey /etc/forge/secure/privkey.pem \
    -out /etc/forge/server.crt

Here's how to extract fingerprints from the new certificate, so you can publish them:

openssl x509 -noout -sha1 -fingerprint -in /etc/forge/server.crt
openssl x509 -noout -md5 -fingerprint -in /etc/forge/server.crt

phpMyAdmin configuration

Enter database login details into /etc/phpmyadmin/config-db.php (/var/lib/phpmyadmin/config.inc.conf doesn't seem to be the right location any more), to allow the database to be manipulated on-line. This makes sure that authentication is done by the HTTP server, rather than by phpMyAdmin itself:

$cfg['Servers'][]['connect_type'] = 'socket';
$cfg['Servers'][]['extension'] = 'mysqli';
$cfg['Servers'][]['auth_type'] = 'config';
$cfg['Servers'][]['user'] = 'root';
$cfg['Servers'][]['password'] = 'mysqlRootPassword';

Make sure forge can read it:

sudo chown root.forge /etc/phpmyadmin/config-db.php
sudo chmod 640 root.forge /etc/phpmyadmin/config-db.php

Disable default exposure of phpMyAdmin, in /etc/phpmyadmin/apache.conf:

#Alias /phpmyadmin /usr/share/phpmyadmin

Database

The database holds details of the following:

  • courtesy accounts
  • hook scripts
  • public keys, used to permit SSH access
  • group membership and invitations
  • user associations
  • SVN/Git/Hg permissions
  • directory meta-data

Each time SVN/Git/Hg permissions or group membership are changed, new SVN/Git/Hg authorization files are generated in /var/forge/service/.

Enter the database server to create a less privileged account, the database itself, and some tables:

sudo mysql --password

You might have to enter adminPassword first for the sake of sudo, but the prompt should be clear about this. Then you will definitely have to enter mysqlRootPassword, which you specified before.

Create a database forge:

CREATE DATABASE `forge` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
USE `forge` ;

Create a less privileged user forge within MySQL to access the forge database routinely:

CREATE USER 'forge'@'localhost' IDENTIFIED BY 'mysqlForgePassword';
GRANT LOCK TABLES, SELECT, INSERT, UPDATE, DELETE, EXECUTE ON  `forge`.* TO 'forge'@'localhost';

You can get the database structure from dbstructure.sql, but note that it is currently on Forge@Lancaster. If that's not up (because you're recovering it), you might still have direct access to the repository, and will have to change the URL accordingly:

svn cat "${BASE}/webtools/repowebman/branches/stable/dbstructure.sql" > dbstructure.sql
mysql forge < dbstructure.sql

Maintenance directory structure

As forge:

mkdir -p /var/forge/service/{{git,hg,svn}-{news,shadow},docroot}/
mkdir --mode=700 -p /var/forge/service/{git,hg,svn}-ssh/.ssh
mkdir -p /var/forge/static/{{git,hg,svn}-hooks,bin}/

Raw repository set-up

If you're starting from scratch, or recovering from dumps, simply create directories for repositories (as forge):

mkdir -p /var/forge/repos/{git,hg,svn}

Otherwise, set up /etc/fstab and mount the appropriate resource on /var/forge/repos/.

If you're rebuilding Forge@Lancaster, some of the repositories containing code to build it are in it! You can use this in the commands below to fetch them:

BASE="$FORGE_SVN_ROOT/"

Otherwise, you're building your own site, and should use this:

BASE='https://scc-forge.lancaster.ac.uk/svn-repos/'

Dynamic web interface

As a sudoer:

mkdir -p ~/works
svn co "${BASE}webtools/repowebman/branches/stable" ~/works/repowebman
cd ~/works/repowebman
make && sudo make install

This installs mainly PHP files in /usr/local/share/repowebman, as root so the web server can't tamper with them. You can always update this with:

cd ~/works/repowebman
svn up && make && sudo make install

SSH account environment

As forge:

mkdir -p --mode=700 ~{svn,git,hg}/.ssh
ln -s /etc/forge/ssh-environment ~svn/.ssh/environment
ln -s /etc/forge/ssh-environment ~svn/.git/environment
ln -s /etc/forge/ssh-environment ~svn/.hg/environment

Hook configuration

Hook scripts run under the maintenance account forge.

Case-insensitive uniqueness check

Create /var/forge/static/svn-hooks/case-insensitive-uniqueness:

#!/bin/bash

REPOS="$3"
VPATH="$2"
TXN="$4"

if ! case-insensitive-uniqueness "$REPOS" "$TXN" ; then
    printf >&2 'Paths below %s in %s must be case-insensitively unique\n' \
               "$VPATH" "$REPOS"
    exit 1
fi

Create an executable hook script in /var/forge/service/scripts/hooks/case-insensitive-uniqueness to prevent commits that would result in some file names being distinct only by case. Use case-insensitive.py from the contrib directory in the Subversion source tar:

wget 'http://svn.apache.org/repos/asf/subversion/trunk/contrib/hook-scripts/case-insensitive.py' -O /var/forge/static/bin/case-insensitive-uniqueness

Make sure both are executable:

chmod 755 /var/forge/static/bin/case-insensitive-uniqueness
chmod 755 /var/forge/static/svn-hooks/case-insensitive-uniqueness

Atom feeds

Create a hook script in /var/forge/static/svn-hooks/make-atom to build Atom feeds:

#!/bin/bash

repodir="$2"
diff="${3:-9}"
repopath="${4}"
rev1="${5}"


rev0="$((rev1 - diff))"
if [ "${rev0}" -lt 0 ] ; then rev0=0 ; fi

repopath="${repopath%%/}"
reponame="${repopath##*/}"


title="$reponame"


FEED_BASE="${FORGE_OPEN_HOME_LOCATION}feeds/"
VIEW_BASE="${FORGE_HOME_LOCATION}manage/svn-info/"
SVN_BASE="${FORGE_HOME_LOCATION}svn-repos/"

FEED_FILE="${FORGE_SVN_NEWS_DIR%%/}/$reponame${repodir%/}.atom"

function common_prefix () {
    local prefix line result prefix_top line_top

    read line
    if [ "${line:0:1}" != '/' ] ; then
        line="/$line"
    fi
    prefix="$line"

    while read line ; do
        if [ "${line:0:1}" != '/' ] ; then
            line="/$line"
        fi
        result=""
        while true ; do
            prefix_top="${prefix%%/*}"
            prefix="${prefix#*/}"
            line_top="${line%%/*}"
            line="${line#*/}"
            if [[ "$prefix_top" != "$line_top" ]] ; then
                break;
            fi
            result="$result$prefix_top/"
        done
        prefix="$result"
    done

    echo "$prefix"
}

function xmlesc () {
    echo "$1" | \
        sed -e 's/&/\&amp;/g' -e 's/</\&lt;/g' -e 's/>/\&gt;/g' \
        -e 's/"/\&quot;/g' -e "s/'/\&apos;/g"
}

mkdir -p "${FEED_FILE%/*}"

(
    printf "<?xml version='1.0' ?>\n"
    printf "<feed xmlns='http://www.w3.org/2005/Atom'>\n"
    printf "<title>Commits on %s:%s</title>\n" "$(xmlesc "$title")" "$repodir"
    printf "<id>${FEED_BASE}%s%s.atom</id>\n" "$reponame" "${repodir%/}"
    printf "<link href='%s%s%s' />\n" \
           "$SVN_BASE" "$reponame" "$repodir"
    printf "<updated>%s</updated>\n" "$(date -u "+%Y-%m-%dT%T.%N%z")"

    for ((depth = diff ; depth > 0 && rev1 >= 0 ; rev1-- )) ; do
        prefix="$(svnlook dirs-changed -r "$rev1" "$repopath" | common_prefix)"
        if [ "${prefix#$repodir}" = "$prefix" ] ; then continue; fi

        log="$(svnlook log -r "$rev1" "$repopath")"
        author="$(svnlook author -r "$rev1" "$repopath")"

        when="$(svnlook date -r "$rev1" "$repopath")"
        when="$(date -d "$when" -u "+%Y-%m-%dT%T%z")"


        printf "<entry>\n"

        printf "<id>${FEED_BASE}%s%s.atom/%d</id>\n" \
               "$reponame" "${repodir%/}" "$rev1"
        printf "<link href='${VIEW_BASE}%s%s?pathrev=%d' />\n" \
               "${reponame}" "$repodir" "${rev1}"

        printf "<title>r%d: %s</title>\n" "$rev1" "$(xmlesc "$prefix")"

        printf "<summary>%s</summary>\n" "$(xmlesc "$log")"
        printf "<updated>%s</updated>\n" "$(xmlesc "$when")"
        printf "<author>%s</author>\n" "$(xmlesc "$author")"

        printf "</entry>\n"

        depth=$((depth - 1))
    done

    printf "</feed>\n"
) | tidy -xml 2> /dev/null > "$FEED_FILE-tmp"

mv "$FEED_FILE-tmp" "$FEED_FILE"

Make it executable:

chmod 755 /var/forge/static/svn-hooks/make-atom

Emails on commit

Create a hook script in /var/forge/static/svn-hooks/email-on-commit to send an email on each commit:

#!/bin/bash

TYPE="${0##*/}"

HOOK="$1"
shift

VPATH="$1"
shift

ADDR="$1"
shift

SUBJ="$1"
shift

REPOS="$1"
shift

REV="$1"
shift

TXN="$1"
shift

function common_prefix () {
    local prefix line result prefix_top line_top

    read line
    if [ "${line:0:1}" != '/' ] ; then
        line="/$line"
    fi
    prefix="$line"

    while read line ; do
        if [ "${line:0:1}" != '/' ] ; then
            line="/$line"
        fi
        result=""
        while true ; do
            prefix_top="${prefix%%/*}"
            prefix="${prefix#*/}"
            line_top="${line%%/*}"
            line="${line#*/}"
            if [[ "$prefix_top" != "$line_top" ]] ; then
                break;
            fi
            result="$result$prefix_top/"
        done
        prefix="$result"
    done

    echo "$prefix"
}

function encodepath() {
    local arg="$1"
    while true ; do
        local first="${arg%%/*}"
        encode "$first"
        if [ "$first" = "$arg" ] ; then return ; fi
        arg="${arg#*/}"
        printf '/'
    done
}

function encode() {
    echo -n "$@" | perl -MURI::Escape -ne 'print uri_escape($_)'
}

function escape() {
    echo -n "$@" | sed -f "${0%/*}/escape-html.sed"
}

author="$(svnlook author -r "$REV" "$REPOS")"
prefix="$(svnlook dirs-changed -r "$REV" "$REPOS" | common_prefix)"
date="$(svnlook date -r "$REV" "$REPOS")"
maildate="$(date -d "$date" '+%a, %d %b %Y %H:%M:%S %z')"
subject="$(printf '%s r%d %s:%s\n' "$SUBJ" "$REV" "${REPOS##*/}" "$prefix")"

(
    printf 'Subject: %s\n' "$subject"
    printf 'To: %s\n' "$ADDR"
    printf 'From: "Forge@Lancaster" <forge@comp.lancs.ac.uk>\n'
    printf 'Date: %s\n' "$maildate"
    printf 'X-SVN-Hook: %s\n' "$HOOK"
    printf 'X-SVN-Author: %s\n' "$author"
    printf 'X-SVN-Repository: %s\n' "${REPOS##*/}"
    printf 'X-SVN-Common-Prefix: %s\n' "$prefix"
    printf 'X-SVN-Revision: %s\n' "$REV"
    if [ "$TYPE" != email-on-commit-signal ] ; then
        printf 'Content-Type: text/html; charset=UTF-8\n'
    fi
    while read dir ; do
        if [ "${dir:0:1}" != '/' ] ; then dir="/$dir" ; fi
        printf 'X-SVN-Path: %s\n' "$dir"
    done < <(svnlook dirs-changed -r "$REV" "$REPOS")
    printf '\n'

    if [ "$TYPE" = email-on-commit-signal ] ; then
        echo blank
    else
        printf '<html lang="en">\n'
        printf '<head>\n'
        printf '<title>%s</title>\n' "$(escape "$subject")"
        printf '</head>\n'
        printf '<body>\n'
        printf '<h1>Forge@Lancaster SVN Commit notification</h1>\n'
        printf '<table>\n'
        printf '<tr><th valign=top align=right>Repository:</th> <td valign=top align=left>%s</td></tr>\n' "$(escape "${REPOS##*/}")"
        printf '<tr><th valign=top align=right>Prefix:</th> <td valign=top align=left><samp>%s</samp></td></tr>\n' "$(escape "$prefix")"
        printf '<tr><th valign=top align=right>Author:</th> <td valign=top align=left><samp>%s</samp></td></tr>\n' "$(escape "$author")"
        printf '<tr><th valign=top align=right>Date:</th> <td valign=top align=left>%s</td></tr>\n' "$(escape "$date")"
        printf '<tr><th valign=top align=right>Revision:</th> <td valign=top align=left><a href="%smanage/svn-info/%s%s?pathrev=%d">%d</a></td></tr>\n' "${FORGE_HOME_LOCATION}" "$(encode "${REPOS##*/}")" "$(encodepath "$prefix")" "$REV" "$REV"
        printf '</table>\n'
        printf '<h2>Log</h2>\n'
        printf '<pre style="white-space: pre-wrap">\n'
        svnlook log -r "$REV" "$REPOS" | sed -f "${0%/*}/escape-html.sed"
        printf '</pre>\n'
        printf '<h2>Files</h2>\n'
        printf '<pre>\n'
        svnlook changed -r "$REV" "$REPOS" | sed -f "${0%/*}/escape-html.sed"
        printf '</pre>\n'
        if [ "$TYPE" = email-on-commit-detailed ] ; then
            printf '<h2>Diff</h2>\n'
            printf '<pre style="white-space: pre-wrap">\n'
            svnlook diff -r "$REV" "$REPOS" | sed -f "${0%/*}/escape-html.sed"
            printf '</pre>\n'
        fi
        printf '</body>\n'
        printf '</html>\n'
    fi
) | /usr/sbin/sendmail "$ADDR"

Make hard links to this file, and make them executable:

ln /var/forge/static/svn-hooks/email-on-commit /var/forge/static/svn-hooks/email-on-commit-detailed
ln /var/forge/static/svn-hooks/email-on-commit /var/forge/static/svn-hooks/email-on-commit-signal
chmod 755 /var/forge/static/svn-hooks/email-on-commit

Read-only repositories

Create a hook script in /var/forge/static/svn-hooks/read-only to prevent any commits:

#!/bin/bash

printf 'Repository is read-only\n'

exit 1

Make it executable:

chmod 755 /var/forge/static/svn-hooks/read-only

Block suffixes

#!/bin/bash

REPOS="$4"
VPATH="$2"
TXN="$5"
SUFFIXES=($3)

function check () {
    local line
    local result
    result=()
    while read line ; do
        line="${line:4}"
        for suffix in "${SUFFIXES[@]}" ; do
            if [ "${line%$suffix}" != "$line" ] ; then
                printf 'Suffix %s forbidden in filename %s\n' \
                    "$suffix" "$line" >&2
                result=("${result[@]}" "$suffix")
            fi
        done
    done
    echo "${result[@]}"
    return ${#result[@]}
}


bad=($(svnlook changed -t "$TXN" "$REPOS" | egrep '^A' | check))
code="$?"
if [ "$code" != 0 ] ; then
    printf >&2 'Forbidden suffix: %s\n' "${bad[@]}"
    printf '\nSee:\n   %scommit\n' "$FORGE_HOME_LOCATION" >&2
    exit 1
fi
exit

Website creation

Log in as forge. Get a load of software. Whether you need all of these depends on whether you use them to maintain this account and the webpages. I include them here to assist rebuilding of Forge@Lancaster:

mkdir -p ~/works
svn co "$BASE"utils/ss-scripts/trunk ~/works/ss-scripts
svn co "$BASE"misc/jardeps/branches/external ~/works/jardeps
svn co "$BASE"utils/mokvino/branches/stable ~/works/mokvino
svn co "$BASE"webtools/m5web/branches/stable ~/works/m5web
svn co "$BASE"misc/svn-site/branches/stable ~/works/docsrc

Prepare to install ss-scripts:

export MAKEFLAGS="-I${HOME}/.install/etc -I${HOME}/.install/include"
mkdir -p ~/.install/{etc,include}
# In ~/.install/etc/ss-scripts-env.mk
PREFIX=${HOME}/.install
CFLAGS += -g -O2
CFLAGS += -std=gnu99 -pedantic -Wall -W
CFLAGS += -Wno-unused-parameter
CPPFLAGS += -D_XOPEN_SOURCE=500
cd ~works/ss-scripts
make && make install

Create ~/.bash_aliases:

#!/bin/bash

export MAKEFLAGS="-I${HOME}/.install/etc -I${HOME}/.install/include"

if [ -r ~/.install/etc/bash.pathcomp ] ; then
    . ~/.install/etc/bash.pathcomp
fi

prefix MANPATH /usr/share/man

prefix PATH ~/.install/bin
prefix MANPATH ~/.install/man

[ -r "/etc/forge/profile.sh" ] && . "/etc/forge/profile.sh"

While keeping your log-in open, log in again to get another shell, and check that MAKEFLAGS is set.

.htaccess

Create /var/forge/service/docroot/.htaccess:

Options +MultiViews +ExecCGI -Indexes

AddLanguage en-GB .en-GB

ErrorDocument 401 /errors/401
ErrorDocument 403 /errors/403

AddType text/xsl .xsl
AddEncoding x-gzip .svgz

Header set opt "\"http://standard-sitemap.org/2007/ns\"; ns=15"
Header set 15-Location "/navigate.xml"

AddHandler send-as-is asis

<Files "favicon.ico">
  AddType text/plain .ico
</Files>

Local package configuration

Create configuration in ~/.install/etc for several packages built locally:

# In ~/.install/etc/svn-site-env.mk
WWWPREFIX=$(FORGE_DOCROOT)/

STDMAP_LNK_SUFFIX=
HTML_LNK_SUFFIX=

INDEXBASE=index

DOCUMENT_BASE=/

COMMAND_PREFIX=/manage/
DBADMIN_PREFIX=/dbadmin/
PIXELMIXER_PREFIX=/graphics/pixelmixer/
# In ~/.install/etc/m5web-env.mk
PREFIX=${HOME}/.install
# In ~/.install/etc/jardeps-env.mk
PREFIX=${HOME}/.install
# In ~/.install/etc/mokvino-env.mk
PREFIX=${HOME}/.install

Building local software

Create a works directory to keep working copies:

cd ~/works/jardeps
make && make install

cd ~/works/mokvino
make && make install

cd ~/works/m5web
make && make install

cd ~/works/docsrc
make

Database back-up procedure

[mysqldump]
user=forge
password=mysqlForgePassword

Perform daily:

mysqldump --skip-dump-date \
   forge | bzip2 -9 > /backup/forge-0.sql.bz2

To restore:

bunzip2 < /backup/forge-0.sql.bz2 | mysql forge
crontab -e
## Every Sunday morning, back up the repositories, database and authorization.
0 1 * * 7 /.install/lib/do-backup > ~/.install/tmp/backup-svn.log 2>&1

Here's a script to back up SVN repositories incrementally:

#!/bin/bash

## Keep track of files to be deleted on exit.
tmpfiles=()
function tidy_up () {
    printf 'Removing %s\n' "${tmpfiles@}"
    rm -f "${tmpfiles@}"
}
trap tidy_up EXIT

## Parse arguments.
reponames=()
while $# -gt 0 ; do
    case "$1" in
        -s)
            shift
            rootdir="${1%/}"
            ;;

        -d)
            shift
            backupdir="${1%/}"
            ;;

        -*)
            printf > /dev/stderr 'Unknown switch %s\n' "$1"
            exit 1
            ;;

        *)
            reponames=("${reponames@}" "$1")
    esac
    shift
done

## Check for misconfiguration.
if -z "$rootdir" ; then
    printf > /dev/stderr 'Set source directory with -s\n'
    exit 1
fi

if -z "$backupdir" ; then
    printf > /dev/stderr 'Set destination directory with -d\n'
    exit 1
fi

printf 'Repository home: %s\n' "$rootdir"
printf 'Back-up directory: %s\n' "$backupdir"


dumpfilesfx=".svn.bz2"

function dump_range () {
    local repodir="$1"
    shift
    local repo="$1"
    shift
    local first="$1"
    shift
    local last="$1"
    shift

    local dumpfile="$backupdir/$repo.$first-$last$dumpfilesfx"
    local dumptmpfile="$dumpfile-tmp"
    tmpfiles=("${tmpfiles@}" "$dumptmpfile")

    svnadmin dump -q --incremental -r "${first}:${last}" "$repodir" | \
        bzip2 -9 > "$dumptmpfile" && \
        mv  "$dumptmpfile" "$dumpfile"
}

## extglob allows us to use +(0-9) to match one or more digits in
## filenames.  nullglob makes sure that, if no filename matches a
## pattern, an empty list is produced.
shopt -s extglob nullglob

for repo in "${reponames@}" ; do
    ## Identify all the files and directories to be processed.
    repodir="$rootdir/$repo"
    dumpfilepfx="$backupdir/$repo."

    ## Skip if we don't have a repository in the specified location.
    if ! svnlook youngest "$repodir" > /dev/null 2> /dev/null ; then
        printf > /dev/stderr '%s: invalid repository\n' "$repo"
        continue
    fi

    ## Get a list of existing files for this repository.
    existing=("${dumpfilepfx}"+(0-9)"-"+(0-9)"${dumpfilesfx}")
    # if "${existing}" = "${dumpfilepfx}+(0-9)-+(0-9)${dumpfilesfx}" ; then
    #   existing=()
    # fi
    existing=("${existing@#${dumpfilepfx}}")
    existing=("${existing@%${dumpfilesfx}}")
    existing=($(printf '%s\n' "${existing@}" | sort -n))

    printf "%s: Got %s\n" "$repo" "${existing*}"
    newrev="$(svnlook youngest "$repodir")"

    existing=("${existing@}" "$((newrev+1))-$((newrev+2))")

    ## Look for gaps.
    expected=0
    for pair in "${existing@}" ; do
        start="${pair%-*}"
        end="${pair#*-}"
        if $expected -ne $start ; then
            first="$expected"
            last="$((start-1))"
            printf '  Preserving %d-%d\n' "$first" "$last"
            dump_range "$repodir" "$repo" "$first" "$last"
        fi
        expected=$((end+1))
    done
done

To back up SVN repositories foo, bar and baz from $FORGE_SVN_ROOT into /mnt/backup-dir-svn/:

backup-svn-repo -s "$FORGE_SVN_ROOT" -d '/mnt/backup-dir/svn' foo bar baz

You should see files of the form foo.r0-r1.svn.bz2 created, for revisions r0 through to r1.


Valid HTML 4.01!
Made with Mokvino
The Standard-Sitemap ProtocolSitemap Supported