An Open Access Peon

Ubuntu 11.10 on Sun X4540 ("Thumper")

2011-10-26T12:45:00.003+01:00

I installed Ubuntu 11.10 Server x64 edition using the Java ILOM client - you must use a 32bit Java client to connect a CD image. I set up a MD RAID1 across the bootable devices of controllers 0 and 1 (first disks on 0 and 1).

Post-install I encountered a blank, black screen on the GRUB 2 stage (i.e. after the kernel selection screen). To fix this edit the boot parameters on the kernel selection screen:

set gfxpayload=text...linux [...] rootdelay=90
To make these changes permanent after booting:

sudo vi /etc/default/grubGRUB_CMDLINE_LINUX_DEFAULT="rootdelay=90"GRUB_GFXMODE=textsudo update-grub

(reboot to check everything worked correctly)

Installing native ZFS and setting up a ZFS pool

sudo apt-get install python-software-propertiessudo add-apt-repository ppa:zfs-native/stablesudo apt-get updatesudo apt-get install ubuntu-zfs

I created a small Perl script to set up a zpool over the remaining 46 disks. The scheme I use is 4 hot spares (the first disk of the other 4 controllers) with 7 raidz RAIDs over 1 disk per controller.

Usage:

Warning! This will destroy any data you have on the disks in your system. Only use this if you *really* know what you're doing.

sudo perl [script.pl] --create

The source code for the script:

#!/usr/bin/perluse Getopt::Long;use strict;use warnings;my $usage = "$0 [--dry-run --create --destroy --dump]";GetOptions(        'dry-run' => \(my $dry_run),        'create' => \(my $opt_create),        'destroy' => \(my $opt_destroy),        'dump' => \(my $opt_dump),        'help' => \(my $opt_help),) or die "$usage\n";die "$usage\n" if $opt_help;my @DISKS;open(my $fh, "<", "/var/log/dmesg") or die "Error opening dmesg: $!";while(<$fh>){        next if $_ !~ /Attached SCSI disk/;        s/^\[[^\]]+\]\s*//;        die if $_ !~ /sd\s+(\d+):0:(\d+):0:\s+\[(\w+)\]/;        my( $c, $t, $dev ) = ($1, $2, $3);        $DISKS[$c][$t] = "/dev/$dev";}shift @DISKS while !defined $DISKS[0];if( $opt_dump ){        foreach my $i (0..$#DISKS)        {                print "Controller $i:\n";                foreach my $j (0..$#{$DISKS[0]})                {                        next if !defined $DISKS[$i][$j];                        print "\t$j\t$DISKS[$i][$j]\n";                }        }}# system disks$DISKS[0][0] = undef;$DISKS[1][0] = undef;my @spares;for(@DISKS[2..$#DISKS]){        push @spares, $_->[0]                or die "Missing disk at $_:0\n";}my @pools;foreach my $i (1..$#{$DISKS[0]}){        foreach my $j (0..$#DISKS)        {                $pools[$i - 1][$j] = $DISKS[$j][$i]                        or die "Missing disk at $j:$i\n";        }}for(@pools){        $_ = "raidz @$_";}if( $opt_destroy ){        cmd("zpool destroy zdata");}if( $opt_create ){        cmd("zpool create -f zdata @pools spare @spares");}sub cmd{        my( $cmd ) = @_;        print "$cmd\n";        system($cmd) if !$dry_run;}

IPv6-Only Ubuntu 10.04 LTS

2011-07-05T15:26:00.003+01:00

To load eth0 on start-up without a configured IPv4 address /etc/network/interfaces:


auto eth0
iface eth0 inet manual
up ifconfig eth0 up
# iface eth0 inet dhcp

For the Google IPv6 public DNS servers add to /etc/resolv.conf:


domain localdomain
search google.com
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

libxml2 and libxslt supported XPath functions

2011-02-11T15:49:00.004Z

The following functions are supported by the libxml2 library for use in XPath statements and hence supported in libxslt for use in transforms. For more information see xpath.c in the libxml2 source.

Note: string length is 'string-length' and not just 'length'!

last()
position()
count()
id()
local-name()
namespace-uri()
string()
string-length()
concat()
contains()
starts-with()
substring()
substring-before()
substring-after()
normalize-space()
translate()
not()
true()
false()
lang()
number()
sum()
floor()
ceiling()
round()
boolean()

CMIS vs Google Documents API vs SWORD

2011-01-19T13:47:00.007Z

The Atom protocol is a very simple mechanism for publishing news feeds - that is, date-ordered small bits of information. An Atom feed is a collection of Atom entries. Each entry contains some basic metadata (title, id) and may have links to other resources. Links of particular interest are 'edit' and 'edit-media' which, respectively, refer to the entry's metadata and media file.

    <?xml version="1.0"?>
    <entry xmlns="http://www.w3.org/2005/Atom">
      <title>The Beach</title>
      <id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
      <updated>2005-10-07T17:17:08Z</updated>
      <author><name>Daffy</name></author>
      <summary type="text" />
      <content type="image/png"
         src="http://media.example.org/the_beach.png"/>
      <link rel="edit-media"
         href="http://media.example.org/edit/the_beach.png" />
      <link rel="edit"
         href="http://example.org/media/edit/the_beach.atom" />
    </entry>

The Atom Publishing protocol (or AtomPub) provides a protocol to add new entries (i.e. to publish to feeds). AtomPub uses the HTTP POST, PUT and DELETE methods to, respectively, create, update or delete entries.

To create an entry the client POSTs an Atom entry to the feed's URL. To update an entry the client PUTs an Atom entry to the entry's URL (replacing anything already there). And lastly, DELETEing an Atom entry URL destroys that entry. The protocol itself is quite readable so I suggest going there if you're lost!

AtomPub is sufficient if you just want to post small entries in XML but often the client wants to e.g. publish a photo, which the new Atom entry will then refer to. There are several approaches to this but the simplest is to use the Atom Multipart Media Resource Creation mechanism, which bundles together the Atom entry and the media file into a single POST.

Atom/AtomPub provides us with a fairly simple tool to publish items onto a Web site. As Institutional Repository (IR) developers we, unfortunately, require a more complex model than just a feed of entries containing one file each. We have more complex metadata and multiple files making up an object. An editorial workflow means items uploaded by users must first be checked by editors before they can be published. There are various other aspects to consider that I won't go into here.

So we like the simplicity of Atom/AtomPub but it doesn't fulfil all of our requirements. Fortunately it is easy to extend AtomPub by injecting additional links and metadata into entries. These links can connect to other URLs that allow complex manipulations to be made on the underlying data structure (hence also to create more complex data structures). OASIS CMIS, SWORD and the Google Documents API are all extensions of AtomPub better known as "AtomPub Profiles". (I'm sure there are others but these are the obvious candidates for IR use.)

OASIS Content Management Interoperability Services (CMIS) is over 200 pages long but, in part, describes an AtomPub profile. I concur with the sentiment here that being asked to implement CMIS won't make your developers happy. The model underpinning CMIS has a hierarchical folder structure. By supplying a special tag in a POST to a feed an Atom entry is created that points to another Atom feed (or 'folder'). In this way Atom entries are effectively typed to be either a 'document' or a 'folder'. Atom entries can be moved to other folders by POSTing them that folder's feed. There is lots more that CMIS adds in, to the extent that I forget what's at the beginning before I get to the end!

root feed
 |
 |-- document entry
 |
 |-- document entry
 |
 |-- folder entry
      |
      |-- folder feed
           |
           |-- document entry
           |
           |-- folder entry
...

The Google Documents API is in a different sphere to CMIS and SWORD. It is specific to Google's API so would need tweaking to be used elsewhere. Similarly to CMIS special syntax passed during a POST to a feed can create a folder-type Atom entry. This entry then points to a new feed which can in turn contain a mix of folder-entries or normal entries. Google support a number of parameters to modify the default behaviour of a URL, for instance downloading a document in a different format.

SWORD is an AtomPub profile developed to support deposit in IRs. SWORD v1 adds several HTTP headers to support more complex publishing behaviour. Often in the repository world one user will be depositing on behalf of another (doctoral student depositing her supervisor's old papers ...). To support this SWORD added the X-On-Behalf-Of header, which supplies the username of the user to deposit as - assuming the current user has permission to do that. Another part of SWORD v1 was to support more complex objects (i.e. multiple files) by defining 'packages'. Packages are collections of files and metadata and are archived together then published, with the server unpacking them to create the complex object. SWORD v2 (at time of writing) will look similar to the previous version but will define means for clients to interact with the packages after upload. OAI-ORE is used to describe the unpacked complex object while content-negotiation will be used to allow clients to retrieve the complex object in agreed format.

repository feed
 |
 |-- document entry
 |
 |-- document entry
 |    |
 |    |-- OAI-ORE/RDF
 |
 |-- document entry
...

All three AtomPub Profiles probably work with a client speaking just AtomPub. The question that is left is which extension of AtomPub is best adopted to achieve our goals. I don't think any of these protocols are entirely satisfactory: SWORD feels like it is working around AtomPub rather than building on it (publishing .zip files?). It isn't clear what IPR Google's profile has nor whether they will take it in a different (incompatible) direction to what we need. CMIS, given it's industrial backing, will likely be essential in the corporate environment but is daunting in its complexity (and that normally means difficult to get right in practise).

Regrettably my influence over any of these profiles is small - as developers we tend to be pushed more by political requirements ("your must support X") than technical merit. I just hope that, given the narrow range these profiles exist in, that they adopt the best bits of each other! (NB I would be interested to hear of any other potential AtomPub profiles)

Ubuntu Maverick 10.0 on Acer Revo 3700

2010-12-03T09:43:00.003Z

These are some rough notes on what I needed to get Ubuntu Maverick 32bit working on the Acer r3700.

wireless

While the kernel rt2860pci driver will run the wireless ok it will cause a hard-lock when it is unloaded (e.g. during shutdown). Thanks to Wolfgang Kufner and Marcus Tisoft for providing a solution - replace the kernel driver with a patched driver from Ralink for the rt3090 (comment #9): https://bugs.launchpad.net/ubuntu/+source/linux/+bug/662288.

sound over hdmi (stereo only tested)

Unmute all digital outputs in Alsa (use right cursor + 'M' until they are all green):

alsamixer -c 1
sudo alsactl store

Testing alsa:

speaker-test -D plughw:1,7

Setting pulseaudio to output via alsa:

sudo gedit /etc/pulse/default.pa

Uncomment and modify the line containing module-alsa-sink:

  load-module module-alsa-sink device=hw:1,7

suspend

Fails to suspend ... haven't found a solution yet.

Enabling support for HTTPS in wkhtmltopdf

2010-06-09T12:26:00.002+01:00

If you get the following error when attempting to web thumbshot an HTTPS-based site with wkhtmltopdf in Fedora Core 13:

QSslSocket: cannot call unresolved function SSLv3_client_method
QSslSocket: cannot call unresolved function SSL_CTX_new
QSslSocket: cannot call unresolved function SSL_library_init
QSslSocket: cannot call unresolved function ERR_get_error
QSslSocket: cannot call unresolved function ERR_error_string

You need to add some additional library links. As root do:

cd /usr/lib64
ln -s libssl.so.10 libssl.so
cd /lib64
ln -s libcrypto.so.1.0.0 libcrypto.so

Hopefully that will fix the problem!

NB I resolved this by using strace to find out where wkhtmltopdf was attempting to find libcrypto and libssl (the problem being the static wkhtmltopdf build was looking for different versions than are installed on FC13):

strace wkhtmltopdf https://mail.google.com/ gmail.pdf 2>&1 | less

Ubuntu 10.04 vnc-based login server

2010-05-20T11:14:00.008+01:00

This recipe is for setting up a VNC login server. This allows you to use a VNC client to access a full GUI on a remote server. If instead you want to get VNC access to your desktop (or share with other users) you need to enable remote desktop.

VNC connections are not encrypted so if you connect directly to the VNC server any login details will be sent in the clear.

Install the required packages:

sudo apt-get install vnc4server xinetd gdm

Restrict GDM to only listening to localhost by adding the following to /etc/hosts.allow:

gdm: ip6-localhost

Enable XDMCP in GDM by setting up /etc/gdm/custom.conf as:

# GDM configuration storage

[daemon]

[security]

[xdmcp]
Enable=true
HonorIndirect=false
# following line fixes a problem with login/logout
DisplaysPerHost=2

[greeter]

[chooser]

[debug]

Create a new xinetd service /etc/xinetd.d/Xvnc (adjust geometry to get different screen sizes):

service Xvnc
{
        type = UNLISTED
        disable = no
        socket_type = stream
        protocol = tcp
        wait = no
        user = nobody
        server = /usr/bin/Xvnc
        server_args = -inetd -query ip6-localhost -geometry 1280x800 -depth 16 -cc 3 -once -SecurityTypes=none
        port = 5901
}

Restart gdm (which will close any current logins!) and xinetd:

sudo service gdm restart
sudo /etc/init.d/xinetd restart

You can then connect to the VNC server using:

vncviewer localhost:5901

slapd configuration hints

2010-03-29T12:40:00.003+01:00

I run an OpenLDAP instance to provide common login credentials for Subversion and Trac.

OpenLDAP is one of those tools that can be nightmarish to get working. LDAP may be 'lightweight' but when put in a stack of browser-web server-LDAP it can be very tricky to work out what's going wrong and where. I spent a weekend fruitlessly trying to debug Apache auth when what I needed to do was stop/start rather than restart (stupid I know).

In OpenLDAP 2.4 the configuration is now moved from a standard "slapd.conf" to LDIF configuration files (in Ubuntu under /etc/ldap/slapd.d/). These can be changed directly or updated via a ldapmodify/ldapadd.

You can add configuration files via the ldapi interface:

sudo ldapmodify  -c -Y EXTERNAL -H ldapi:/// -f FILENAME.LDIF

Beware that if you add a configuration that breaks slapd it will shut down right after you add the configuration. So it's a good idea to backup /etc/ldap/slapd.d/ before making any untested changes (restore/start slapd).

A quick example for querying the LDAP server (note: -ZZ requires TLS, omit this for unsecured connections):

ldapsearch -ZZ -b dc=eprints,dc=org -h localhost -v -D cn=USERNAME,ou=people,dc=eprints,dc=org -w PASSWORD

To start slapd in debugging/console mode do (see the OpenLDAP documentation for values for the -d argument):

slapd -h 'ldap:/// ldaps:/// ldapi:///' -F /etc/ldap/slapd.d/ -d 16383

If you start slapd as root it will write its configuration files as root, so be careful to restore permissions on /etc/ldap/slapd.d/ back to openldap:openldap. If slapd starts as root but not from init.d it may be due to permissions problems (e.g. can't read a certificate file).

I found it difficult to find a working example of creating a slapd database. Here's what I used to create an HDB-based database (paths are Ubuntu):

# Load modules for database type
dn: cn=module{0},cn=config
objectClass: olcModuleList
cn: module{0}
olcModulePath: /usr/lib/ldap
olcModuleLoad: {0}back_hdb

# Create directory database
dn: olcDatabase={1}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {1}hdb
olcDbDirectory: /var/lib/ldap
olcSuffix: dc=eprints,dc=org
olcRootDN: cn=admin,dc=eprints,dc=org
## FIXME below!
olcRootPW: xxx
olcLastMod: TRUE
olcDbCheckpoint: 512 30
olcDbConfig: {0}set_cachesize 0 4194304 0
olcDbConfig: {1}set_lk_max_objects 2048
olcDbConfig: {2}set_lk_max_locks 2048
olcDbConfig: {3}set_lk_max_lockers 2048
olcDbIndex: uid pres,eq
olcDbIndex: cn,sn,mail pres,eq,approx,sub
olcDbIndex: objectClass eq

Translating permissions from slapd.conf to LDIF is relatively easy. If, like me, you need a few tries at getting this write use this as a template (ldapmodify as above will replace *all* existing olcAccess lines):

# Set up permissions on dc=eprints,dc=org
dn: olcDatabase={1}hdb,cn=config
changetype: modify
replace: olcAccess
olcAccess: {0}to dn.base=""
    by * read
olcAccess: {1}to dn.base="cn=Subschema"
    by * read
olcAccess: {2}to dn.subtree="ou=people,dc=eprints,dc=org"
    by group/groupOfUniqueNames/uniqueMember="cn=superusers,ou=groups,dc=eprints,dc=org" write
    by self write
    by users auth
    by anonymous auth
olcAccess: {3}to *
    by * none

Your permissions will likely need to be very different to those given here (which I've abbreviated anyway). Because I'm using apache mod_authz_ldap I have to provide a two-stage authentication, which requires having a 'superuser' account that can search for the relevant cn entry before performing the user authentication.

I hope this saves somebody a headache!

Check your house boundaries

2010-03-19T10:31:00.002Z

I'm currently trying to purchase a house. We have an agreed price, the vendor has vacated the property but we're got stuck on legal issues going on 4 months.

The issue is the previous owner to the current one expanded the garden into unused land, expanding the garden nearly twofold in size. She then sold the property on with land which wasn't on the title deed, leaving the current vendor trying to sell something she doesn't have title to. A conservatory extension has also been added without a permission required by a deed convenant. These issues only came to light after we agreed an offer and after I had spent money on a survey and the solicitor's initial searches.

The Home Information Pack (HIP) does include a copy of the land registry entry. It was a simple process to marry-up the plot (as indicated on the land registry) with a satellite image of the plot. Having done this it is obvious that the plot does not correspond to the land registry, ringing alarm bells. With the benefit of hind-sight I could've avoided the financial exposure of surveys and solicitor costs before having the vendor sort these problems out.

In a more general sense, it's unclear who if anyone checks that the boundaries for a property are the same as those shown on the land registry. My full structural survey didn't include a boundary survey and the solicitor doesn't visit the site (and are seemingly technology illiterate).

As it seems with all house-buying processes the best person to actually check these things is you. When viewing a property's surrounding land you should definitely ask the estate agent whether the sale includes the entire plot and whether they have checked that against the land registry (which they should get as part of the HIP).

Generating Flash Video (FLV) using FFMpeg

2009-09-07T10:27:00.002+01:00

I encountered a problem generating Flash Video (.flv) format files using the latest versions of ffmpeg and libmp3lame. The generated files didn't contain a duration so wouldn't play nicely in the FLV Player embedded player (the player would buffer and play the 5 seconds buffer then stop).

After some investigation I found the problem was due to a buffer error with the libmp3lame library that, while it didn't stop a file being generated, prevented ffmpeg embedding the duration in the flv output. Here's a bug entry for Ubuntu but the OP erronously claims the encoding succeeds - it does to a degree, but any post-encoding operations by ffmpeg won't happen (setting the duration for FLV happens after the encoding).

After much fruitless time trying to fix the broken lame library I used the AAC codec instead:

ffmpeg -i FOO.MPG -acodec aac FOO.FLV

Installing Microsoft Access 97 with Office 2007

2009-08-20T10:04:00.002+01:00

Installing Access '97 onto a machine with a later version of Office already installed is troublesome. Before installing '97 follow these steps:

Install the Jet 3.5 update (fixes problem with creating system database file):

Jet 3.5 - http://support.microsoft.com/kb/172733

Follow the instructions here in the "Access 2000 Is Already Installed" section:

http://support.microsoft.com/kb/241141

If you don't do the "hatten.ttf" trick you will get this error: "Microsoft Access can't start because there is no license for it on this machine".

Now install Access '97 as normal (you may want to change it's installation directory).

Building xapian-bindings-php under Fedora Core 10

2009-06-23T15:50:00.005+01:00

Fedora Core includes packages for Xapian, an Open Source Search Engine Library. Due to a licensing problem Fedora Core doesn't include the PHP bindings for Xapian. Regardless of licensing spats I need to use Xapian from PHP so worked out a patch for the spec file (below) and the steps needed to build the missing xapian-bindings-php RPM:


# rpm building dependencies
sudo yum -y install rpmbuild rpmdevtools autoconf automake libtool zlib-devel gcc-g++
# set up the RPM build tree (and macro file ~/.rpmmacros)
rpmdev-setuptree
# dependencies for building xapian-bindings
sudo yum -y install xapian-core-devel python python-devel ruby ruby-devel php php-devel
# download source RPM
yumdownloader --source xapian-bindings-python
# install source file
rpm -ivh xapian-bindings-*
# copy the patch below to rpmbuild/SPECS/xapian-bindings.spec.diff
# patch the spec file
patch --dry-run -p0 -d rpmbuild/SPECS/ -i xapian-bindings.spec.diff && \
patch -p0 -d rpmbuild/SPECS/ -i xapian-bindings.spec.diff
# build all packages from the installed RPM spec file
rpmbuild -ba rpmbuild/SPECS/xapian-bindings.spec
# RPMs for your platform will be output to rpmbuild/RPMS/

I needed to do this to build a xapian-bindings-php RPM, which required patching of the spec file:


--- xapian-bindings.spec.orig   2008-12-08 13:20:55.000000000 +0000
+++ xapian-bindings.spec    2009-06-23 15:45:05.000000000 +0100
@@ -12,6 +12,7 @@
 BuildRequires: python-devel >= 2.2
 BuildRequires: autoconf automake libtool
 BuildRequires: ruby-devel ruby
+BuildRequires: php-devel php
 BuildRequires: xapian-core-devel == %{version}
 Source0: http://www.oligarchy.co.uk/xapian/%{version}/%{name}-%{version}.tar.gz
 Buildroot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
@@ -42,11 +43,21 @@
 indexing and search facilities to applications. This package provides the
 files needed for developing Ruby scripts which use Xapian

+%package php
+Group: Developer/Libraries
+Summary: Files needed for developing PHP scripts which use Xapian
+
+%description php
+Xapian is an Open Source Probabilistic Information Retrieval framework. It
+offers a highly adaptable toolkit that allows developers to easily add advanced
+indexing and search facilities to applications. This package provides the
+files needed for developing Ruby scripts which use Xapian
+
 %prep
 %setup -q -n %{name}-%{version}

 %build
-%configure --with-python --with-ruby
+%configure --with-python --with-ruby --with-php
 make %{?_smp_mflags}

 %install
@@ -59,6 +70,9 @@
 mv %{buildroot}%{buildroot}/usr/share/doc/%{name}/python %{buildroot}/usr/share/doc/%{name}-python-%{version}
 mv %{buildroot}%{buildroot}/usr/share/doc/%{name}/ruby %{buildroot}/usr/share/doc/%{name}-ruby-%{version}
 rm -rf %{buildroot}%{buildroot}/usr/share/doc/%{name}/
+# Fix location of PHP 5 include file
+mkdir -p %{buildroot}/usr/share/php/
+mv %{buildroot}%{buildroot}/usr/share/php5/xapian.php %{buildroot}/usr/share/php/xapian.php

 %clean
 [ "%{buildroot}" != "/" ] && rm -rf %{buildroot}
@@ -74,6 +88,11 @@
 %{ruby_sitearch}/_xapian.so
 %{ruby_sitelib}/xapian.rb

+%files php
+%defattr(-, root, root)
+%doc AUTHORS ChangeLog COPYING NEWS README
+/usr/lib64/php/modules/xapian.so
+/usr/share/php/xapian.php

 %changelog
 * Mon Dec 08 2008 Adel Gadllah  1.0.9-1

PPTP VPN under Ubuntu Intrepid 8.10

2009-03-10T10:05:00.006Z

It seems there are some issues with PPTP VPN under Ubuntu's latest Intrepid (as-of March 2009). When I tried to connect I would get an error in /var/log/syslog of "EAP: peer reports authentication failure" (amongst the general PPTP messages). Depending on your PPTP server you may have the same problem or entirely unrelated issues but I suspect mine is a very common set-up.

The Ubuntu bug here - https://bugs.launchpad.net/ubuntu/intrepid/+source/network-manager-pptp/+bug/259168 - contained the answer, which is to both enable MPPE and to disable EAP authentication.

Here is what I did:

1. Add a new VPN connection
2. Only add the username (do not set the password in the connection settings):

3. Under "Advanced" check "Use Point-to-Point Encryption (MPPE)":

4. Next, run gconf-editor and find your VPN connection and add a new key of "refuse-eap"/String/"yes":

5. Connect using the VPN and give it your password.

If you ever change settings using the Network Manager applet you will lose the "refuse-eap" setting and will need to re-add it.

Automounting a CIFS share in Ubuntu

2008-09-09T09:49:00.004+01:00

autofs is a tool that mounts and unmounts devices on demand. To use autofs to mount CIFS shares (Windows/Samba) in Ubuntu do the following:

$ sudo apt-get install autofs smbfs smbclient

Then edit /etc/auto.master and uncomment the smb line:

#
# $Id: auto.master,v 1.4 2005/01/04 14:36:54 raven Exp $
#
# Sample auto.master file
# This is an automounter map and it has the following format
# key [ -mount-options-separated-by-comma ] location
# For details of the format look at autofs(5).
#/misc  /etc/auto.misc --timeout=60
/smb    /etc/auto.smb
#/misc  /etc/auto.misc
#/net   /etc/auto.net

Restart autofs:

$ sudo /etc/init.d/autofs restart

You can now view and mount CIFS shares by changing to the directory /smb/HOSTNAME/SHARENAME e.g. to mount and change directory to "photos" on "bart":

$ cd /smb/bart/photos

You can even list the shares on "bart" by doing:

$ ls /smb/bart/

What the default scripts don't provide is the ability to have per-connection mount options. In particular I wanted to mount some hosts read only and using UTF-8. I modified the /etc/auto.smb script as follows:

...

# This file must be executable to work! chmod 755!

key="$1"
mountopts="-fstype=cifs"
smbopts=""
credfile="/etc/auto.smb.$key"
optsfile="/etc/auto.smb.$key.opts"

if [ -e $optsfile ]; then
        . $optsfile
fi

...

I then created a connection specific file /etc/auto.smb.bart.opts containing:

mountopts="$mountopts,ro,iocharset=utf8"

Now any mounts on "bart" will be set read only and use the UTF-8 character set.

DG834/DG834G to OpenSwan VPN

2008-09-04T08:47:00.005+01:00

The Netgear DG834 ADSL routers support IPSEC based Virtual Private Networks (VPN). The DG834 uses the Open Source Openswan software (http://www.openswan.org/). This blog provides the configuration details for how I connected an intranet LAN behind a DG834 to a Ubuntu-based Linux server in another LAN via the Internet.

Here's my setup:

NAS - [192.168.0.1] DG834G [starsky] - {inet}

{inet} - [hutch] ADSL ROUTER [192.168.1.1] - Backup

Where:

The NAS is a Western Digital network drive.
The Backup is a normal PC running Ubuntu with a RAID 5 SATA storage.
Both ADSL connections have static IPs.
starsky's public IP is 10.1.1.1
hutch's public IP is 10.2.2.2
Backup is set as the DMZ server in ADSL ROUTER.

The goal was to copy data from the NAS to the Backup and to provide Windows share access to the Backup from the 192.168.0.0/24 subnet.

The DG834G is running firmware version V2.10.22. I used the VPN Wizard to set up the VPN endpoint initially and then revised it in the VPN policy editor. The Policy Name can be anything (e.g. "Bob"). As I want this connection to be up all the time I set the "IKE Keep Alive" and set the connection to "Initiator and Responder". The Address Data is the host name of ADSL ROUTER (hutch). The Pre-shared Key should be something difficult to guess.

On Backup I installed Openswan and followed the defaults during installation:

$ sudo apt-get install openswan

I added to /etc/ipsec.conf:


conn Bob
        type=tunnel
        leftid=10.2.2.2 # hutch
        left=%defaultroute
        leftsubnet=192.168.1.0/24
        right=10.1.1.1 # starsky
        rightsubnet=192.168.0.0/24
        keyexchange=ike
        auto=start
        auth=esp
        authby=secret
        pfs=no
        rekey=no
        ike=3des-sha1-modp1024
        esp=3des-sha1

And to /etc/ipsec.secrets (where pre-shared key is the same as you entered into the DG834):

# secrets for "Bob"
hutch starsky
      10.2.2.2 10.1.1.1: PSK "pre-shared key"

I then restarted Openswan:

sudo /etc/init.d/ipsec restart

Once the VPN has established (look in /var/log/auth.log) I can access Backup by browsing to it's IP address: //192.168.1.100/SHARENAME.

I'm not sure with the above configuration you can access the 192.168.1 subnet from the 192.168.0 subnet. Answers on a postcard ...

ADSL ROUTER is actually a DG834Gv4, but the firmware seems to be buggy. In the default firmware it won't allow Windows shares to be accessed across it and in the latest firmware (V5.01.09) the VPN connection is unreliable and won't re-establish unless the router is rebooted.

The above "works for me", but I'm by no means an IPSEC or Openswan expert so welcome any feedback/corrections.

Setting the output page size with ps2pdf

2007-09-07T16:04:00.000+01:00

Ghostscript's ps2pdf utility converts Postscript format files into Adobe PDF. One drawback with this utility is that, by default, it always outputs in Ghostscript's default page size, regardless of the postscript document's page layout. This is probably sensible if the document is destined for printing, but at the moment PDF is also the best cross-platform way to express large spaces of graphics (in this case directed-tree graphs).

I've managed to get around this - mostly - as follows:

Use ImageMagick's identify to get the Postscript document dimensions:
$ identify test.ps test.ps PS 609x1528 609x1528+0+0 PseudoClass 256c 909kb 0.020u 0:01
Execute ps2pdf with the -dDEVICEWIDTHPOINTS and -dDEVICEHEIGHTPOINTS arguments
$ ps2pdf -dDEVICEWIDTHPOINTS=679 -dDEVICEHEIGHTPOINTS=1598 test.ps test.pdf

What I haven't been able to work out is how to suppress the margins in the resulting PDF. To avoid the Postscript document being cropped I added 70 points to the width and height.

Automatically creating home directories on login

2007-05-10T11:24:00.000+01:00

PAM (pluggable authentication modules) allows you to automatically create home directories when a user first logs into a system. This means they will get the skeleton shell init scripts (so e.g. they get a decent prompt).

Add to /etc/pam.d/system-auth (below the pam_deny.so line):

session     required      /lib/security/$ISA/pam_mkhomedir.so skel=/etc/skel umask=0022

That will automatically create a home directory in /home the first time any new user logs into the system.

Recovering a corrupted OpenLDAP database

2007-04-02T10:41:00.000+01:00

Warning! I do not give any warranty that this will work, it may make things worse. Always backup. A lot.

You will require the 'db_recover' utility for db4 (lives in the db-utils RPM).

Backup your LDAP database (in /var/lib/ldap).

Stop the LDAP server if it is running:

/etc/init.d/ldap stop

Change to the OpenLDAP directory:

cd /var/lib/ldap

Execute db_recover:

db_recover -v

Reset the file permissions:

chown ldap.ldap *
chmod 600 *

Restart the server:

/etc/init.d/slap start

ROAR Google Earth Tours

2007-03-17T23:51:00.000Z

I've added KML support to ROAR, which enables repository locations to be overlayed on Google Earth. Not all ROAR repositories are Geo-located and many are very roughly located.

To add the ROAR overlay (in Google Earth) click the Add menu, Network Link and enter the following URL:

http://roar.eprints.org/?action=kml&type=institutional&order=recordcount&kml=marker

Or, to get coloured boxes:

http://roar.eprints.org/?action=kml&type=institutional&order=recordcount&kml=height

All repositories (not just institutional):

http://roar.eprints.org/?action=kml&order=recordcount&kml=height

etc. Have fun!

Hmmm Microsoft Windows Vista

2007-03-11T18:36:00.000Z

I've tried out Vista on my 'leisure' machine but I'm going to have to revert to XP. Which is a shame, as I've found the Flip3D interface a lot nicer than XP's (accelerated windows are great).

My gripes are (in no particular order) ...

Hardware support is patchy: no support for my HP 2600N printer, no support for nForce 4 onboard sound ... not exactly rare or elderly bits of equipment.
Vista is a memory hog: why does it need to gobble 500MB of memory before you do anything? 2GB is now the required amount for a Vista-based gaming platform.
User Account Control spammage: under Unix if you give SUDO your password it'll remember for a period of time. In Vista you want to look in the logs, access Control Panel, access a sub-part of Control Panel, look at what's currently running ... that's a whole lotta 'Continue' clicking. Unless you disable it for a while (but you have to go digging for that).
Save-as no longer defaults to Documents (formerly known as 'My Documents), but to the user's unix-like home directory.
Connecting to VPN is a clickfest - 'Connect to' can't be a menu like XP. Control Panel retains that capability and it can be a menu if you revert to 'classic' start menu.
Medieval Total War doesn't work: ok, minor gripe, but it was the second game I tried running and it didn't work.
Creative X-Fi loses digital out: have to dig into the audio settings to disable/re-enable it to get it to work (audio devices have separate 'Configure' and 'Properties' options - surely you configure the properties of the device?).
Random freezing: I suspect some drivers aren't stable yet.
Changing the hibernate button to shut down is an 8-step process (buried deep inside the Control Panel power settings, nowhere near the button itself). Not exactly green to default to 'sleep'.
Searching the Start Menu: in the beginning there was the command-line, it had auto-complete (unless you were Microsoft). Then there were clickable icons in Windows (3.x). Then there was the Start Menu. Then there was searching the start menu, with auto-complete. Only 20 years to end up where we started. Perhaps it's a different paradigm, but so far it's mostly just annoying.

Vista has much improved auditing facilities (performance manager, logging etc.) for those expert enough to understand them or, at least, to glean the clues given. I disabled Shadow Copy (or at least I think I did) to stop the constant post-install disk thrash.

In conclusion if you don't play games (or at least only play the newest games), have the very latest hardware (<1 year), have oodles of memory (1GB+), want fancy visual effects and have £150 to burn then Vista is for you (NB I accessed Vista through site license).

The case of the vanishing interoperability

2007-03-07T12:11:00.000Z

So, you have your nice, shiny new repository. You've put a lot of effort into it to encourage your faculty to use it, perhaps even to learn to love it. You've made sure its indexable by Google, followed the relevant accessibility guidelines and generally made sure it's doing its primary job: making your faculty's work as accessible as possible. You did do that, right?

UQ eSpace is the new incarnation of the University of Queensland's (AU) institutional repository which uses Fez (a Web interface to Fedora). Previously they were using GNU EPrints 2. (At this point I should let you know I am an EPrints developer, but right now I'm talking with my ROAR hat on.) I've picked on UQ eSpace as an example, but I think these criticisms hold for a lot of IRs out there.

What has drawn my attention to UQ eSpace is the disappearance of their OAI interface. In December 2006 it disappeared, resulting in UQ eSpace no longer being trackable in ROAR. I don't keep a proactive eye on these things, so it can take a while for me (or someone else) to notice something's disappeared. Ok, so as the developer of ROAR it's time to go hunting for the OAI interface ...

But first, a little digression. In EPrints 3 we've added a note to EPrint's home page that gives the (human-readable) location of the OAI interface (e.g. see http://demoprints3.eprints.org/). That makes my life (with ROAR hat on) a whole lot easier. NB EPrints was already the easiest of the repository softwares to locate the OAI interface for because it always resided at /perl/oai2, compared to e.g. DSpace with its combinations of 'request' 'dspace' and 'oai' terms in the OAI URL. When you have an index of 800-odd entries, consistency and obvious descriptive data are the difference between spending 10 seconds or 10 minutes editing an entry. Some people might suggest I should email administrators but that assumes an obvious point of contact is provided (again, rarely the case).

... back to the hunting. The first step in the trail of the OAI interface is whether its broken or gone. UQ eSpace returns a 404 at the ROAR-registered location (http://espace.library.uq.edu.au/oai). Not so good, but at least its a genuine 404 page. The interface has obviously gone and has been for a while. Next stop, the repository home page (http://espace.library.uq.edu.au/). All very pretty and familiar. Thumbs-up also for putting a clear description on the home page. But a scan of the home page shows no reference to OAI. How about Help? (I've not come across a site yet that provided the OAI interface in help, but one lives in hope)

Help documentation will be available here soon.

Hmm, not so good. Let's take a look at the software behind UQ eSpace. A search for Fedora and OAI throws up a promising lead - try accessing /fedora/oai. 404. Oh dear. Perhaps the Wiki for Fez might help?

In Fez 1.2 you have the option of configuring the Fedora OAI provider to provide access to your content.

Uh huh. So it exists, maybe, but where? Strangely enough I can find out the OAI location for EPrints from that same Wiki. How about Fez' documentation? 403 Forbidden. Hurumpf.

It's at this point I hang up my deerstalker hat and smoking pipe. Perhaps the UQ eSpace OAI interface doesn't exist yet, but I wouldn't know either way. It's not difficult to help out services that want to promote you, you just need to put some clear guidance on your site: who are you? what do you contain? how do I harvest your content? how do I contact you? In fact, obvious stuff if you think about - it's what scholars having been doing for over a century.

Ordinarily I'd skip past UQ eSpace without comment but they've triggered another of my pet-hates. A well-designed, modern Web page should consist of well-structured content in HTML/XHTML and one or more style sheets that translate that content into the appearance you want. This enables better accessibility (for poor-sighted/disabled users), cross-browser compatility and cross-device compatibility (screens, printers, hand-held devices etc.). There are many accessibility benefits to this: the page is laid out in its logical structure, allowing screen readers to work, users can increase colour contrast and all text can be scaled by the user (vs. text rendered as images). I use a plugin for Firefox called Web Developer that with a single keystroke can disable all styling on a Web page. My definition of a well-designed Web page is that when you disable styling, it falls out into a well structured, logical page. That means all the custom layout/colouring/backgrounds should disappear. Let's try my test on UQ eSpace's home page?

With styling UQ eSpace has a top bar, two columns and a bottom bar. This is a classic layout with high-level, little-used options and branding at the top (links to the institution, user-login functions), content and news in the two columns and technical data at the bottom (but no contact details). Fonts are changed to sans-serif and reduced in size (a little unfriendly to high-res screens). Various title bars have background colouring/images.

Without styling not a lot changes: the fonts revert to serif, links revert to their default colours. The column layout is generated through tables - which accessibility guidelines will tell you should only be used for data (allowing a screen-reader user to skip over large data sets). The top bar links are rendered as a single mapped image: for our poor-sighted user who can't read the (small) text-as-images, or is using a screen reader lets try disabling images. Unfortunately the university links now disappear completely (no ALT-tags). The backgrounds persist (which makes it difficult to get a high-contrast or printer-friendly version).

Why does this matter? Fez is being widely implemented in Australia and is promoted as a competitor to existing repository interfaces. I strongly believe that good Web design does not have to take longer or result in poorer interface design when compared to circa Netscape 5 technology. It does however require an appreciation of and adherance to standards - after all, that's what they're there for. Before I'm accused of throwing bricks in glass houses I do acknowledge that accessibility is tough but I've found modern XHTML/CSS Web design isn't difficult and can go a long way towards a better structured Web (if for no other reason than to get pages that are printer-friendly).

I hope UQ eSpace and the Fez developers will in future address both interoperability (i.e. where the heck is it?) and accessibility. I watch and wait with interest.

Removing Duplicate RPM Packages

2007-02-23T17:39:00.001Z

To find a list of packages that have a newer version installed use:

rpm --last -qa | perl -n -e '/^(\S+)-\S+-\S+/; print "$&\n" if $SEEN{$1}; $SEEN{$1} ||= $_;' | sort | uniq >dupes.txt

The --last argument causes RPM to list packages by install date (newest first). We remove any packages with the same name as a more recently installed package.

To actually remove the packages (as root):

for i in $(cat dupes.txt); do rpm -e $i && echo $i; done

Depending on dependencies you may need to run this more than once.

Changing the Starting Frame of a Fluid Animation in Blender

2007-01-07T21:53:00.000Z

I've become somewhat enamoured with Blender recently (the open source 3D modeling and ray-tracing program). Once you've got a handle on the interface it's a really functional modeler, at least with my limited experience.

Anyway, a particularly 'cool' feature in Blender is support for fluid dynamics. This allows you to define some 'fluid' and fling it around inside a three dimensional space. Something I got stuck on however was how to get the fluid animation to start after the first frame. If you know beforehand what frame you want to start the animation at you can set the start-frame and away you go. If, like me, you just dived in there and didn't want to waste the fluid calculation then you need to get your hands a bit dirty.

The meshes generated from the fluid calculation are stored in a directory with file names like 'project_blah_XXXX.gz' where XXXX is the frame number. So if you want to shift the starting frame you need to rename all these files, changing the numbers to start at your needed frame. Here's a Perl script that'll do this for you (you ought to take a backup before doing this!):

#!/usr/bin/perl
# Copy this to the directory containing your fluid calculation
# files then do: perl rename.pl

die "Usage: $0 \n" unless @ARGV;
my $SFRAME = shift;

opendir(DIR,".");
my @files = sort { $b cmp $a } grep { $_ !~ /^\./ } readdir(DIR);
closedir(DIR);
foreach(@files)
{
next unless /_\d{4}\./;
my $new = $_;
$new =~ s/(\d+)/sprintf("%04d",$1+$SFRAME)/eg;
print "$_\t$new\n";
rename($_,$new);
}

Google CSE added to the Registry of Open Access Repositories

2006-11-16T15:04:00.000Z

A few weeks ago OpenDOAR announced the inclusion of a experimental Google CSE ("customised search engine"). Google released their CSE (or 'co-op') tool on the 26th October, and its a testiment to Google's skill at identifying new niches that CSE has already gained such interest.

Being OpenDOAR's nearest (good natured) competitor its incumbent on me to make sure we don't fall behind in the technical stakes - quality of content is a very subjective issue that I'll leave to others to judge. So, what's involved in setting up a CSE?

Google CSE is the normal Google web index, but constrained to a human-edited list of web sites. The CSE is identified by a unique (Google-generated) identifier that, when passed to Google co-op with a query, returns matching pages from only those sites. As the creator of a CSE you can either point users at the CSE's virtual home page (at Google) or embed it in your own site. The level of integration into your own site's look and feel depends on how long you're prepared to spend implementing javascript wrappers. There are, however, some constraints made by Google, in particular commercial users must allow advertising to be included, and the "Google Custom Search" line must be included either in the search box or near by (which the Google-supplied code snippets do for you).

What Google CSE means for us registries is we can now (theoretically) provide full-content searches of registered repositories with a minimal of effort. This is what OpenDOAR have done, and we've followed suit in our own search interface.

In addition to constraining the search to given sites Google CSE provides 'refinements' - editor-provided key terms that either filter the list of sites, or weight certain sites higher in the search results. Refinements allow the CSE creator to provide sub-customised searches to more finely control the search results, the typical example being to create a CSE for a topic area (e.g. tropical diseases) then to provide refinements for different types of users (e.g. medical practitioners).

To create the Google CSE and refinements for ROAR I created two exports: the TSV and Context files. The TSV file contains the URLs of the sites to be included, labels for each site and the site's weighting. A label is an identifier that can be used to refine search results (e.g. Australian repositories are labelled with 'country_au').

The Context file contains the basic search engine configuration (title, description etc.) and what effect refinements have on the search results. Refinements can change the weighting of sites, alter the query or filter out given sites. In the ROAR CSE I've set all the labels to 'filter' (i.e. only include sites that contain the given label).

So, that's the theory, but in practise Google CSE refinements don't appear to work like that (or work at all). If you try out the ROAR search, first of all you'll notice you get very few matches back and secondly you actually get more matches the more refinements you use. Hopefully this will be resolved in time, in the meanwhile prepare to be confused!

PHP gd and Perl gd segfaulting

2006-09-25T12:27:00.002+01:00

I make a lot of use of GD (through charts) in my services, which also use a mix of Perl and PHP scripts. Unfortunately PHP defaults to using an internal copy of the GD library that, when used, causes a segfault if GD is also used by mod_perl (which will use the system GD library). The segfault is the result of attemping to use two versions of the GD library at once.

The solution to this problem is to use the system GD library in PHP. I use RedHat enterprise on my servers and want to use RPMs to allow ease of keeping up to date with bug fixes. Fortunately it's pretty easy to modify the RPM source to fix this isolated problem, without having to install a compiled PHP.

Download and install the source RPM (NB make sure you install the correct version):

up2date --get-source php
rpm -ivh /var/spool/up2date/php-*.src.rpm

Change to the rpm source build directory:

cd /usr/src/redhat

Create a new file called 'php.spec.patch' containing this:

--- php.spec.orig       2006-09-25 12:00:23.000000000 +0100
+++ php.spec    2006-09-25 12:00:44.000000000 +0100
@@ -433,7 +433,7 @@
        --with-exec-dir=%{_bindir} \
        --with-freetype-dir=%{_prefix} \
        --with-png-dir=%{_prefix} \
-       --with-gd=shared \
+       --with-gd=shared,/usr \
        --enable-gd-native-ttf \
        --without-gdbm \
        --with-gettext \

Patch the php.spec file (-l ignores whitespace):

patch -l SPECS/php.spec <php.spec.patch

Build the binary RPM (will take a while):

rpmbuild -bb SPECS/php.spec

Install the new php-gd rpm:

rpm -Uvh RPMS/php-gd-*.rpm

You may want to add php-gd to the skip-list in /etc/sysconfig/rhn/up2date, otherwise if RedHat update php it'll cause your Web server to seg fault.

Javascript Graphs

2006-09-13T14:21:00.000+01:00

Citebase now has the Correlation Generator again, which was always in need of some fixing.

I've replaced the Java-based charts with Javascript ones, based on Emil A Eklund's Chart Widget. The advantages of the Javascript charts are that they load quicker (not requiring a Java VM) and that it's much easier for me to customize them to my needs.

While the chart widget is compatible with IE and Safari I haven't bothered to put in all the wrapper code for the AJAX and element searching in my glue code - I figure this is such a specialist tool it's not unreasonable to expect users to download Firefox.

While I'm happy enough with the implementation for the Generator to be published, the charts could really do with a mechanism to zoom - most of the interesting stuff in citation and download data happens very quickly, hence can be eclipsed by the 'long-tail'.

Regardless of the current drawbacks, I think it looks and works pretty well.

NFS Server Gone Away

2006-08-19T11:06:00.000+01:00

So your servers are happily serving away when somebody knocks over your (inaccessible) NFS server. What happens next seems to be dependent on the mount settings used, but the bottom line is any processes that attempt to open the mounted NFS system will hang in 'D' state (most likely nfs_wai(t)). As far as I can tell 'D' means the process is locked in a kernel call, which means the process itself can't be touched. And you certainly can't unmount (umount -f) the file system because 1) there are files open by those unkillable processes and 2) in order to unmount an NFS mount the NFS server needs to be contacted to let you know there's no file locks open (except it isn't there!).

How do you recover an NFS client that has been overrun with 'D' state processes? The answer is remarkably quick and simple, if you know how. You can't kill 'D' state processes and you can't restart (because the system will hang while trying to kill them). The reason it hangs is because NFS is trying to contact the NFS server, not getting a response, timing out and trying again. But, bizarrely, if you give the NFS server's IP address to another machine, NFS will connect, find the share is no longer there and give up. So:

ifconfig eth0 add <NFS server IP address>

Which should cause all your nfs_wait processes to reset, so you can kill any dead processes and unmount the file system. Removing the IP address requires adding the sub-interface name (because you just added two IP addresses to one interface):

ifconfig eth0:0 del <NFS server IP address>

You probably need NFS running on the temporary machine too ...

Managing LDAP Administration

2006-07-19T12:55:00.000+01:00

Managing LDAP seems like a tool that creates a lot of problems while solving others. One major headache I have with LDAP is making trivial changes to database entries, for example if I've had a bug that hasn't created an entry correctly (me, bugs, never!). The best tool I've found for making tweaks to an LDAP installation is ldapadmin by Tihomir Karlovic:

http://ldapadmin.sourceforge.net/

Much love to tools that work and do what you want them to do!

Making Word Documents Public

2006-07-11T14:51:00.000+01:00

Having been at the sharp end of trying to retrieve information from various file formats I've come to the conclusion that editing formats are easier to get information out of than distribution formats. Or to put that another way, in 99% of cases, give me the Microsoft Word file rather than the Adobe PDF.

But the trouble with Microsoft Word (being Microsoft) is it has a habit of letting other people use your resources without your wishing them to. In Word this might be exposing sensitive (or embarrassing) comments or providing an edit history through the 'track changes' facility. Fortunately Microsoft heeded the numerous occasions where an organisation has been caught out and produced the Remove Hidden Data tool (download RHD here).

After you install the RHD tool, open your Word file then click "Remove Hidden Data" to save the document to a new, clean(er) file. Of course, if you're handling very sensitive stuff then you probably want to print-to-PDF and accept your resulting file will be less functional.

Managing Statistics

2006-07-10T16:45:00.000+01:00

Citebase provides a number of different analyses of citation and usage (hits/downloads) data. Part of the problem with this is how to provide a reasonable way to access disparate and relatively complex information. In re-designing Citebases interface I've introduced two new features that will hopefully make using its statistical data easier.

As you can see in the image these two new features are tabbed-navigation of the various citation-based navigations and usage breakdowns (using javascript + iframes). The highlighted (in red) is a "sparkline" that gives a very compact indication of the citation history. I'm not sure if this is the creator of sparklines, but I've borrowed the concept from Edward Tufte - see Ask E.T.: Sparklines. While my implementation is definitely 'cool' it would be nice to use the final point technique (the little red dot on Tufte's examples) as well, but I've not thought of how to utilise that for citations (where the neighbouring number is the total citations, which would necessitate having a cumulative graph (99% of which would be a straight line on such a small scale). Perhaps when Citebase goes live someone will come up with an inspired suggestion.

It's surprising just how much better Citebase looks when the spacing is fixed (subscript and superscript as used in formulas ordinarily cause the HTML line spacing to vary). Along with improvements to the abbreviating code and a general increase in images (to make more curved and shadowed space) I think it looks pretty spiffy.

Now I have the user registration code and statistical graphing stuff left to do.

Open access and science

2006-06-20T18:15:00.000+01:00

I've been asked to contribute to a 'focus' section on open access:

More precisely, do you think that open-access publication speeds up scientific dialog between researchers and, consequently, should be extended to the whole scientific literature as quickly as possible?
Do you really think that Open-Access Articles Have a Greater Research Impact?
If yes, what are, according to you the main consequences on communication between scientists?

Nico Pitrelli (Deputy editor of JCOM - Journal of Science Communication)

A number of potential and identified benefits have been associated with open access (providing free access to users of research literature). Of foremost interest is whether open access papers receive more citations (and downloads) than papers that are only available through a subscription or similar payment. Eysenbach's work is the latest study to confirm the general finding that open access does increase citation impact. (Obviously, once all research is open access, there can be no citation advantage attributable to the free/non-free comparison.)

Eysenbach has attempted to measure only the free/non-free variable by comparing articles in a journal with an open access option and has argued that other studies mix up a number of potentially conflated variables (number of authors, chronology issues etc.). Regardless of the finer points I argue that so far the evidence points towards open access articles receiving more citations and, given there is very little cost in providing open access (by author self-archiving the pre-print and/or post-print), the potential benefits outweigh the 'risks'. This isn't to say authors will suddenly see citations where they didn't before - these studies compare averages and also seem to indicate those papers that would already be high impact benefit most from open access.

There are other potential benefits that open access could provide but perhaps aren't of immediate interest to authors. One of these is that the duration of the research cycle is reduced: an author writes an article, is read by other authors that then cite that article in their own articles. In physics this is the result of rapid pre-printing - authors write an article and simultaneously post the pre-print to the physics arXiv e-print service as well as submit to a journal (several physics publishers even accept submissions direct from the arXiv). Rapid, free access to pre-prints in physics has dramatically improved the rate of communication in that subject. This isn't to say journals have been side-tracked in physics - far from it - as it appears (in studies performed by Michael Kurtz) that authors cite the pre-print, then switch to reading and citing the journal article once published.

Another potential benefit of open access is that it opens up the market for providing services to researchers. Putting all scholarly research on the web, free to access, will allow it to be indexed by a wide range of services. We have already seen Google move into this area with their Scholar service and there are similar moves afoot by the other big players (Microsoft and Yahoo). There is also considerable interest by funding agencies in using open access to help promote the research they fund (Wellcome Trust in the UK). There are also research tools built by the academic community - Citeseer (for computer science) and my own Citebase (for the arXiv). These tools gain their usefulness from the seamless way a user can move from the service to the full-text, without having to pay access fees. They are also potentially more powerful than existing bibliographic databases, because the full-text is freely accessible hence can be made fully searchable.

The consequences for scientists are twofold. Firstly as authors they will increasingly be expected or even required to provide free access to their research results, either by publishing in open access journals or by author self-archiving their articles in an institutional or subject-based repository. These mandates may come from their institution, funding agency or even government. As users of research material scientists will find it easier to locate and gain access to the full-text of research articles (I already find a Google web search to be the quickest way to locate a cited paper). They will also increasingly see articles being automatically and autonomously measured and evaluated by third-party services. This is both to serve the needs of research agencies (who need to evaluate the impact of the researchers they fund) but also to provide better search and alerting services.

Ultimately open access is about using the power of the web - to provide instant, near-free access to information - to maximise the benefit of the investment in research. To do anything else is a betrayal of the public investment in science.

I see data but do you CDATA?

2006-06-14T17:27:00.000+01:00

For a technology so widely used HTML and family can throw up a lot of roadblocks in the way of progress. I've been working with javascript ('AJAX') with a view to improving the responsiveness of Citebase's web interface. An important requirement is to maintain a 'single-page-interface' but to allow differing analyses and links to be loaded into that page through a menu-like mechanism: ordinarily I'm a fan of keeping-it-simple, which in this case would mean as many different pages as there are functions, but I suspect most users would get lost in the myriad of page reloads.

One javascript library (the J in AJAX) suggested to me by a colleague was 'DOJO' ( http://www.dojotoolkit.org/), which provides a lot of javascript-based widgets that turn your web page into something more akin to a windows GUI. But, and you'll notice this if you go to that page, it breaks the forward/backward browser history navigation. In which case we're somewhere closer to the entire-web-page in a Macromedia flash file than I'm comfortable with. (I dislike flash-based things because it prevents me using the neat tools that Firefox otherwise provides, like forward/backward navigation on my 5-button mouse, keyboard scrolling/searching/link selection etc.)

After several days trying different techniques I've ended up back where technology was several years ago: the iframe. Coupled with a bit of javascript to make interactive menus (so we don't have to go to the server to get more options) iframes keep the forward/backward navigation working and can be automatically stretched by javascript so as to fit the content i.e. no scrollbars floating around the middle of a page. Now the question is how I provide hooks in the page for non-javascript, non-iframe users - least I forget how web pages look to web spiders.

Where does CDATA come into all this? It turns out if you XHTML DOCTYPE a libxml2 DOM then try to add a 'script' element libxml2 decides the text must be a CDATA section, but doesn't render it in a fashion compatible with browser's javascript parsing. Reading around it seems this is because 'script' is defined as containing CDATA (or PCDATA) in the XHTML specs, which strikes me as an unfortunate collision of standards and how things actually end up working (or not, in this case). I can understand why this has come about (mixing of content types within a single document), but 'little' things like this can be really annoying e.g. using DOJO requires setting up a configuration variable, loading its js file, then loading up particular widgets. Putting that into external files would require another two files, with one line in each.

I've pretty much come to the conclusion AJAX is a really nice technology for some specific areas. AJAX is good for complex user interaction (e.g. form-filling) but bad for simple interaction (e.g. navigation, scrolling and reading). Firefox makes a good job of AJAX (and I suspect Opera is better still), but as long as 80% (?) of users insist on using Microsoft's crippleware we're stuck in the slow lane. Then again, perhaps Internet Explorer 7.0 will solve all AJAX's problems ...

What the ping?

2006-06-11T13:15:00.000+01:00

I've been trying to get multicast working on my home ADSL router (a Linux-based PC using a Speedtouch 'frog' USB modem). This is because the BBC and several UK ADSL providers are running multicast trials, including my own ISP Plus.net. While Plus.net seem to have multicast enabled (I see the IGMP messages on the ppp device), getting Linux to actually do something multicast routing wise seems to be one of those "if you're in the club" topics: the preserve of those who find command-line interfaces exciting.

Out-of-date documentation annoyances - if you've read the incomplete multicast routing howto it ends (before the "to be continued") with pinging the 'all-hosts' group (224.0.0.1) to see who'll respond, except sometime between when that was written circa 1999 and now, responding to broadcast packets was disabled by default. If you want your Linux-kernel machine to respond do:

echo 0 >/proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

Now, assuming multicast is alive and well on the Linux router and my Windows XP PC the next step is to get some routing going between the LAN and ADSL. I'm currently battling xorp, which seems to get horribly confused by an interface on an IPv6 enabled machine but which doesn't have an IPv6 address. Like, say, like the ADSL PPP connection? (NB I have IPv6 courtesy of a tunnel to the BT broker service - not that there's much out there in IPv6 land. I'm not even going to go near the kernel patches required to make multicast routing work on IPv6.)

And the purpose of this fruitless exercise? In the UK the BBC and ITV terrestial stations are broadcasting the World Cup, both of which should be on the multicast ... laptop + multicast TV = portable World Cup!

2006-06-09T18:13:00.000+01:00

Chris Gutteridge once commented that Citebase's abstract ('citations' ) page is rather overloaded with information. With that in mind I've been reworking Citebase's display in a similar fashion to Celestial. Of course, with Citebase this is a much bigger job.

A problem for pretty much any web site is how to provide lots of information while avoiding 'overloading'. I've always thought Citebase had a reasonably clean interface, but that references, citations and co-citations we laid out sequentially on page resulted in a lot of scrolling to navigate around.

So I've now redesigned the citations page making use of some 'AJAX' to provide buttons that - when selected - load a different chunk of citation data into the page. Looking at the users of Citebase most use Internet Explorer, followed by Firefox etc. In Firefox this loading/reloading is pretty clean and works with forward/backward navigation. With some suitable 'fixes' to get hold of the HTTPRequest object in IE (http://developer.apple.com/internet/webcontent/xmlhttpreq.html) the same sub-pages can be dynamically loaded. However, when navigating back to a page using forward/backward IE shows the original page (and not the one modified by AJAX). So, do I change to a total-page reload (now much easier to do having broken down the page into distinct methods) or find some way to hack IE?

It's also become apparent IE has a really bad way of handling button padding. It uses a percentage of the text width to pad and ignores any stylesheeting. It turns out the way to fix this is to use "* button { overflow: visible; }", which is a css hack to effect only IE.

The drip-feed of emails concerning the mismatch of citation-count figures in Citebase continues. This is because the citation counts in some places are taken from the record and in others from the duplicates. I thought I had got this correct, but it seems not.

Reinventing the Wheel and Other Exercises

2006-06-08T10:14:00.000+01:00

So, it looks like the re-write of Celestial is nearing its conclusion. As it was Celestial was getting to be a pain to maintain, as well as being particular user un-friendly. Now the OAI sets membership has been separated into per-repository tables, set-based selection performance is greatly increased (at least until a single repository ends up getting stupidly big). There's now a proper interface for managing subscriptions - the ability to register for reports on Celestial's harvesting.

Internally I've adopted a more-modular approach to the web interface, with each section in its own (lightly wrapped) .pm file. This has made it surprisingly easy (and neat) to add additional outputs. All of the existing functionality is there: the OAI interface, ListFriends, repository listing and editing, but now is all through a common Apache interface.

So, stuff I've learnt in this exercise (NB this is under Redhat Enterprise 4):

Apache::RequestIO is needed to enable $r->read(), otherwise CGI fails on HTTP POST (had similar problems with hashes and requiring APR::Table). The mod_perl use of modules is pretty infuriating.
There are some interesting, subtle issues in XML::LibXML when trying to generate SAX events from a sub-tree. Basically, the thing that SAX events are generated from has to be the root node, which means if you want to include a DOM fragment in another structure that generates SAX events the subtree has to be extracted and set as the root node of another DOM. So far, so annoying, but another bug inside LibXML caused me a headache. It seems LibXML segfaults if you try to set a subtree as the documentElement (presumably because it frees() the old root element, clobbering the subtree you just set as the documentElement). If you're wondering why you would ever want to do this, well my OAI library is based on sticking DOM fragments into a perl OO structure that outputs by generating SAX events.
The HTTP connector classes across browsers behave the same, but are called different things and set self to different things in the callback. Why does this matter? Well, the different names can be handled using javascript voodoo, but if you want to open up multiple HTTP connectors getting a handle to the particular connector that triggered a callback is impossible (with the exception of Opera, which apparently sets self to be the connector object). Instead, for my Celestial AJAX experimentation, I had to store each connector in an array, then interogate each one in turn to spot the one that was in a ready state. More regrettable global-fudges to get around stateless callbacks.

This is the first post to this blog - if you're reading it, blimey. Essentially this will be my effort to document problems I've grappled with and the general grind of working on the tools I've developed and now support (Citebase, Celestial and ROAR). This blog is aimed at myself - as I'm useless at keeping a lab-book and can't find what I want in them anyway - but if it helps you, that's great.