linux

Recently I encountered an issue where the read/write performance of Ubuntu's software RAID configuration was relatively poor. Fortunately, others have encountered this problem and have documented a potential cause and solution here:

• Ubuntu RAID Tweak
• Linux RAID mdraid "stripe_cache_size" vs. transfer rate

The short story is that Ubuntu uses some very conservative defaults for RAID caching. Whilst this may ensure reliable behavior across a range of hardware, it does mean that for many read/write performance will be lacklustre. The solution to this problem is to define a more aggressive caching options on any software RAID partitions that are in use.

One of the nice features of VMWare Server 2.0 is that it supports the forwarding of USB devices to virtual machines. Unfortunately when it comes to Linux the VMWare team have leveraged an old method (/proc/bus/usb) for scanning the USB bus which newer distributions, such as Ubuntu Server 8.04 no longer support.

To resolve this problem the "old" method for scanning for USB devices must be enabled in the underlying operating system. In the case of Ubuntu Server 8.04 this is a case of editing the file /etc/init.d/mountdevsubfs.sh and uncommenting the following section:

#
# Magic to make /proc/bus/usb work
#
mkdir -p /dev/bus/usb/.usbfs
domount usbfs "" /dev/bus/usb/.usbfs -obusmode=0700,devmode=0600,listmode=0644
ln -s .usbfs/devices /dev/bus/usb/devices
mount --rbind /dev/bus/usb /proc/bus/usb

Reboot the server and /proc/bus/usb should be functional once more.

Activating a USB device within a virtual machine

Once the underlying USB subsystem is configured the USB device needs to be associated with a virtual machine. For this to occur the virtual machine must have the USB Controller added to its virtual hardware configuration. If the controller is not already part of the virtual machine's configuration shutdown the VM, add the device and restart.

The VMWare web console with USB device selection (click to enlarge)

Assuming there are USB devices attached to the server, once the virtual machine boots a small USB icon will appear within the VMWare web management console. Click on the icon and select the relevant USB device to attach it to the running virtual machine.

All going well the USB device will appear within the virtual machine as an accessible device. VMWare Server remembers this selection, so the next time the virtual machine (or server itself) is restarted the USB device will automatically be attached to the running VM.

VMWare Server 2.0 is emerging as a capable, zero cost alternative to VMWare ESX when used in combination with Ubuntu Server 8.04LTS. Unfortunately "out of the box" performance can be a little disappointing, especially when running guest Windows virtual machines. What follows are a few system tweaks that can improve performance without hampering overall system stability. I have not come up with these myself, instead they are pruned from the following pages:

• VMWare FAQ: I need more performance out of my VMware environment
• VMWare Communities: Performance tuning in Server 2.0
• Ubuntu Server Installation with VMware Server

Kernel parameters

In addition to the default Ubuntu Server kernel parameters, the following should be appended to the end of /etc/sysctl.conf.

vm.swappiness=0
vm.overcommit_memory=1
vm.dirty_background_ratio=5
vm.dirty_ratio=10
vm.dirty_expire_centisecs=1000
dev.rtc.max-user-freq=1024

Once added reboot the server to ensure their application is successful and permanent.

Create an in-memory temp drive

In the host's /tmp directory create a new directory named vmware (e.g. /tmp/vmware). This will be used as the mount point for a tmpfs (in-memory) partition for storing VM related, temporary files.

Edit /etc/fstab and add the /tmp/vmware partition to your list of mount points:

tmpfs /tmp/vmware tmpfs defaults,size=100% 0 0

Now if you execute the following command the tmpfs filesystem will be mounted at /tmp/vmware:

sudo mount /tmp/vmware

If successful, reboot the Ubuntu server to ensure the tmpfs partition is mounted at boot time.

VMWare Server configuration

Edit the /etc/vmware/config file and ensure the following configuration declarations are set:

prefvmx.minVmMemPct = "100"
prefvmx.useRecommendedLockedMemSize = "TRUE"
mainMem.partialLazySave = "TRUE"
mainMem.partialLazyRestore = "TRUE"
tmpDirectory = "/tmp/vmware"
mainMem.useNamedFile = "FALSE"
sched.mem.pshare.enable = "FALSE"
MemTrimRate = "0"
MemAllowAutoScaleDown = "FALSE"

These configuration declarations instruct VMWare Server to keep all virtual machines in memory and not to write unused blocks to disk. It also sets the temporary directory to the newly created tmpfs partition at /tmp/vmware.
Restart the VMWare Server process (sudo /etc/init.d/vmware restart) or reboot the server for these changes to take effect. The net result should be notably smoother virtual machine performance, especially when it comes to Windows guests.

Virtual machine tips

• Always use fully allocated disk images.
• Do not use snapshots as they are approximately 20% slower.
• Always install the VMWare Tools package.
• If running Linux make sure the kernel is compiled for running within a VM, or is using the correct boot time parameters.

Centos is a "free" distribution of Red Hat Enterprise Linux which I enjoy using. Whilst it does not have Debian's apt-get for package management it does have yum, which is not as fast but still works pretty well in most circumstances. Unfortunately today I ran into a problem in a clean install of Centos 5.3 where yum was returning the following error:

Error Message: Metadata file does not match checksum

A quick look around "the Google" turned up the relatively simple solution; at the terminal execute as root:

yum clean all
yum makecache
yum update

This process will take a little time, but the end result should be a nice, clean yum repository cache, complete with no annoying meta-data errors.

This post explains how to setup a Squid HTTP proxy to transparently authenticate users against a Novell eDirectory. In the Novell eco-system Border Manager is the venerable choice for an internal firewall and proxy but it is showing its age. This guide is based on this Novell Cool Solution. Unlike Border Manager, which requires the CLNTRUST client-side tool, the setup described works without the need for any desktop client software.

How it works

Within a Novell managed network the eDirectory stores authenticated user's I.P. addresses. Squid performs an LDAP search against eDirectory using the incoming I.P. address of the client. If successful the authenticated username is returned and a proxy session established. If the search comes up empty Squid prompts the client to manually enter their credentials for authentication against the eDirectory. If this too fails the proxy request is denied.

eDirectory 8.8 incompatability

This solution currently only works with eDirectory < 8.8 because Novell has slightly changed the format they store network addresses in newer versions. At the time of writing I have not been able to test against eDirectory 8.8 so I cannot determine the required code changes or test results. Hopefully in the near future this situation will change.

Squid's external_acl_type option

Transparent authentication is made possible thanks to Squid's external_acl_type configuration option. This allows external identities and groups to be identified via any external script. Once Squid is installed setting up transparent eDirectory authentication is a two step process:

1. Create and tweak the squid_edir_iplookup.pl file.
2. Edit the squid.conf configuration file

Recently I put together a webcam for the Aorangi Ski Club's lodge on Ruapehu. The system consists of a second-hand laptop, an old Canon camera and Ubuntu Linux. Images are automatically captured every hour, timestamped and posted to Flickr. The end result is a pretty impressive looking Flickr slideshow.

To put a similar setup together follow these steps.

Step 1: Get the gear

Find a Linux compatible laptop and a Canon camera compatible with the Capture software.

SELinux is as painful to use sometimes as it is powerful when it comes to locking down server permissions. Unfortunately even with distributions such as Red Hat which supports SELinux out of the box, you will still experience problems.

One such issue I came across recently was automounting CIFS shares on boot using netfs. At startup the netfs service was returning an "error 13 - error opening credentials file" when attempting to mount the CIFS shares. The problem was the SELinux was not allowing the netfs script to access the file that contained the CIFS authorisation details.

For example, my /etc/fstab had the following entry:

//WINDOWSSERVER/SHARE          /mnt/windowsshare             cifs    credentials=/etc/samba/auth.cifs        0 0

And in the /etc/samba/auth.cifs file were the following details:

username=windowsuser
password=windowspassword

The solution to the problem was to change a SELinux boolean parameter with the following command (found here):

setsebool -P allow_mount_anyfile 1

This lets the mount command open any referenced file, effectively side-stepping the netfs error. Sure it is not 100% secure but it works without having to completely disable SELinux which seems to be most people's answer to any problems.

Ubuntu 6.06LTS is a useful platform for VMWare because it has a small footprint by todays standards and is supported by Canonical until 2011. Unfortunately installing VMWare Server 1.0 can be a little painful given there are no binary kernel modules for Ubuntu in this release.

Fortunately there are some excellent guides for installing VMWare on this platform such as this one from HowtoForge. What follows is an installation script based on the HowtoForge guide that saves the administrator a lot of time and solves a bug along the way.

Google recently released the highly anticipated Android mobile phone platform to developers. Android promises to be a more consistent and powerful environment for mobile applications compared to what currently exists in the fragmented mobile market. Whilst many people were disappointed that Android was not a Google-branded iphone; from a developers perspective if it can gain broad adoption it will make the developing powerful, Internet-centric mobile applications significantly easier.

One of the most interesting aspects of Android is that it is released under the Apache v2 software license. This license grants obligation-free use of the code to any party. This is different to other popular open-source licenses like the GPL which requires source-code modifications to be made publically available. In the competitive mobile phone market such an obligation is problematic which is why Sun releases the Java Mobile Edition (ME) under different open and closed source licenses.

A question that was hanging around Android was how Google had managed to release a Java mobile platform under the Apache license given that the licenses Sun release JavaME under are not compatible. Stefano Mazzocchi points out on his blog that Google have outflanked Sun by releasing a platform that supports the Java language but does not use Sun's Java compiler or the Java byte-code at its core. Instead Google have created Dalvik, a virtual machine released under the Apache license which understands how to compile Java source code into its own byte-code for execution.

This move outflanks Sun's licensing policies, essentially cutting them out of the Android equation. It is a gutsy move by Google but it does free them to focus on developing a platform they have complete control over rather than working in partnership with Sun. From the perspective of Java as a language this move wouldn't seem to pose any problems as the mobile and desktop/server worlds have always been quite distinct. Plus if anything Google's use of the Apache Harmony JavaSE libraries may actually make developing for the mobile and desktop more consistent than Sun's distinct JavaME and JavaSE implementations.

All things considered this news has made Android more interesting from my perspective. Before I heard this it was just another JavaME implementation but now it sounds like Google will have the capability to do some really interesting things. What has yet to be seen is what level of support (if any) this platform will have on the iPhone. Google and Apple have a strong relationship there and it would seem like Dalvik runtime would be a natural fit on the device if it is lightweight, fast and provides developers with the ability to write applications for both Android and the iPhone. 

Today the lead developer of the CUPS (Common Unix Printing System) project announced that in February 2007 Apple purchased the CUPS source code and took him on as a staff member. CUPS is significant within the *NIX world because it is arguably the most well supported and feature-rich printing system available. Apple has used it within OSX from the outset and personally I feel it is the best implementation of CUPS available thanks to the Aqua interface and the fact that the majority of printers just work without any effort.

There would seem to be no immediate danger of CUPS code being completely closed sourced considering it is currently released under the GPL2 license. What the code purchase suggests is that Apple probably plans on internally re-licensing the code under a closed source license for many, if not all of the platforms that could make use of it, i.e. Mac, AirPort and the iPhone.

Dual licensing such as this is not uncommon by vendors who control the rights to GPL code. It is a good way of benefiting from an open source development model and community whilst still being able to ship a modified version of the code on closed devices or only binary form. License flexibility such as this will become increasingly important as the GPL3 license is adopted as it resolves many of the loopholes vendors used to ship GPL code in what effect were closed devices (Tivoization).

Now that Apple does have a formal stake in CUPS my biggest hope is that they can spare a graphic designer to give the website and the CUPS interface with a much needed aesthetic overhaul and maybe a better logo...