Thursday, July 12, 2012

Compacting VirtualBox Linux partitions

After re-installing Debian in a VirtualBox VM guest, the dynamically-allocated VDI obviously still contained the old data. Google didn't find a quick recipe on how to fix this - so here it is:

  1. Mount the VDI using:
    vdfuse -f volume.vdi /mnt
  2. zerofree the ext partition (look at the partition table and the file sizes in /mnt if unsure):
    zerofree /mnt/Partition1
  3. Compact the VDI:
    VBoxManage modifyhd volume.vdi --compact
This approach works from a Linux host for ext2/ext3 partitions within the VDI. It should also be possible to run zerofree from within the guest system, after changing to runlevel 1 and re-mounting the system partition read-only.

Update: The first two steps also work with VMDKs. Before Step 3, the VMDK has to be converted to VDI though:
VBoxManage clonehd volume.vmdk volume.vdi --format VDI

Thursday, April 5, 2012

Hacking Network Manager to solve auto-mounting in Gnome/Ubuntu - for good

A while ago I submitted a patch to the Gnome Network Manager mailing list, which enhances Network Manager to auto-mount network drives when connected to known networks (I've written about this issue before). The source is on github, along with more info.

Although the patch is not upstream, here's a quick post mortem on how it evolved.

Setting the stage

As a newbie in Gnome-development, I used this info on the Gnome Wiki to get a head start. I preferred Eclipse over anjuta because I'm already familiar with it. I had to install the following Ubuntu packages for being able to compile Network Manager:
autotools libtool libgtk-3-dev libclutter-1.0-dev libgda-4.0-dev libgconf2-dev intltool gtk-doc-tools libdbus-glib-1-dev libgudev-1.0-dev libnl-dev uuid-dev libnss3-dev ppp-dev libgnome-keyring-dev libnotify-dev
Getting & building the Network Manager and Network Manager applet (nm-applet) from source is described on their wiki. For installing to a workspace directory, I used this configure line with Network Manager:
./autogen.sh --prefix=/home/dom/workspace/NetworkManager --sysconfdir=/etc --localstatedir=/var
Telling nm-applet to link against the custom Network Manager required these build steps:
export PKG_CONFIG_PATH=/home/dom/git/NetworkManager:/home/dom/workspace/NetworkManager/lib/pkgconfig

./autogen.sh --prefix=/home/dom/workspace/nm-applet --sysconfdir=/etc --localstatedir=/var
For running the modified versions, the Ubuntu-packaged ones have to be killed before.

Finding the hooks

After successfully running my custom-build versions I could start hacking. The mounting was to be done as the current session user (to be able to use the stored credentials). As the Network Manager daemon runs as root, I looked at the nm-applet source. At first I was looking for a single point of code where the connection events (connect/disconnect) occur. As far as I can tell, there is none (in the applet). So I dived into the connection-type-specific sources, which is applet-device-wifi.c for WiFi.

I isolated the wireless_device_state_changed function to be the most reliable place for getting the WiFi connect/disconnect events. For testing, I triggered shell scripts that would already mount/unmount network drives.

Mounting & unmounting the drives

While the shell scripts were an easy solution, they're not quite easy to maintain with a GUI, and not easy to package. So in the next step, I replaced the scripts with using the gvfs API directly.

The scripts would work with a URI, such as smb://server/share (passed to the gvfs-mount command). The hardest part was finding a way to mount such a URI using the API: gvfs contains quite some layers of abstraction, and only GFile provides a way of creating an object using such a URI. It figured out that g_file_mount_enclosing_volume was the function I was looking for. For unmounting, finding g_file_find_enclosing_mount and g_mount_unmount_with_operation were rather easy now.

So I could now mount fixed network drives upon WiFi connect.

GUIifaction...

Mounting fixed drives was nice for my own use, but for real live the next TODO was enhancing the connection editor with appropriate settings. Using glade, the interface was quickly built. Integrating it into the applet was a matter of copy & paste (I used the IPv4 page as copy-base, as it also provides a table).

The only thing that took a while was the dropdown for the supported protocols within the table.

... and the quest for persistent settings

I now had a table view that could edit the network drives to be auto-mounted - now I had to persist the settings. To keep things simple, I decided to not store individual columns, but assemble them into the URIs that are required for mounting. So each row is basically just one string - and the table itself is an array of strings. nm-applet does not store the connection configurations itself, but passes them to the Network Manager daemon via dbus to have them stored there (which makes a lot of sense).

So I had to pass the array of URI strings via dbus to Network Manager, where I had to take them and persist them with the other configuration settings. This is where I really missed collection data types and built-in serialization that Java or C#/.NET provide... it cost me hours finding the right dbus data type and getting all the tiny code fragments right.

Now I could edit network drives in the GUI, they would be mounted on connect and unmounted on disconnect.

Making it human

So it worked now, but it requires an advanced user to actually use the connection editor. Thinking a while on a more intuitive solution, I came up with capturing manual mount events and then asking if they should be persisted. Implementing this was fun: Everything worked as expected quite quickly.

But wait: Why do I have to enter the WPA key every time I added/removed a network drive this way?

Solving this took really long: In the connection editor, the network drives are persisted with the other settings - I could just "backpack" them. Modifying them outside the connection editor meant I had to write code for persisting them on my own - which was really straightforward in the first place.

The network credentials, though, are stored in some special way - they have to fetched separately via dbus. If the network settings are saved without fetching and re-adding the credentials, they're overwritten. The code for getting this right required a callback chain that is not trivial. And as soon as it worked, I started receiving segfaults. It was only with the support of gicmo, an experienced Gnome coder, that I was able to use gdb ("thread apply all bt full") and valgrind to identify the memory leak.

This is also the reason the patch only works with WiFi-connections at the moment: For every other connection type, it has to be determined if there are secret settings stored this way. If so, fetching and re-adding those has to be implemented.

The result

Here's some screenshots of how the patch looks like:




Wednesday, March 21, 2012

Auto-mounting network drives in Linux/Ubuntu when server is available

After switching the network connection of my Ubuntu desktop box from LAN to Wifi/WLAN, my pam_mount configuration ceased to work. The plain reason: The Wifi connection is established by the Gnome Network Manager only after I login. pam_mount fails to mount them right before the connection is established. While I'm sure that there's some way to work that out, I chose a quick but reliable solution inspired by this thread.

In detail, here's my script that is run after login:

#!/bin/bash

SERVER=
while [ -z "$SERVER" ] ; do
SERVER=`host server | grep address`
done

gvfs-mount smb://server/share

It surely isn't perfect, but solves the need. If you want to use it, replace the two occurences of server with the name of your Windows/Samba server and share with the name of the network share you want to mount. You can of course place more gvfs-mount lines at the bottom.

It might also be a good starting point for some if-up / ip-up scripts, or even in laptop situations where the drives are only mounted when connected to the 'right' network (identified by the host lookup).

Any comments?

Monday, February 6, 2012

Findings on Bluetooth connectivity between iPhone & Android

I tried to come up with a way to discover iPhones and Android phones via Bluetooth in both directions in the last couple of days. As a tech-savvy, that sounded like a piece of cake: The Bluetooth menus on both devices feature a "discover" and "discoverable" feature, right?

On Android, everything went smooth. Except, you can't set a device to be discoverable for an unlimited timespan.

Then came the iPhone: In brief, it doesn't work. In detail, this is what I tried:
  • Looked for a Bluetooth API. There is none except iOS 5's Bluetooth 4 Low Energy (LE) one, which requires a BT 4 capable hardware - which is only the iPhone 4S currently. On my iPad 2 it doesn't discover anything. Same on my neighbor's iPhone 4S: Obviously, this API can only detect low range BT 4 LE devices. The "dual-mode" (enabling the chipset to run in the otherwise incompatible 2/3 and 4 versions, according to Wikipedia) seems unsupported.
  • Looked for 3rd party frameworks. There are some, but they require a rooted iPhone and/or take any chance of getting the App into the App store.
  • The high-level API called GameKit is only able to connect to other iOS-devices, and there are quite some people who tried to connect somehow - without success.
  • Tried to use the Bonjour low-level API (dns_sd.h) and found two problems with that:
    1. Discovering services via Bluetooth unsurprisingly only seems to work on active Bluetooth connections: I paired and connected my iPad to my Mac (had to initiate the connection from the iPad to the Mac, the other way does not seem to find a usable combination of profiles). After disabling WiFi on the iPad, I ran the BonjourWeb sample on it and
      dns-sd -A
      on the Mac to get provide a Bonjour service to be discovered. Success! The iPad discovered the sample service (Proof: disappeared when stopping the service on the Mac). The bad news: The iPad didn't discover anything as soon as the Bluetooth connection was dropped (without even touching the pairing). Side note: Without an active connection, the Mac doesn't provide a Bluetooth interface that Bonjour can bind to, so even if iOS didn't require an active connection, the other side probably would anyway. Learned assumption: iOS probably accepts Bluetooth GameKit connections (i.e. Bluetooth PAN(U) connections) temporarily without asking the user to provide a link-layer to Bonjour...
    2. I tried to setup Bonjour (zeroconf) via Bluetooth on Android. I used the existing JmDNS for that, which is capable of binding to any device that has an IP. I quickly found that the Android Bluetooth API doesn't support the PAN(U) profile (starting with Android 3.0, the API seems extended though), and even on my rooted Android phone I would have needed to add some kernel modules... Plus, JmDNS also requires an IP address that is only assigned after a connection has been established (that's the behavior on my Linux box at least).
So for now, iPhone apps stay blind when it comes to discovering other "smart" Bluetooth devices: Be it Android-phones, Linux boxes or even Macs...

(post has been updated on Feb 8)