Entries by admin | s9y testdrive

Character collation

Posted by christian on Wednesday, April 6. 2016

So, recently I came across this funny behaviour on a SLES11sp4 machine:

sles11$ netstat -ni | awk '/^[a-z]/' 
Kernel Interface table
Iface   MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   1500   0     3562      0      0      0     1955      0      0      0 BMRU
lo    16436   0       20      0      0      0       20      0      0      0 LRU

Wait, what? Why is the (uppercase) string "Kernel" matched against the lowercase "[a-z]" search expression? The same command on a SLES12sp1 machine does the Right Thing:

sles12$ netstat -ni | awk '/^[a-z]/' 
eth0   1500   0      685      0      0      0      438      0      0      0 BMRU
lo    65536   0       12      0      0      0       12      0      0      0 LRU

Apparently, this is not an unknown problem and can indeed be fixed by providing another LC_COLLATE variable:

$ netstat -ni | LC_COLLATE=C awk '/^[a-z]/' 
eth0   1500   0     3711      0      0      0     2032      0      0      0 BMRU
lo    16436   0       20      0      0      0       20      0      0      0 LRU

While providing a different LC_COLLATE variable did help, this still smells like a bug in SLES11, as the configured locales were exactly the same:

sles11$ locale 
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

sles11$ locale -k LC_COLLATE
collate-nrules=4
collate-rulesets=""
collate-symb-hash-sizemb=2039
collate-codeset="UTF-8"

sles11$ locale | md5sum 
677d9b3dbdf9759c8b604f294accd102  -

sles12$ locale | md5sum 
677d9b3dbdf9759c8b604f294accd102  -

Interestingly enough, both installations differ greatly in the way they look up locale information:

sles11$ echo | strace -e open awk '/^[a-z]/' 
open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/lib64/libdl.so.2", O_RDONLY)     = 3
open("/lib64/libm.so.6", O_RDONLY)      = 3
open("/lib64/libc.so.6", O_RDONLY)      = 3
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3



sles12$ echo | strace -e open awk '/^[a-z]/' 2>&1 | grep -v ENOENT
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/locale/en_US.utf8/LC_CTYPE", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib64/gconv/gconv-modules.cache", O_RDONLY) = 3
open("/usr/lib/locale/en_US.utf8/LC_COLLATE", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/locale/en_US.utf8/LC_MESSAGES", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/locale/en_US.utf8/LC_NUMERIC", O_RDONLY|O_CLOEXEC) = 3
open("/usr/lib/locale/en_US.utf8/LC_TIME", O_RDONLY|O_CLOEXEC) = 3
open("/dev/null", O_RDWR)               = 3
+++ exited with 0 +++

Alas, no bug has been reported yet :-\

While this appears to be documented behaviour, it's still very confusing and may even violate the Principle of Least Surprise. FWIW, GNU/grep behaves as expected on both systems, no matter the collation:

$ echo Abc | egrep --color '[[:lower:]]'
Abc

PS: I forgot to mention how cool SUSE Studio is - this SLE12 test VM was up & running in minutes and accessible via SSH too and I didn't even have to fire up my local VirtualBox instance! :-)

umask & symbolic links on MacOS X

Posted by christian on Wednesday, March 9. 2016

This just annoyed me again:

$ umask 0022
$ touch foo
$ umask 0066
$ ln -s foo bar

$ ls -lgo foo bar
-rw-r--r--  1   0 Mar  9 14:17 foo
lrwx--x--x  1   3 Mar  9 14:17 bar -> foo

$ sudo -u nobody cat foo bar
$

OK, this seems to work (the permissions are checked on the target, not the symlink), but not so with directories:

$ umask 0022
$ mkdir -p foo/file
$ umask 0066
$ ln -s foo bar

$ ls -ldgo foo bar
drwxr-xr-x  3   102 Mar  9 15:02 foo
lrwx--x--x  1     3 Mar  9 15:03 bar -> foo

$ sudo -u nobody ls -l bar
ls: bar: Permission denied
lrwx--x--x  1 admin  wheel  3 Mar  9 14:23 bar

Interestingly enough, it works if we append a slash to the symlink:

$ sudo -u nobody ls -lgo bar/
total 0
drwxr-xr-x  2  68 Mar  9 14:24 dir

This is annoying when a user has a more stringent umask for normal use, but temporarily elevates its privileges to install software, without adjusting the umask first. To clean up this mess afterwards, we can re-create the affected symbolic links:

$ umask 0022
$ find . -type l ! -perm -g+r | while read l; do
   target=$(readlink "$l") && rm -f "$l" && ln -svf "$target" "$l"
done
./bar -> foo

$ ls -ld foo bar
drwxr-xr-x  4 admin  wheel  136 Mar  9 14:37 foo
lrwxr-xr-x  1 admin  wheel    3 Mar  9 14:38 bar -> foo

Note: this has been seen in MacOS 10.10.5 on a Journaled HFS+ file system.

OpenBSD & CVS

Posted by christian on Monday, February 29. 2016

Every now and then I start up my OpenBSD VM to see how things are in BSD-land. And of course, after the VM has been asleep for a few month, I'd like to update the system too. As OpenBSD still uses CVS to manage their source repositories (for various reasons), we may have no other choice but to use it:

$ cd /usr/src/
$ time cvs -q up -rOPENBSD_5_8 -Pd
[...]
U usr.sbin/zic/zic.8
U usr.sbin/zic/zic.c
P usr.sbin/ztsscale/ztsscale.c
  158m51.96s real     0m16.85s user     7m34.07s system

The tree is about 780 MB in size and took 2.6 hours to complete. And we haven't even started the build yet. Wat?

There's an unofficial Git tree for openbsd-src, but before we revert to that, let's try the recommended alternative, CVSync.

Let's look at the available repositories first:

$ cvsync cvsync://anoncvs.usa.openbsd.org/
Name: openbsd, Release: rcs
 Comment: OpenBSD CVS Repository
Name: openbsd-cvsroot, Release: rcs
Name: openbsd-ports, Release: rcs
Name: openbsd-src, Release: rcs
Name: openbsd-www, Release: rcs
Name: openbsd-x11, Release: rcs
Name: openbsd-xf4, Release: rcs
Name: openbsd-xenocara, Release: rcs

We're just interested in openbsd-src for now:

$ sudo mkdir -m0775 /cvs && sudo chgrp wsrc /cvs       # We're not using doas yet.
$ cat /etc/cvsync_openbsd.conf
config {
       hostname anoncvs.usa.openbsd.org
       base-prefix /cvs

       collection {
               name openbsd-src release rcs
               umask 002
       }
}

$ cvsync -c /etc/cvsync_openbsd.conf

The initial sync took well over 3 hours to complete, but successive runs tend to complete in a few minutes, much less than updating with plain cvs.

However, the result is unusable yet:

$ ls -1 /cvs/src/sys/arch/`uname -m`/conf          
Attic
GENERIC,v
GENERIC.MP,v
Makefile.i386,v
RAMDISK,v
RAMDISK_CD,v
files.i386,v
ld.script,v

No, we have to checkout a local copy now, before we can start using it:

$ cd /usr/src
$ cvs -d /cvs checkout -P src
$ cvs -d /cvs up -Pd

Only now we'll be able to actually update the system. At last, the Git checkout was quick and so much less painful:

$ time git clone https://bitbucket.org/braindamaged/openbsd-src.git openbsd-src-git
[...]
real    12m57.329s
user    4m5.468s
sys     0m54.316s

$ cd $_
$ ls -1 sys/arch/`uname -m`/conf
files.i386
GENERIC
GENERIC.MP
ld.script
Makefile.i386
RAMDISK
RAMDISK_CD

Vacation pictures

Posted by christian on Sunday, January 10. 2016

The holidays are over and I had to dig through heaps of vacation pictures and wanted to create a little photo gallery for my fellow relatives to click through. After past experiments with Zenphoto and Piwigo, I wanted to switch to a much more simpler solution. One that wouldn't require a database backend and maybe didn't break after a few update cycles.

Looking at static image gallery generators I decided to try llgal again. The command line switches are more difficult to remember than tar, but here we go:

llgal --www --sort revtime --ct %Y-%m-%d -a -d . -k --title "Pictures of Foo"

This will process pictures in the current directory, with the following options:

--www           make all llgal files world-readable
--sort revtime  sort pictures in reverse-mtime (oldest pictures on top)
--ct %Y-%m-%d   use image timestamps as captions, YYYY-mm-dd
-a              write image sizes under thumbnails on index page
-d              operate in directory <dir>
-k              use the image captions for the HTML slide titles
--title         title of the index of the gallery

So far, so good. But some obstacles had to be tackled first:

Each picture on the camera was ~3-5 MB each and I didn't want to upload these large files to the gallery. So I resized the pictures with some photo program (not with GraphicsMagick) but now the file's mtime got mangled. GNU/touch was able to fix this.
The pictures were taken with two cameras. Unfortunately, one of the cameras had its system time off by two hours - this had to be fixed as well.

As all the pictures (from both cameras) are now in one directory, this is how it looked like:

$ exiftool -s DSCN_001.jpg IMG_002.jpg | grep ^DateTimeOriginal
DateTimeOriginal                : 2015:12:23 18:01:00
DateTimeOriginal                : 2015:12:23 16:03:00

In reality, DSCN_001.jpg was taken at 16:01 and should be listed before IMG_002.jpg. Luckily exiftool is able to correct the EXIF data:

export delta="00:00:00 02:00:00"            # format is YY:mm:dd HH:MM:SS
ls DSCN* | while read f; do
  echo "FILE: $f"
  exiftool -P -ModifyDate-="$delta" -DateTimeOriginal-="$delta" -CreateDate-="$delta" "$f"
  touch -r "$f" -d '-2 hours' "$f"
done

Although we corrected the file's mtime already, it was still mangled by the previous export step. Let's just extract the exact date from the EXIF data and correct the mtime again:

ls *JPG | while read f; do
  echo "FILE: $f"
  TZ=PST8PDT touch -d "$(exiftool -d %Y-%m-%d\ %H:%M:%S -s "$f" | awk '/^DateTimeOriginal/ {print $3,$4}')" "$f"
done

After another llgal run, the pictures were now listed in their correct order and ready to be consumed :-)

RTNETLINK answers: No such process

Posted by christian on Wednesday, November 25. 2015

A colleague of mine presented me with a weird routing problem today and it took me a while to understand what was going on. The task was simple: add a network route via a certain gateway that can only be reached via a certain network interface. Let's re-create the setup:

# ip addr change 10.10.0.3/24 dev eth2
# ip link set eth2 up
# ip addr show dev eth2 scope global
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 08:00:27:d0:34:51 brd ff:ff:ff:ff:ff:ff
    inet 10.10.0.3/24 scope global eth2

Let's add a new route then:

# ip route add 10.20.0.0/24 via 10.10.0.1 dev eth2
RTNETLINK answers: No such process

Huh? Our eth2 is UP and should be able to reach 10.10.0.1, right? Let's look at the routing table¹⁾:

# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.56.0    0.0.0.0         255.255.255.0   U         0 0          0 eth0
0.0.0.0         192.168.56.1    0.0.0.0         UG        0 0          0 eth0

Aha! For some reason the machine has lost its network route on the eth2 interface. Well, the machine has been online for a while and we don't know which admin did what and why. But although eth2 is configured and UP, it cannot reach its own network w/o a network route. Of course, the "ip addr change" does that automatically²⁾ and we staged the whole thing for illustration purposes.

Let's add the missing route and try again:

# ip route add 10.10.0.0/24 dev eth2 
# netstat -rn 
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.10.0.0       0.0.0.0         255.255.255.0   U         0 0 0 eth2
192.168.56.0    0.0.0.0         255.255.255.0   U         0 0 0 eth0
0.0.0.0         192.168.56.1    0.0.0.0         UG        0 0 0 eth0

# ip route add 10.20.0.0/24 via 10.10.0.1 dev eth2
# netstat -rn 
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.20.0.0       10.10.0.1       255.255.255.0   UG        0 0 0 eth2
10.10.0.0       0.0.0.0         255.255.255.0   U         0 0 0 eth2
192.168.56.0    0.0.0.0         255.255.255.0   U         0 0 0 eth0
0.0.0.0         192.168.56.1    0.0.0.0         UG        0 0 0 eth0

Yay! :-)

¹⁾ Sometimes the output from the iproute2 tools are not as easy to parse and I'll use good ol' net-tools again.
²⁾ Unless we were to assign a /32 address to the interface, e.g. "ip addr change 10.10.0.0/32 dev eth2"

You say Tomato, I say Shibby

Posted by christian on Monday, October 12. 2015

So, this Netgear router went on to become a brick and could not be resurrected from the dead. But I had an old WRT54GL still lying around that could be used until I decide which router to buy as a replacement.

Unfortunately, as the WRT54GL is now 10 years old, neither DD-WRT nor OpenWRT ships any recent version of their firmware for this model. So what else is out there?

Enter Tomato, yet another firmware-for-your-router-project. And while the original project appears to be dormant, many mods were created and some of them are still active. I went with the most recent one, called Tomato by Shibby which offers images with both the Linux 2.4 and 2.6 kernel. I went with the latest K26 release (for MIPS1) that would fit into the 4 MB of flash memory on this router:

$ w3m http://tomato.groov.pl/download/K26/build5x-131-EN/ | grep MIPSR1 | sort -nk4
file   tomato-K26-1.28.RT-MIPSR1-131-MiniIPv6.zip         2015-08-06 3610
file   tomato-K26-1.28.RT-MIPSR1-131-Mini.zip             2015-08-06 3786
file   tomato-K26-1.28.RT-MIPSR1-131-MiniVPN.zip          2015-08-06 4157

Verify the checksum:

$ wget http://tomato.groov.pl/download/K26/build5x-131-EN/MD5SUM
$ md5sum -c MD5SUM 2>&1 | grep OK
image/tomato-K26-1.28.RT-MIPSR1-131-MiniIPv6.trx: OK

The Tomato image can be installed through the DD-WRT or OpenWRT GUI, as the firmware should be recognized by both systems. On OpenWRT, this can also be done from the command line:

sysupgrade -v -n tomato.trx

With that, Tomato by Shibby was up & running on this old WRT54GL :-)

$ ssh router
Tomato v1.28.0000 MIPSR1-131 K26 MiniIPv6
 ======================================================== 
 Welcome to the Linksys WRT54G/GS/GL [router]
 Uptime:  14:15:23 up 1 day, 14:40
 Load average: 0.22, 0.05, 0.01
 Mem : used 92.2% (11.78 of 12.78 MB)
 WAN : 12.34.56.78/22 @ 00:12:23:34:45:56
 LAN : 192.168.0.1/24 @ DHCP: 192.168.0.100 - 192.168.0.149
 WL0 : SSID @ channel: Worldwide9 @ 00:12:34:56:78:90
 ========================================================

VirtualBox network performance

Posted by christian on Wednesday, September 9. 2015

Some time ago I had some network performance issues with a VirtualBox guest and I was able to solve it by switching to a different NIC type. But I wanted to find out how the different types are performing and also if there's a difference between the different network modes too. And yes, there is! :-)

Results I

After some test runs, here are the results:

 HOST: Debian/GNU Linux 8.2 / Kernel 4.2.0 x86_64 (vanilla) / VirtualBox 5.0.4
GUEST: Debian/GNU Linux unstable / Kernel 4.2.0-trunk-amd64

Am79C970A / hostonly  580 Mbits/sec
Am79C970A / bridged  473 Mbits/sec
Am79C970A / natnetwork  640 Kbits/sec ¹⁾
Am79C970A / nat  396 Mbits/sec

Am79C973 / hostonly  569 Mbits/sec
Am79C973 / bridged  285 Mbits/sec
Am79C973 / natnetwork  640 Kbits/sec
Am79C973 / nat  438 Mbits/sec

82540EM / hostonly  1.89 Gbits/sec
82540EM / bridged  1.86 Gbits/sec
82540EM / natnetwork  640 Kbits/sec
82540EM / nat  449 Mbits/sec

82543GC / hostonly  1.85 Gbits/sec
82543GC / bridged  1.91 Gbits/sec
82543GC / natnetwork  640 Kbits/sec
82543GC / nat  357 Mbits/sec

82545EM / hostonly  1.85 Gbits/sec
82545EM / bridged  1.90 Gbits/sec
82545EM / natnetwork  640 Kbits/sec
82545EM / nat  389 Mbits/sec

virtio / hostonly  705 Mbits/sec
virtio / bridged  682 Mbits/sec
virtio / natnetwork  640 Kbits/sec
virtio / nat  129 Mbits/sec

The clear winner appears to be 82543GC (Intel PRO/1000 T Server) for bridged mode or 82540EM (Intel PRO/1000 MT Desktop) for hostonly mode.

Results II

And again on a (slower) MacOS X host:

 HOST: MacOS 10.10.5 / X86_64 / VirtualBox 5.0.4
GUEST: Debian/GNU Linux 8.0 / Kernel 4.1

NIC: Am79C970A / MODE: hostonly  29.6 MBytes/sec
NIC: Am79C970A / MODE: bridged  29.9 MBytes/sec
NIC: Am79C970A / MODE: natnetwork  25.2 MBytes/sec
NIC: Am79C970A / MODE: nat  25.8 MBytes/sec

NIC: Am79C973 / MODE: hostonly  28.7 MBytes/sec
NIC: Am79C973 / MODE: bridged  30.0 MBytes/sec
NIC: Am79C973 / MODE: natnetwork  1.38 MBytes/sec
NIC: Am79C973 / MODE: nat  23.4 MBytes/sec

NIC: 82540EM / MODE: hostonly  45.4 MBytes/sec
NIC: 82540EM / MODE: bridged  38.2 MBytes/sec
NIC: 82540EM / MODE: natnetwork  61.3 MBytes/sec
NIC: 82540EM / MODE: nat  47.0 MBytes/sec

NIC: 82543GC / MODE: hostonly  43.0 MBytes/sec
NIC: 82543GC / MODE: bridged  44.7 MBytes/sec
NIC: 82543GC / MODE: natnetwork  64.7 MBytes/sec
NIC: 82543GC / MODE: nat  49.3 MBytes/sec

NIC: 82545EM / MODE: hostonly - (VM would not start)
NIC: 82545EM / MODE: bridged - (VM would not start)
NIC: 82545EM / MODE: natnetwork - (VM would not start)
NIC: 82545EM / MODE: nat - (VM would not start)

NIC: virtio / MODE: hostonly  43.3 MBytes/sec
NIC: virtio / MODE: bridged  46.6 MBytes/sec
NIC: virtio / MODE: natnetwork  10.9 MBytes/sec
NIC: virtio / MODE: nat  13.8 MBytes/sec

Here, the winner appears to be virtio for bridged mode and again 82540EM (Intel PRO/1000 MT Desktop) for hostonly mode. This time, both nat and natnetwork were working, with very different performance patterns.

Results III

On a different system, the iperf results varied greatly and I decided to run the test script longer and multiple times:

for a in {1..10}; do
   echo "### $a -- `date`"
  ~/bin/vbox-nic-bench.sh vm0 300 2>&1 | tee vbox_nic_"$a".log
done

Looking at the report files we can already see that the "hostonly" network mode was the fastest, so let's run the report function over all the output files and sort by the fastest NIC:

$ for a in vbox_nic_*.log; do
   ~/bin/vbox-nic-bench.sh report $a | grep hostonly | sort -u
done | sort -nk6 | tail -5
NIC: 82540EM / MODE: hostonly  228 MBytes/sec
NIC: 82540EM / MODE: hostonly  228 MBytes/sec
NIC: 82545EM / MODE: hostonly  228 MBytes/sec
NIC: 82543GC / MODE: hostonly  229 MBytes/sec
NIC: 82540EM / MODE: hostonly  231 MBytes/sec

So, either NIC (82540EM or 82543GC) should be the fastest in our setup.

¹⁾ For some reason, I couldn't get the new natnetwork mode to work on Linux. iperf measured "640 Kbits/sec" while in fact no data was transferred:

HOST$ iperf -t 3 -c 127.0.0.1 -p 15001
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 15001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 51056 connected with 127.0.0.1 port 15001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-18.5 sec  3.06 MBytes  1.39 Mbits/sec


GUEST$ sudo tcpdump -ni eth2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes
17:05:36.569862 IP 192.168.0.104.51056 > 192.168.15.4.5001: Flags [S], seq 6583, win 32768, options [mss 1460], length 0
17:05:39.574354 IP 192.168.0.104.51056 > 192.168.15.4.5001: Flags [S], seq 6583, win 32768, options [mss 1460], length 0
17:05:42.579472 IP 192.168.0.104.51056 > 192.168.15.4.5001: Flags [S], seq 6583, win 32768, options [mss 1460], length 0
17:05:45.584319 IP 192.168.0.104.51056 > 192.168.15.4.5001: Flags [S], seq 6583, win 32768, options [mss 1460], length 0
17:05:48.589318 IP 192.168.0.104.51056 > 192.168.15.4.5001: Flags [S], seq 6583, win 32768, options [mss 1460], length 0
17:05:51.593294 IP 192.168.0.104.51056 > 192.168.15.4.5001: Flags [S], seq 6583, win 32768, options [mss 1460], length 0
17:05:54.594851 IP 192.168.0.104.51056 > 192.168.15.4.5001: Flags [S], seq 6583, win 32768, options [mss 1460], length 0

More memory with ZRAM

Posted by christian on Saturday, August 29. 2015

Some of my machines have only little memory and I looked for a better way to utilize what little memory there is in the system. Without being able to increase the physical memory available, there are basically 3 options here:

Disable unused applications
Reduce the memory footprint of running applications
Cheat :-)

After exhausting the first two options, I remembered some memory management mechanisms for the Linux kernel that were introduced a while ago but I've never used them so far:

KSM

KSM (Kernel Samepage Merging) is a memory-saving de-duplication feature but it's only really useful for applications using the madvise(2) system call and is often used when hosting virtual machines, e.g. KVM. I'm not running KVM on this machine but activated it anyway - but no application seems to use madvise(2) and it didn't help anything regarding memory usage.

zswap

There's zswap, a lightweight compressed cache for swap pages. But instead of a real swap device, the compression takes place in a dynamically allocated memory pool. To enable it, the system must be booted with zswap.enabled=1 but I didn't want to reboot my system just yet, so I skipped that option for now.

Update: I've enabled zswap in the same VM from the zram test below and ran the same test - but the results are rather irritating:

$ awk '{print $NF}' /proc/cmdline 
zswap.enabled=1

$ i=0; while true; do
   [ $i -gt 2000 -a `expr $i % 50` = 0 ] && printf "$i  "
   bash -c "sleep 10000 &"; i=$((i+1))
done
2050  2100  2150  2200  2250  2300  ^C

$ free -m
              total    used  free  shared  buff/cache  available
Mem:            241    192       3       0     46        7
Swap:           127    127       0

$ pgrep -c sleep
2333

$ grep -r . /sys/kernel/debug/zswap/
/sys/kernel/debug/zswap/stored_pages:24754
/sys/kernel/debug/zswap/pool_total_size:50696192
/sys/kernel/debug/zswap/duplicate_entry:0
/sys/kernel/debug/zswap/written_back_pages:32762
/sys/kernel/debug/zswap/reject_compress_poor:456
/sys/kernel/debug/zswap/reject_kmemcache_fail:0
/sys/kernel/debug/zswap/reject_alloc_fail:0
/sys/kernel/debug/zswap/reject_reclaim_fail:0
/sys/kernel/debug/zswap/pool_limit_hit:16253

We max out at ~2300 instances of bash & sleep which is even less than when running without any compression...?

zram

zram has been around for a while now and looked like the most promising contender. On a machine with 1GB RAM, I'd allocate 75% for our compressed swap device:

$ modprobe zram
$ echo 768M > /sys/block/zram0/disksize
$ mkswap /dev/zram0
$ swapon -p2 /dev/zram0

The machine is quite busy and it doesn't take long until it starts swapping to our new swap device¹⁾:

$ grep . /sys/block/zram0/{num_{reads,writes},{compr,orig}_data_size}
/sys/block/zram0/num_reads:1953540
/sys/block/zram0/num_writes:2333028
/sys/block/zram0/orig_data_size:671141888
/sys/block/zram0/compr_data_size:283552915

The compression ration is quite good, we're using only 42% of our precious real memory. I wanted to do some tests though to see if this can be measured in some kind of micro benchmark. In a 256MB Fedora Linux VM, we started GNU/bash along with /bin/sleep over and over again, let see how far we got:

$ i=0; while true; do
   [ $i -gt 2400 -a `expr $i % 50` = 0 ] && printf "$i  "
   bash -c "sleep 10000 &"; i=$((i+1))
done
2450  2500  2550  2600  2650  2700 ^C

$ pgrep -c sleep
2711

$ free -m
              total    used  free  shared  buff/cache  available
Mem:            241    192       3       0     45        5
Swap:           127    127       0

All memory is used up and starting any more programs is almost impossible now. This was repeatable, it always stopped around ~2700 instances and then came to a grinding halt. Let's try again with ZRAM:

$ pkill sleep
$ modprobe zram && echo 128M > /sys/block/zram0/disksize && mkswap /dev/zram0 && swapon -p2 /dev/zram0
$ i=0; while true; do
   [ $i -gt 2500 -a `expr $i % 100` = 0 ] && printf "$i  "
   bash -c "sleep 10000 &"; i=$((i+1))
done
2600  2700  2800  2900  3000  3100  3200 ^C

$ pgrep -c sleep
3201

$ free -m
              total    used  free  shared  buff/cache  available
Mem:            241    186       2       0     52        6
Swap:           255    209      46

With ZRAM enabled, it maxes out at ~3100, and makes it up to 3200 if we wait a bit longer (although we still seem to have 46MB free swap available). Again, this is also repeatable. And since we're only starting the same program over and over again, our compression ratio is even better¹⁾:

$ grep . /sys/block/zram0/{num_{reads,writes},{compr,orig}_data_size}
/sys/block/zram0/num_reads:55002
/sys/block/zram0/num_writes:70129
/sys/block/zram0/orig_data_size:128086016
/sys/block/zram0/compr_data_size:34523998

Btw, did someone say DriveSpace? :-)

¹⁾ Note: these sysfs entries will be deprecated in future kernel versions,

Filesystem data checksumming, pt. II

Posted by christian on Sunday, July 5. 2015

After my last post on filesystem data checksumming it took me a while until I could convince myself to actually set up regular checks of all the (important) files on my filesystems. The "fileserver" is a somewhat older machine and checksumming ~1.5TB of data takes almost 4 (!) days. Admittedly, the fact that I chose SHA-256 as a hashing algorithm seems to contribute to this long runtime. This being a private file server, MD5 would've have probably been more than enough.

But I wanted to know if this would really make a difference and wrote a small benchmark script, testing different programs and different digests on a particular machine. As always, the results will differ greatly from machine to machine - the following results are for this PowerBook of mine:

$ time ./test-digests.sh test.img 30 2>&1 | tee out.log
[....]
=> This took 3.5 hours to complete!

$ grep ^TEST out.log | egrep -v 'rhash_benchmark|SKIPPED' | sort -nk7
TEST: coreutils / DIGEST: md5 / 58 seconds over 30 runs
TEST: openssl / DIGEST: sha1 / 64 seconds over 30 runs
TEST: rhash / DIGEST: sha1 / 64 seconds over 30 runs
TEST: openssl / DIGEST: md5 / 75 seconds over 30 runs
TEST: rhash / DIGEST: md5 / 84 seconds over 30 runs
TEST: perl / DIGEST: sha1 / 121 seconds over 30 runs
TEST: rhash / DIGEST: sha224 / 140 seconds over 30 runs
TEST: openssl / DIGEST: sha224 / 141 seconds over 30 runs
TEST: rhash / DIGEST: sha256 / 141 seconds over 30 runs
TEST: openssl / DIGEST: sha256 / 169 seconds over 30 runs
TEST: coreutils / DIGEST: sha1 / 177 seconds over 30 runs
TEST: rhash / DIGEST: ripemd160 / 305 seconds over 30 runs
TEST: openssl / DIGEST: ripemd160 / 447 seconds over 30 runs
TEST: perl / DIGEST: sha256 / 637 seconds over 30 runs
TEST: perl / DIGEST: sha224 / 641 seconds over 30 runs
TEST: coreutils / DIGEST: sha256 / 653 seconds over 30 runs
TEST: coreutils / DIGEST: sha224 / 657 seconds over 30 runs
TEST: perl / DIGEST: sha384 / 660 seconds over 30 runs
TEST: perl / DIGEST: sha512 / 661 seconds over 30 runs
TEST: rhash / DIGEST: sha512 / 693 seconds over 30 runs
TEST: openssl / DIGEST: sha384 / 694 seconds over 30 runs
TEST: rhash / DIGEST: sha384 / 695 seconds over 30 runs
TEST: openssl / DIGEST: sha512 / 696 seconds over 30 runs
TEST: coreutils / DIGEST: sha512 / 1513 seconds over 30 runs
TEST: coreutils / DIGEST: sha384 / 1515 seconds over 30 runs

I've marked two entries here:

Originally I used coreutils to calculate a SHA-256 checksum of each file. In the test run above this takes 11 times longer to complete than MD5 would have taken.
Even if I decide against MD5 and choose SHA-1 instead, I'd have to switch to openssl because for some reason coreutils takes almost 3 times longer to complete.

The outcome of these tests means that I'll probably switch to MD5 for my data checksums - this also means that I have to 1) re-generate an MD5 checksum for all files and 2) remove the now-obsolete SHA-256 from all files :-\

Update 1: I omitted cksum and sum from the tests above, as they're not necessarily faster than other checksum tools:

$ n=30
$ for t in sum cksum openssl\ {md4,md5}; do
    START=$(date +%s)
    for a in `seq 1 $n`; do
        $t test.img > /dev/null
    done
    END=$(date +%s)
    echo "TEST: $t / $(echo $END - $START | bc -l) seconds over $n runs"
done | sed 's/ md/_md/' | sort -nk4
TEST: openssl_md4 / 56 seconds over 30 runs
TEST: md5sum / 58 seconds over 30 runs
TEST: sum / 75 seconds over 30 runs
TEST: openssl_md5 / 76 seconds over 30 runs
TEST: cksum / 78 seconds over 30 runs

But again: these tests will have to be repeated on different systems, it could very well be that cksum ~~might really be faster than everything else on another machine~~ - maybe not :-)

Update 2: And it helped indeed: removing the SHA-256 checksum and calculating & attaching the MD5 checksum on 1.5TB of data (88k files) took "only" 31 hours. Which is still a lot, but a lot shorter than the "almost 4 days" we had with SHA-256 :-) Also, the next run won't have to remove the old checksum - it only has to do the verification step. What skewed this number even more was the fact that backups were running on the machine while it was doing the re-calculating stuff, so hopefully the next run will be even shorter.

Update 3: Just to document the cronjob running these regular checks from now on:

0 4 1 */2 *  root  /usr/local/sbin/cron-checksum_file.sh all

This will be run on the first day every second month at 4am.

Update 4: I just had to verify the checksum of two ISO images and did another comparison on the same PowerBook G4 machine:

$ ls -goh *.iso
-rw-r--r-- 1 3.2G Jul 24 16:20 file1.iso
-rw-r--r-- 1  440 Jul 24 19:58 file1.iso.sum
-rw-r--r-- 1 4.2G Jul 24 16:51 file2.iso
-rw-r--r-- 1  440 Jul 24 19:58 file2.iso.sum

$ for a in md5 sha1 sha256 sha512; do echo "$a"; time "$a"sum -c *.iso.sum; echo; done
md5
real    8m17.404s
user    0m56.588s
sys     0m28.444s

sha1
real    11m12.638s
user    3m20.220s
sys     0m28.044s

sha256
real    21m12.057s
user    12m47.092s
sys     0m37.156s

sha512 
real    40m56.836s
user    29m55.444s
sys     0m39.684s

So, each of the chosen "stronger" algorithm bascially doubles the execution time of a "weaker" one. Again, md5 is more than enough for our use case here.

A kingdom for a music player, pt. II

Posted by christian on Saturday, July 4. 2015

For a long time I've looked for a better music player for the desktop. After a while I got tired of how slow graphical music players got when running on this ~20k songs library. At the end I returned to the command line and couldn't be happier.

My most-used command for playing music is now:

$ find /mnt/nfs/media/MP3 ! -path "*/MP3/Hoerspiele/*" -type f | \
    sort -R | while read a; do mpg123 -v "$a"; done

Unknown files (e.g. cover pictures) are just skipped and I can even ^C to skip a song or pause with ^Z. That's all I really wanted :-) I'ts even possible to skip the intro for a certain radio show:

$ mpg123 -k 2000 [...]

On MacOS X, sort doesn't support "-R" and we can use Perl for this:

$ find /mnt/nfs/media/MP3 -type f | tail -30 | \
    perl -MList::Util=shuffle -e 'print shuffle(<STDIN>);' | \
    while read a; do afplay -d "$a"; done

VirtualBox: switching to Host-only networking

Posted by christian on Tuesday, June 9. 2015

There are many ways to provide network connectivity to VirtualBox guests. The most common ones, in short:

Network Address Translation (NAT): this is the default mode. VirtualBox will act as a DHCP server, providing guests with internal addresses and connectivity to the outside world. But no routing is provided and thus guests cannot be reached from the outside.
Bridged Networking: a virtual NIC is bridged to a physical NIC on the host host, guests have full network connectivity and can be reached from the outside world. However, an external DHCP and DNS service may be needed.
Internal networking: similar to bridged networking, but only the host and guests on the same host will be able to connect to the guest.
Host-only networking: a hybrid between bridged and internal networking. Guests can connect to each other, but no real NIC has to be present on the host. DHCP / DNS can be provided by VirtualBox or externally.

For a long time I've just used Bridged networking - it was easy to setup and worked like a charm. Of course, this incurred some administrative overhead: for every VM a DNS name had to be registered. At home, dnsmasq is running on the router and will be able to provide DHCP & DNS to the guests. With static names and IP addresses for the guests, a simple mapping scheme had to be implemented:

When a new VM gets created, modify its MAC address to correspond to a certain range and match the last octet to its (future) IP address. E.g. for a guest with the future IP address of 192.168.0.31, set its MAC address to 08:00:27:e2:81:31.
Add both entries to /etc/ethers and the IP address / hostname mapping to /etc/hosts.

This worked very well for a long time but was always dependent on the dnsmasq installation being around. When connected to a different network, the guests will not be able to rely on the DHCP & DNS setup. Also, the physical NIC the guest network is bridged to may not be online. Think of laptops, sometimes connecting via Wifi, sometimes via ethernet.

And so I decided to take a look at Host-only networking. First we have to create (and configure) the host-only interface:

$ vboxmanage hostonlyif create 
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interface 'vboxnet0' was successfully created
$ vboxmanage hostonlyif ipconfig vboxnet0 --ip 192.168.56.1

Disable any VirtualBox DHCP servers:

$ vboxmanage list dhcpservers
[...]
$ vboxmanage dhcpserver remove --netname NetworkName
$ vboxmanage list hostonlyifs
Name:            vboxnet0
GUID:            607cb20b-9848-4313-b522-3ccd6cd01be9
DHCP:            Disabled
IPAddress:       192.168.56.1
NetworkMask:     255.255.255.0
IPV6Address:     fe80:0000:0000:0000:0800:27ff:fe00:0000
IPV6NetworkMaskPrefixLength: 64
HardwareAddress: 0a:00:27:00:00:00
MediumType:      Ethernet
Status:          Up
VBoxNetworkName: HostInterfaceNetworking-vboxnet0

With the virtual NIC in place, we have to configure the guests:

$ vboxmanage showvminfo vm1 | grep NIC\ 1
NIC 1:           MAC: 080027e28131, Attachment: Bridged Interface 'wlan0', ...

$ vboxmanage modifyvm vm1 --nic1 hostonly --hostonlyadapter1 vboxnet0
$ vboxmanage showvminfo vm1 | grep NIC\ 1
NIC 1:           MAC: 080027e28131, Attachment: Host-only Interface 'vboxnet0', ...

Although we could use the internal DHCP server from VirtualBox, we would not be able to provide our elaborate mapping scheme. Let's setup a small, local dnsmasq installation:

$ sudo apt-get install dnsmasq-base
$ tail -n2 /etc/{ethers,hosts}
==> /etc/ethers <==
08:00:27:e2:81:30       vm0
08:00:27:e2:81:31       vm1

==> /etc/hosts <==
192.168.56.30  vm0
192.168.56.31  vm1

$ grep ^[a-z] dnsmasq.conf 
dhcp-authoritative
read-ethers
dhcp-leasefile=/var/run/dnsmasq/dhcp.leases
dhcp-range=192.168.56.30,192.168.56.100,255.255.255.0,12h
port=53
domain=vbox
expand-hosts

Note: we're not using dnsmasq as a DNS server on our host. Our virtual machines only need to be reachable from localhost anyway and we'll just use /etc/hosts. However, we cannot disable the DNS function in dnsmasq (by setting port=0) because then dnsmasq won't send DHCP offers for the matching MAC address. I was about to use port=2053 to allow dnsmasq to run as a non-root user, but of course dnsmasq still needs to bind to port 67 to act as a DHCP server. Also, with port set to any other port than 53, guests would not be able to refer to other guests by its name, because resolv.conf doesn't understand port numbers:

vm1$ dig vm0 -p 2053 @192.168.56.1 | grep ^[a-z]
vm0.                  0       IN      A       192.168.56.30

Almost there. We can now startup the VM and it should get its 192.168.56.30 assigned via DHCP. We should be able to connect to the guest, but we don't seem to be able to reach any other destination except the local network from insie the guest. For that, we have to enable IP forwarding in the host.

Linux host

# iptables -A FORWARD -i vboxnet0 -s 192.168.56.0/24 -m conntrack --ctstate NEW -j ACCEPT
# iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# iptables -A POSTROUTING -t nat -j MASQUERADE
# sysctl -qw net.ipv4.ip_forward=1

Now we should be able to connect to the guest with a static DNS name or IP address and we should be able to connect to the outside world from within the guest.

MacOS X host

On MacOS X, the magic commands would be:

# sysctl net.inet.ip.forwarding=1 net.inet.ip.fw.enable=1
net.inet.ip.forwarding: 0 -> 1
net.inet.ip.fw.enable: 1 -> 1

# grep ^net /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet.ip.fw.enable=1

Enable NAT through pf.conf(5):

# grep -B1 nat\  /etc/pf.conf 
rdr-anchor "com.apple/*"
nat on en1 from 192.168.56.0/24 to any -> (en1)

# pfctl -f /etc/pf.conf
# pfctl -e

Note: the nat entry must follow the rdr-anchor entry, it cannot be just appended to the end of the file.

Homebrew: GitHub API rate limit exceeded

Posted by christian on Monday, June 8. 2015

I'm a big MacPorts fanboy but since Homebrew is all the craze for a few years now, I tried to give it another look.

$ mkdir homebrew && homebrew
$ curl -L https://github.com/Homebrew/homebrew/tarball/master | \
         tar --strip 1 -xzvf -
$ sudo mv homebrew /opt/homebrew && sudo chown -R root:wheel /opt/homebrew
$ sudo brew update
Initialized empty Git repository in /opt/homebrew
[...]

OK, so far - so good. Let's search for some packages, shall we?

$ brew search foo
[...]

$ brew search bar
[...]

$ brew search ssh
autossh      git-ssh      libssh       mpssh        pssh         ssh-copy-id  sshrc        sshuttle     tmux-cssh    zssh
csshx        gssh         libssh2      mussh        rssh         sshguard     sshtrix      stormssh     vassh
homebrew/fuse/sshfs                 homebrew/php/php54-ssh2             homebrew/php/php56-ssh2             Caskroom/cask/ssh-tunnel-manager
homebrew/php/php53-ssh2             homebrew/php/php55-ssh2             Caskroom/cask/bassshapes            Caskroom/cask/sshfs
Error: GitHub API rate limit exceeded for 172.16.30.4 (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)
Try again in 59 minutes 32 seconds, or create an personal access token:
  https://github.com/settings/tokens
and then set it as HOMEBREW_GITHUB_API_TOKEN

Wait, wat? brew is asking the remote repo if a package is available? I've just run brew update:

   update   - Fetch the newest version of Homebrew and all formulae 
              from GitHub using git(1).

But indeed, the search command will perform an online search:

   search  - [...] The search for text is extended online to some popular taps.

Fortunately one can set HOMEBREW_NO_GITHUB_API=1 to stop this madness.

mod_cheeseburger

Posted by christian on Thursday, May 21. 2015

This is just awesome:

$ sudo apt-get install ttf-ancient-fonts
$ export PS1="\u@\h🍔  "
dummy@acorn🍔

The font was probably used to render ancient symbols, but somehow managed to implement U+1F354 too:

$ printf 🍔 | od -x
0000000 9ff0 948d
0000004

IMAPExpire

Posted by christian on Saturday, May 9. 2015

Being an Alpine user, I have several rules and filters in place, especially for all those countless mailing lists I'm subscribed to. Specifically, I only want the mails of, say the last 3 weeks kept in certain mail folders, but I don't need the whole archive of lkml to be stored on my disk. To do that, there's a rule in my .pinerc to implement that:

patterns-filters2=LIT:pattern="/NICK=purge_old-threads/AGE=(21,INF)/FLDTYPE=SPEC/FOLDER={localhost\/user=dummy\/tls\/novalidate-cert}INBOX.Misc.lkml,{localhost\/user=dummy\/tls\/novalidate-cert}INBOX.Misc.bugtraq,[...]" action="/FILTER=1"

This is just an excerpt but maybe you get the idea: the filter is called purge_old-threads and it deletes mails older than 21 days. So far, so good.

But the filter stanza is actually quite long and hard to maintain and only gets triggered when I actually change into the mail folder and look at its contents. Alpine doesn't do any automagic housekeeping, so when I don't read lkml for a few weeks, the mailfolder grows and the incoming flow of mails just pile up. Then, when I get around to read lkml again, the filter kicks in and has to crawl through ~20k messages and delete all the older ones, which might take a while to complete.

So, I wanted to know if there's a way to do this without these rather cryptic Alpine rule sets. During my search I came across IMAPExpire. It's a nice Perl script that uses IMAP::Client, which looks a lot like Net::IMAP::Client (and has been packaged for Debian too), except it's not - we really need IMAP::Client here and we'll try to install it from CPAN:

$ env | grep PERL
PERL5LIB=/home/dummy/.perl5/lib/perl5
PERL_MB_OPT=--install_base "/home/dummy/.perl5"
PERL_LOCAL_LIB_ROOT=/home/dummy/.perl5
PERL_MM_OPT=INSTALL_BASE=/home/dummy/.perl5

$ cpan
cpan[1]> install IMAP::Client
[...]
IMAP::Client is up to date (0.13).

cpan[4]> i IMAP::Client 
Module id = IMAP::Client
    CPAN_USERID  CONTEB (Brenden Conte <conteb@cpan.org>)
    CPAN_VERSION 0.13
    CPAN_FILE    C/CO/CONTEB/IMAP-Client-0.13.tar.gz
    UPLOAD_DATE  2006-09-28
    MANPAGE      IMAP::Client - Advanced manipulation of IMAP services w/ referral support
    INST_FILE    /home/dummy/.perl5/lib/perl5/IMAP/Client.pm
    INST_VERSION 0.13

With that in place, imapexpire.pl should work now. Don't forget the --test switch when trying this out:

$ cat > ~/.imap-pw
s3cr3tpassw0rd
^D

$ ./imapexpire.pl --test --user dummy --passfile ~/.imap-pw --age 21 \
        --debug 9 --folders INBOX.Misc.lkml
<< * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN] Dovecot ready.
> 0001 NOOP
<< 0001 OK NOOP completed.
> 0002 LOGIN dummy s3cr3tpassw0rd
<< 0002 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in
> 0003 LIST "" "INBOX.Misc.lkml"
<< * LIST (\HasNoChildren) "." INBOX.Misc.lkml
<< 0003 OK List completed.
TEST  : You're running in test mode, so the deletions wont actually take place
ACTION: Delete mail which arrived before 20-Apr-2015 from: INBOX.Misc.lkml
>> 0004 SELECT "INBOX.Misc.lkml"
<< * FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk)
<< * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk \*)] Flags permitted.
<< * 36 EXISTS
<< * 14 RECENT
<< * OK [UNSEEN 1] First unseen.
<< * OK [UIDVALIDITY 1204617147] UIDs valid
<< * OK [UIDNEXT 6464] Predicted next UID
<< * OK [HIGHESTMODSEQ 855] Highest
<< 0004 OK [READ-WRITE] Select completed (0.370 secs).
>> 0005 UID SEARCH BEFORE 3-May-2015
<< * SEARCH 6428 6429 6430 6431
<< 0005 OK Search completed (0.034 secs).
Deleting 4 messages from INBOX.Misc.lkml

This should be put into a script of course, running over every mailing list folder I'm subscribed to. If we're confident enough that no real email folder will be purged (and our ~~backups~~ restores are working), a cronjob could be created too :-)

httpdiff

Posted by christian on Thursday, April 2. 2015

Inspired by httpdiff and httpdiff.com (in combination with colordiff):

$ diff -u  <(curl -sI  http://nerdbynature.de) \
           <(curl -sIL http://nerdbynature.de/s9y/) | colordiff 
--- /dev/fd/63  2015-04-02 16:28:18.000000000 -0700
+++ /dev/fd/62  2015-04-02 16:28:18.000000000 -0700
@@ -1,6 +1,12 @@
-HTTP/1.1 301 Moved Permanently
+HTTP/1.1 200 OK
 Date: Thu, 02 Apr 2015 23:28:18 GMT
 Server: Apache
-Location: http://nerdbynature.de/s9y/
-Content-Type: text/html; charset=iso-8859-1
+X-Powered-By: PHP/5.4.38
+X-Session-Reinit: true
+X-Blog: Serendipity
+Cache-Control: private, pre-check=0, post-check=0, max-age=0
+Expires: 0
+Pragma: no-cache
+Set-Cookie: s9y_54ff07474dc18d0b1f7=e1; path=/
+Content-Type: text/html; charset=UTF-8

Mon	Tue	Wed	Thu	Fri	Sat	Sun
← Back	October '17
						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30	31