s9y testdrive

Monday, May 13. 2013

GNU/screen & UTF-8

In a GNU/screen session on this box, this happens sometimes:

$ rm foo
rm: remove regular empty file ?foo??

Why does it show a question mark instead of an apostrophe?

Running the same command with a different or no LANG setting seems to help:

$ LANG=C rm foo
rm: remove regular empty file 'foo'?

$ LANG="" rm foo
rm: remove regular empty file 'foo'?

$ echo $LANG
en_US.UTF-8

I usually start my screen sessions with -U:

  > Run screen in UTF-8 mode. This option tells screen that your terminal sends and 
  > understands UTF-8 encoded characters. It also sets the default encoding for new
  > windows to ‘utf8’.

So, we have LANG set to en_US.UTF-8 and screen(1) started in UTF-8 mode and still have character issues? Weird. But then I remembered: I use -U only when starting the screen session, not when resuming the session:

% screen -US MySession

$ echo $LANG; rm foo
en_US.UTF-8
rm: remove regular empty file 'foo'?

Upon resuming the session with "screen -dr" the question marks were back again. Resuming with -drU fixes that.

Posted by christian in bits at 22:08 | Comments (0) | Trackbacks (0)

Wednesday, April 17. 2013

whois: Invalid charset for response

As if MacOS didn't have enough charset problems, here's another one:

$ /usr/bin/whois denic.de
% Error: 55000000013 Invalid charset for response

Although the problem has been reported to DENIC years ago, they still send out UTF-8 data if the handles contain e.g. umlauts.

But why can't the MacOS version of whois(1) handle UTF-8 data? A quick look on the binary reveals:

$ strings /usr/bin/whois
[...]
de.whois-servers.net
-T dn,ace -C US-ASCII %s

So, the -T dn,ace -C US-ASCII seems to be hardcoded, as we can see in the source:

#define GERMNICHOST	"de.whois-servers.net"
[...]
if (strcmp(hostname, GERMNICHOST) == 0) {
		fprintf(sfo, "-T dn,ace -C US-ASCII %s\r\n", query);

There's no -C switch to pass to whois(1) to change this behaviour. Experimenting with LC_ALL environment variables did not help either.

What did help was to pass options directly to their whois server

$ /usr/bin/whois -h whois.denic.de -- "-T dn,ace denic.de"

This way, -C US-ASCII is skipped and the (UTF-8) output can be displayed just fine.

Of course, we could also install whois from Macports, it seems to handle UTF-8 data just fine (although it had a similar problem years ago):

$ sudo port install whois

$ /opt/local/bin/whois denic.de | file -
/dev/stdin: UTF-8 Unicode text

$ /opt/local/bin/whois denic.de
[...]
[Tech-C]
Type: PERSON
Name: Business Services
Organisation: DENIC eG
Address: Kaiserstraße 75-77

Posted by christian in bits at 15:06 | Comments (0) | Trackbacks (0)

Monday, April 15. 2013

Mozilla defaults

Every now and then I come across a new machine I've never logged in to before and starting Firefox for the first time. And then I always have to make my way through oh so many preference knobs and about:config entries just to get it into a usuable state.

So, while I knew the configuration could be tweaked via user.js, I never got around actually creating this file and adding some sensible defaults to it. Well, that's been done now. And with site-wide defaults, it's even more fun!

In short:

Create local-settings.js in defaults/pref/ underneath the Firefox installation directory.
Create firefox.cfg in the Firefox installation directory.
Create user.js inside your profile directory and fill it with some sensible defaults.

Of course, we can skip the first two steps and just fill user.js with the contents of firefox.cfg but we have to replace defaultPref and lockPref entries with user_pref.

Posted by christian in bits at 15:42 | Comments (0) | Trackbacks (0)

Wednesday, April 10. 2013

You don't exist, go away!

After opening my laptop today, the first thing was of course to login to various systems, as I usually do. But this time I couldn't and instead was greeted with:

  $ ssh foobar
  You don't exist, go away!

At first I thought the remote system was at fault, but ssh would print the same message for every other system I was trying to login to. This had been reported by others already and after just clicking those links I tried again and this time ssh was able to login w/o a problem. So, while this was only a temporary issue, let's recap and dig into that once again.

Apparently, the error message is generated by the client:

$ strings `which ssh` | grep away
You don't exist, go away!

It's right there in ssh.c:

pw = getpwuid(original_real_uid);
	if (!pw) {
		logit("You don't exist, go away!");
		exit(255);
	}

So, the call to getpwuid() failed. Now, why would it do that? In the manpage it says:

   These functions obtain information from DirectoryService(8),
   including records in /etc/passwd

And /etc/passwd was there all the time (hah!), so maybe DirectoryService(8) screwed up? Let's see if we find something in /var/log/system.log:

14:59:57 coreservicesd[54]: _scserver_ServerCheckin: client uid validation failure; getpwuid(502) == NULL
14:59:58 loginwindow[376]: resume called when there was already a timer
14:59:58 coreservicesd[54]: _scserver_ServerCheckin: client uid validation failure; getpwuid(502) == NULL

There it is. Now, restarting coreservicesd (or securityd) would have helped, but by now the system was fully waken up from sleep and getpwuid() was able to do what it does - and ssh was working again, too. If it happens again and won't recover by itself - we know what to do :-)

Posted by christian in bits at 15:42 | Comments (0) | Trackbacks (0)

Thursday, April 4. 2013

Zero padding shell snippets

I was looking for a way to zero-pad a number sequence in bash. While the internet was helpful as usual, one particular post had lots of examples in its comments, very neat stuff.

Of course, with so many differeant approaches, this called for a benchmark! :-)

$ time bash padding_examples.sh bash41 1000000 > /dev/null
real    7m38.238s
user    3m7.056s
sys     0m7.884s

$ time   sh padding_examples.sh printf 1000000 > /dev/null
real    1m39.314s
user    0m41.244s
sys     0m2.064s

$ time   sh padding_examples.sh    seq 1000000 > /dev/null
real    0m10.883s
user    0m5.016s
sys     0m0.040s

So, seq(1) is of course the fastest - if it's not installed, use printf.

Update: with bash-4.0, the following is also possible:

$ time echo {01..1000000} > /dev/null
real    0m38.852s
user    0m14.948s
sys     0m0.260s

However, this will consume a lot of memory:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
23468 dummy     25   5  189m 186m 1380 R  43.1 14.9  0:28.48  bash

Posted by christian in benchmarks at 14:16 | Comments (0) | Trackbacks (0)

Monday, April 1. 2013

rTorrent: Hash check on download completion found bad chunks, consider using "safe_sync"

rTorrent would not complete a download and print the following:

* file.foo
* [OPEN]  506.3 /  785.3 MB Rate: 0.0 / 0.0 KB Uploaded: 248.8 MB  [T R: 0.49]
* Inactive: Hash check on download completion found bad chunks, consider using "safe_sync".

Initiating a check of the torrent's hash (^R) succeeded and then rTorrent tried to download the remaining part of the file - only to fail again, printing the same message :-\

Setting safe_sync (which got renamed to pieces.sync.always_safe) did not help. There's a longish and old ticket that got closed as "invalid". While this might have been the Right Thing^TM to do (see the LKML discussion related to that issue) there was another hint: decreasing max_open_files (which got renamed to network.max_open_files) to a lower value, say 64. Needless to say that this didn't help either, so maybe there's something else going on here.

strace might be able to shed some light on this, so let's give it a try. After several hours (and a good night's sleep) a 2GB strace(1) logfile was waiting to be analyzed. I only needed the part of the logfile up to where the error message occured first - and from there on upwards I'd search for negative exitcodes, as they will denote some kind of error. And lo and behold, there it was:

    mmap2(NULL, 292864, PROT_READ, MAP_SHARED, 13, 0x31100) = -1 ENOMEM (Cannot allocate memory)

Before we continue to find out why we failed, let's see how much memory we tried to allocate here. mmap2() is supposed to "map files or devices into memory":

    void *mmap2(void *addr, size_t length, int prot, int flags, int fd, off_t pgoffset);

In our case, size_t is 292864 (bytes) with an offset of 0x31100. However, this offset is in "pagesize units". So, what is our page size?

$ uname -rm && getconf PAGE_SIZE
3.9.0-rc4 ppc
4096

Let's calculate the size rTorrent was trying to mmap2() here:

$ bc
obase=16
4096
1000

ibase=16
obase=A
1000 * 31100                <== PAGE_SIZE * 0x31100
823132160

ibase=A
823132160 + 292864          <== add size_t
823425024

So, 823425024 bytes are 786 MB - we have 1.2 GB RAM on this machine and some swapspace too. Not too much, but this box mmap()'ed larger files than this before - why would mmap2() fail with ENOMEM here?

Maybe this "reduce max_open_files" hint tipped me off but now I remembered playing around with ulimit(3) a while ago. So maybe these ulimits were too tight?

And they were! Setting ulimit -v ("per process address space") to a larger value made the ENOMEM go away and rTorrent was able to complete the download:

$ ls -lgo file.foo
-rw-r----- 1 823425024 Apr  1 11:38 file.foo

...with the exact same size mmap2() was trying to allocate. Btw, we could've checked the file before rTorrent completed the download, because it's an sparse file anyway.

Update: while raising the ulimit(3) certainly resolved the ENOMEM issue, the torrent would still not complete successfully. Turns out it was a kernel bug after all, but it was resolved rather quickly.

Posted by christian in bits at 13:19 | Comments (0) | Trackbacks (0)

Thursday, March 7. 2013

Linux symlink restrictions

This happened:

$ uname -sr && id
Linux 3.8-trunk-amd64
uid=1000(dummy) gid=1000(dummy) groups=1000(dummy)

$ ln -s /usr/local/src /tmp/foo
$ ^D

# id
uid=0(root) gid=0(root) groups=0(root)

# ls -l /tmp/foo
lrwxrwxrwx 1 dummy dummy 14 Mar  7 21:30 /tmp/foo -> /usr/local/src

# ls -l /tmp/foo/
ls: cannot access /tmp/foo/: Permission denied

Huh? root is not allowed to follow a user's symlink? Turns out this is a feature now:

  > The solution is to permit symlinks to only be followed when outside
  > a sticky world-writable directory, or when the uid of the symlink and
  > follower match, or when the directory owner matches the symlink's owner.

And since /tmp usually has the sticky bit set, the access to /tmp/foo is denied. This access restriction is accompanied by audit messages:

 type=1702 audit(1362720689.110:28): op=follow_link action=denied \
           pid=22758 comm="ls" path="/tmp/foo" dev="sda1" ino=252

This can be tweaked via sysctl:

# sysctl -w fs.protected_symlinks=0
fs.protected_symlinks = 0

# ls -l /tmp/foo/
total 12092
-rw-r--r-- 1 root  staff    6794 Jan  9 22:59 bar
[...]

# sysctl -a | grep protected
fs.protected_hardlinks = 1
fs.protected_symlinks = 0

...but the default (fs.protected_symlinks=1) is kinda neat, now that I know about it :-)

Posted by christian in bits at 21:26 | Comments (0) | Trackbacks (0)

Sunday, February 24. 2013

A kingdom for a music player!

The quest for ~~the~~ my perfect music player continues. I won't go into the details, why iTunes sucks so much, others have done this already.

A long time ago, I started with Songbird. They were crazy enough to take Mozilla's XULRunner runtime and turned it into a music player. But it had the browser still built in and came with lots of features, making this thing quite bloated. Maintaining ~20k songs became very unpleasant, CPU usage was definitely too high for a music player.

Then there's Cog, which is neat, but was more of an ad-hoc player and not meant to maintain a music library.

Later on, Unixhaus suggested to use Clementine. Crossplatform, open-source, nice interface and a Nyanalyzer - what more could one ask for? Sticked with it for quite a while, but there were still 20k songs to manage and cpu usage was rather high. 15% CPU usage on a MacBook Pro just for playing music? And often enough CPU usage spiked for a few minutes, doing something and then dropped to 15% again. Very annoying.

OK, what else is out there? Winamp! No, seriously, there's Winamp for Mac. Yeah, I tried it, I admit. Again with the 20k songs, Winamp performed quite well...but: it feels kinda creepy having Winamp on a Mac. Try it and ~~see~~ feel for yourself :-\

Another contender was Ecoute. It's has a 15-day-free-trial version, then it's US$8 in the Mac App Store. It's got a very nice UI, but after a while it kept freezing again and again and became unusable. Good thing they offered a trial version!

I haven't tried Enqueue yet. It's US$ 10 in the Mac App Store but unfortunately there's no trial version. It looks a lot like iTunes, exactly what I'm trying to avoid.

Someone mentioned Vox - their tagline is "The Lightweight Music App for Mac OS X". OK, I haven't tried it yet, but from looking at their screenshots they seem to have a different understanding of lightweight.

So, what now? Clementine is the player I used most of the time, but gave up some months ago, trying out other alternatives.

A few days ago I thought "Hm, I wonder what happed to Songbird?". They are at version 2.x now and I felt like giving it another shot. They still have this browser stuff underneath, but the application feels a lot faster now, they cut down on bloated plugins in the initial installation (though there are plenty addons to choose from) and for now Songbird seems to have made it and it's very usable, for me. And, like Clementine, it's cross-platform and open-source (sans the Nyanalyzer) and for now it's working just fine.

I could end this entry with "Stay tuned for updates" but I hope that I won't need to update this post and that Songbird continues to stay usable.

Posted by christian in bits at 02:53 | Comments (0) | Trackbacks (0)

Saturday, February 23. 2013

Remove U3 from an USB flash disk

This "SanDisk Cruzer" USB flash drive has this nasty U3 thingy included. Every time it connects with a host, this U3 partition reappears and gets in the way, trying to do super smart stuff.

So, how to get rid if this ~~malware~~ feature? Overwriting the whole device did not help. There's a removal tool...for Windows. There's a removal tool for Macs too, but only up to MacOS 10.6 :-\

For Linux, there's u3_tool. This hasn't been updated in a while but let's hope we won't need tools like this in the future. Here's a short description to get u3_tool going:

 svn co https://u3-tool.svn.sourceforge.net/svnroot/u3-tool/trunk u3-tool-svn
 cd trunk u3-tool-svn
 aclocal
 autoconf
 automake --add-missing
 ./configure --prefix=/opt/u3-tool
 make && sudo make install

After the build completed, let's use it:

$ /opt/u3-tool/sbin/u3-tool -i -v /dev/sdb
Total device size:   1.88 GB (2017525760 bytes)
CD size:             16.00 MB (16777216 bytes)
Data partition size: 1.86 GB (2000748544 bytes)

$ /opt/u3-tool/sbin/u3-tool -p 0 -v /dev/sdb

WARNING: Loading a new cd image causes the whole device to be wiped. This INCLUDES
 the data partition.
I repeat: ANY EXISTING DATA WILL BE LOST!

Are you sure you want to continue? [yn] y

$ echo $?
0

And it worked :-) Good riddance, U3!

Posted by christian in bits at 01:28 | Comments (0) | Trackbacks (0)

Monday, February 18. 2013

Repair broken MP3 files

Some application (which shall remain nameless) complained about "invalid MP3 formatted files", but gave no clue about what exactly was "invalid". The files played just fine, their ID3 tag were displayed too, so this application appeared to be overly picky about these files. Let's have a closer look:

$ file foo.mp3
foo.mp3: Audio file with ID3 version 2.3.0, contains: MPEG ADTS, layer II, v1, 192 kbps, 44.1 kHz, Stereo

Aha! Although the file was named .mp3 it really was an mp2 file. Of course, most program can play MPEG-2 just fine, but this application refused to do so.

The ever-so-faithful lame was quick to help:

$ mv foo.{mp3,mp2}
$ lame --mp2input foo.mp2 foo.mp3

Also, the ID3 version said "2.3.0", which is perfectly valid but maybe there was something else wrong with these files so I needed some magic program to check (and repair) this file's ID3 tag. mid3iconv (from python-mutagen) is supposed to do just that:

$ mid3iconv -d foo.mp3
Updating foo.mp3
TPE1=Snafu
TDRC=2002
TIT2=FooBar
TRCK=1
TALB=Baz
TCON=Electronic

Now our file looked like this:

foo.mp3: Audio file with ID3 version 2.4.0, contains: MPEG ADTS, layer III, v1, 128 kbps, 44.1 kHz, JntStereo

...and the application was happy to process this file :-)

Posted by christian in bits at 23:41 | Comments (0) | Trackbacks (0)

Saturday, February 16. 2013

smartctl & external disks

When monitoring S.M.A.R.T. values of disks in a Unix system, smartmontools is usually the way to go.

Unfortunately monitoring external disk enclosures may be difficult or not possible at all. I haven't seen a firewire enclosure that supported SMART commands yet.

USB enclosures tend to work, but I noticed that the scheduled self-tests would not complete. For example, for the following disk a short (S) self-test is scheduled every day at 3am and a long (L) self-test every saturday at 6am:

$ cat /etc/smartd.conf
/dev/disk/by-id/scsi-SSAMSUNG_HD103UJ -d sat -a -o on -S on \
                           -s (S/../.././03|L/../../6/06) -I 190 -I 194 -W 5

But so far every test did not complete:

$ smartctl -d sat -l selftest /dev/sdb
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 0
Warning: ATA Specification requires self-test log structure revision number = 1
Num  Test_Description    Status           Remaining  LifeTime(hours)
# 1  Extended offline    Aborted by host      00%     25980
# 2  Short offline       Aborted by host      00%     25973
[...]

Someone else had a similar problem and suggested to run "smartctl -a /dev/disk... every few seconds while the self-tests are in progress, so that the disk would not shut down. Preliminary tests showed that this helped in my case as well.

From now on the self-test schedule in smartd.conf will be accompanied by some cronjob doing just this:

while smartctl -d sat -l selftest /dev/sdb 2>&1 | \
               grep -q "Self-test routine in progress"; do 
      sleep 30
done

The crontab(5) entry for the schedule above:

# m h  dom mon  dow   command
0   3    \*   \*    \*   script /dev/disk/by-id/scsi-SSAMSUNG_HD103UJ
0   6    \*   \*    6   script /dev/disk/by-id/scsi-SSAMSUNG_HD103UJ

We might add some fuzzyness to this "script" of course, so that it will work when the actual self-tests starts a bit late.

Posted by christian in bits at 17:06 | Comments (0) | Trackbacks (0)

Thursday, January 24. 2013

SSH/HTTPS multiplexer

Hm, this nmap scan looked funny:

PORT    STATE SERVICE VERSION
22/tcp  open  ssh     OpenSSH 5.2 (protocol 2.0)
80/tcp  open  http    Gatling httpd 0.13
443/tcp open  ssh     OpenSSH 5.2 (protocol 2.0)

SSH listening on :443, yet the site was serving a website there? Looking around a bit I came across a few SSH/HTTP/HTTPS multiplexers. There are even binary packages out there for a few distributions, nice! So, how is it done?

ssh-https

When using ssh-https.c, the ports are hardcoded:

$ grep execl ssh-https.c
                execl("/bin/nc", "/bin/nc", "localhost", "8443", NULL);
                execl("/bin/nc", "/bin/nc", "localhost", "22", NULL);

$ gcc -o ssh-https ssh-https.c
$ mv ssh-https /usr/local/sbin/

SSH will continue to listen on :22, the webserver will have to listen on :8443 and ssh-https will listen on :443:

$ grep ssh-https /etc/inetd.conf
https   stream  tcp  nowait  nobody  /usr/sbin/tcpd /usr/local/sbin/ssh-https

sslh

sslh is a bit more flexible, as ports can be passed on the command line:

$ grep sslh /etc/inetd.conf
https   stream  tcp  nowait  sslh  /usr/sbin/tcpd /usr/sbin/sslh \
       --listen 10.0.0.23:443 --inetd --ssh localhost:22 --ssl localhost:8443

In any case, we should now have 3 listening ports:

$ netstat -anptu | grep LISTEN
[...]
tcp    0      0 0.0.0.0:22      0.0.0.0:\*   LISTEN    2211/dropbear
tcp    0      0 0.0.0.0:443     0.0.0.0:\*   LISTEN    6510/inetd
tcp    0      0 0.0.0.0:8443    0.0.0.0:\*   LISTEN    6012/lighttpd

And it's even working :-)

$ ssh-keyscan -p 443 10.0.0.23
# foo SSH-2.0-dropbear_2012.55

$ wget -qO- https://10.0.0.23/
Hello, world :-)

Posted by christian in bits at 18:41 | Comments (0) | Trackbacks (0)

Thursday, January 17. 2013

MacOS X disk I/O

While we're at it, some more "benchmarks". On a MacBook Pro with a Crucial m4 SSD inside:

$ dd if=/dev/rdisk0 bs=1024k count=2048 2>/dev/null | pv > /dev/null
   2GiB 0:00:09 [ 215MiB/s]

Oddly enough, the blockdevice of the same disk had much worse performance:

$ ls -lgo /dev/{r,}disk0
brw-r-----  1    14,   0 Dec 14 08:13 /dev/disk0
crw-r-----  1    14,   0 Dec 14 08:13 /dev/rdisk0

$ dd if=/dev/disk0 bs=1024k count=2048 2>/dev/null | pv > /dev/null
   2GiB 0:01:06 [30.8MiB/s]

On the same machine, Debian/wheezy was running in a VirtualBox virtual machine, we're still getting half the performance:

vm$ dd if=/dev/sda bs=1024k count=2048 2>/dev/null | pv > /dev/null
   2GB 0:00:18 [ 109MB/s]

And inside this virtual machine, another Debian/wheezy installation was running as a Xen DomU virtual machine, performance halves again:

vm|domU$ dd if=/dev/xvda1 bs=1024k count=2048 2>/dev/null | pv > /dev/null
   2GB 0:00:29 [69.4MB/s]

Posted by christian in benchmarks at 02:38 | Comments (0) | Trackbacks (0)

Wednesday, January 16. 2013

dm-crypt benchmarks

While setting up encrypted swap on yet another Linux machine, I wondered what the "best" crypto algorithm would be.

There are plenty of combinations of ciphers, modes, hash algorithms and keysizes to chose from with cryptsetup(8), let's see if we can find a fast, yet sufficiently "secure" one.

Before testing these combinations I wanted to find out which combinations were actually possible. E.g. setting up a dm-crypt device with aes-cbc-plain with a keysize of 128 or 256 bit would be possible - but any larger keysize was rejected. There were many "invalid" combinations, for reasons rooted deeply in their mathematical properties. So, let's find out these valid combinations then:

[...]
cryptsetup -c $CIPHER -s $KEYSIZE -d /dev/urandom create test /dev/sdc 2>/dev/null
if [ $? = 0 ]; then
      echo "Valid combination: cipher $CIPHER size $KEYSIZE"
else
      echo "Invalid combination: cipher $CIPHER - size $KEYSIZE"
fi

After quite some iterations over a predefined set of combinations (12 ciphers, 7 modes, 14 hashing algorithms, 5 keysizes), there were 1125 valid combinations left. Yeah, testing took a while :-)

Now we wanted to see which combinations performed "best". As stated above, the usecase was a blockdevice for encrypted swap - so "fast, yet pretty secure" were the criteria to look for. As a (very) simple test, the following should be done for each newly set up crypto block device:

while [ $i -lt 30 ]; do
      # Empty all caches, including filesystem buffers  
      sysctl -qw vm.drop_caches=3
      dd if=/dev/mapper/test of=/dev/null bs=1M 2>/dev/null
      i=$((i+1))
done

The results, summarized:

cipher_null* is the fastest - but it's absolutely insecure, because...well, it's a NULL cipher :-)
Interestingly, PCBC mode is sometimes very slow. As per the Wikipedia article, this mode is not very common anyway, so we'll not choose this one.
As expected, Twofish is very fast, along with AES and Blowfish.

Ruling out some of the obvious combinations and omitting some of the "exotic" algorithms, these are my winners:

$ egrep -v 'cipher_null|-(ecb|lrw|pcbc)-|-plain|md[45]|rmd|tgr|crc32|sha1' results | \
    grep ' / 256' | sort -nk5  | head -10
twofish-ctr-essiv:sha256 / 256 : 66            <== 66 seconds for 30 runs. Lower is better.
twofish-cbc-essiv:sha256 / 256 : 69
twofish-xts-essiv:sha256 / 256 : 73
aes-xts-essiv:sha256 / 256 : 79
blowfish-cbc-essiv:sha256 / 256 : 81
blowfish-ctr-essiv:sha256 / 256 : 86
aes-ctr-essiv:sha256 / 256 : 90
aes-cbc-essiv:sha256 / 256 : 91
camellia-xts-essiv:sha256 / 256 : 103
serpent-ctr-essiv:sha256 / 256 : 103

Now my /etc/crypttab proably looks like this:

swap /dev/sda2 /dev/urandom swap,cipher=twofish-xts-essiv:sha256,size=256,hash=sha512

Word of caution: this is a benchmark - some arbitrary test for a very special usecase, executed on one machine and one machine only (Fedora 18 in an ESX virtual machine, equipped with 2 AMD Opteron 848 processors). Before applying these results to your environment, run the benchmark yourself or, better yet: write your own benchmark for your usecase!

Posted by christian in benchmarks at 23:21 | Comments (0) | Trackbacks (0)

Thursday, November 15. 2012

Mediawiki restore

The other day this box went down and I could not access my Mediawiki installation. The box was meant to come back online later on but I really wanted to read the wiki, now. Luckily I had somewhat fresh backups from the mediawiki installation - so why not using them? Just to see how quickly a full restore would take.

So I fired up an openSUSE 12.1 installation, running in a virtual machine. The base system was already installed, a few more packages were needed now:

zypper install nginx mysql-community-server php5-fpm php5-mysql php5-intl php5-gd

Note: Mediawiki likes to utilize object caching, such as xcache or APC. However, PHP modules for openSUSE like php-APC or php5-xcache are only available via extra repositories. For the sake of simplicity, let's skip those now.

With these packages installed, their configuration comes next. This may be a bit openSUSE centric and other distributions may work differently.

For PHP-FPM, only the following parts were changed from its original configuration:

 $ cp -p /etc/php5/fpm/php-fpm.conf{.default,}
 $ cat /etc/php5/fpm/php-fpm.conf
 [...]
 [global]
 pid       = /var/run/php-fpm.pid
 error_log = /var/log/php-fpm.log
 
 [www]
 listen    = /var/run/php5-fpm.sock
 user  = nobody
 group = nobody
 pm = dynamic
 pm.max_children = 50
 pm.start_servers = 20
 pm.min_spare_servers = 5
 pm.max_spare_servers = 35

Enable and start PHP-FPM:

chkconfig php-fpm on && service php-fpm start

Next up is nginx. A very basic configuration:

 $ cat /etc/nginx/nginx.conf
 user                    nginx;
 worker_processes        1;
 error_log       /var/log/nginx/error.log;
 pid             /var/run/nginx.pid;
 
 events {
        worker_connections      1024;
        use                     epoll;
 }
 
 http {
        include         mime.types;
        default_type    application/octet-stream;
        access_log      /var/log/nginx/access.log;
 
        # This will access our PHP-FPM installation via a socket
        upstream php5-fpm-sock {
                server          unix:/var/run/php5-fpm.sock;
        }
 
   server {
        listen          80;
        server_name     suse0.local;
        root            /var/www;
        index           index.html index.php;
        autoindex       on;
        access_log      /var/log/nginx/suse0.access.log;
 
        location ~ \.php?$ {
                try_files $uri =404;
                include fastcgi_params;
                fastcgi_pass php5-fpm-sock;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_intercept_errors on;
         }
   }
  }

Enable and start nginx (and MySQL):

  $ chkconfig nginx on && service nginx start
  $ chkconfig mysql on && service mysql start

By now we should have a working webserver, able to serve PHP pages. Now to the actual Mediawiki installation:

  $ cd /var/www
  $ wget http://download.wikimedia.org/mediawiki/1.20/mediawiki-1.20.0.tar.gz{,.sig}

Import their GPG keys and verify the signature:

  $ wget https://www.mediawiki.org/keys/keys.txt -O - | gpg --import 
  $ gpg --verify mediawiki-1.20.0.tar.gz.sig

Unpack:

  $ tar -xzf mediawiki-1.20.0.tar.gz 
  $ ln -s mediawiki-1.20.0 mediawiki
  $ cd mediawiki

With that in place, we could go to http://suse0.local/mediawiki/ and use the install wizard to install a basic, but empty Mediawiki. Once this is done, we restore a few things from our backup:

  $ tar -C ../backup/mediawiki/ -cf - LocalSettings.php extensions images | tar -xvf -
  $ bzip2 -dc ../backup/DB_wikidb.sql.bz2 | mysql -D wikidb

In our case, a few modifications to LocalSettings.php had to be made:

 # We have not yet set up any rewrite rules so short urls
 # won't work for now
 ## $wgArticlePath               = "/wiki/$1";
 
 # Disable this one for now
 ## $wgServer              = "https://www.example.org";
 
 # Our database details are different of course:
 $wgDBtype           = "mysql";
 $wgDBserver         = "127.0.0.1";
 $wgDBname           = "wikidb";
 $wgDBprefix         = "mw_";          # Our original database used a table prefix!
 $wgDBuser           = "root";
 $wgDBpassword       = "s3cr3t";
 
 # no APC/Xcache for openSUSE just now
 ## $wgMainCacheType       = CACHE_ACCEL;

Also: check those ''extensions'' or disable them if things don't work as expected.

Now, let's run the update.php script, to address any version differences of our new Mediawiki instance:

  $ php maintenance/update.php --conf `pwd`/LocalSettings.php

Done! The Mediawiki installation should now work. If it doesn't, try to set a few more things in LocalSettings.php:

 # At the very top:
 error_reporting( E_ALL | E_STRICT );
 ini_set( 'display_errors', 1 );
 
 $wgShowExceptionDetails        = true;
 $wgShowSQLErrors               = true;

Good luck! :-)

Posted by christian in bits at 23:48 | Comments (0) | Trackbacks (0)

(Page 1 of 21, totaling 307 entries) » next page