tr: Illegal byte sequence

Sunday, April 11. 2010

tr: Illegal byte sequence

$ cat /tmp/foo
ÿs
$ tr -d \r < /tmp/foo 
tr: Illegal byte sequence

Whoops? Let's take a closer look:

$ od -x /tmp/foo 
0000000      73ff    0a0d
0000004

So, it's some unicode character (0xff), a small "s" (0x74), then a CR (0x0d, which I'm trying to remove) and a newline (0x0a) at the end. Turns out it's how MacOS 10.6 handles unicode characters. Specifying a different locale seems to help:

$ LC_CTYPE=C tr -d \r < /tmp/foo 
ÿs

Posted by christian in bits at 01:34 | Comments (3) | Trackback (1)

Trackbacks

Trackback specific URI for this entry

whois: Invalid charset for response
As if MacOS didn't have enough charset problems, here's another one: $ /usr/bin/whois denic.de % Error: 55000000013 Invalid charset for response Although the problem has been reported to DENIC years ago, they still send out UTF-8 data if the ha

Weblog: s9y testdrive
Tracked: May 13, 12:17

Comments

Display comments as (Linear | Threaded)

Thanks. This has helped me clean up a massive 4Gb file!

#1 Robin (Homepage) on 2011-03-21 07:30 (Reply)

Thanks very much :-) I18n strikes again!

#2 Jon (Homepage) on 2012-04-24 06:48 (Reply)

thank you for posting this, christian!

#3 ryan on 2012-12-13 06:09 (Reply)

Add Comment