Re: Re: Do you know a tool to decode "UTF-8 twice"

On Mon, Oct 28, 2013 at 9:48 AM, Buck Golemon <buck_at_yelp.com> wrote:

>
>
>
> On Mon, Oct 28, 2013 at 6:06 AM, "Jörg Knappen" <jknappen_at_web.de> wrote:
>
>> Hi Steffen,
>>
>> data aren't that easy. There are non-latin1-characters encoded in the
>> UTF8 part. I expect
>> among others typographic apostrophes, polish characters, some
>> mediaevalist characters like
>> ũ (u with tilde). Maybe, there is also some greek inside, but I am not
>> sure about that.
>>
>> --Jörg Knappen
>>
>> *Gesendet:* Montag, 28. Oktober 2013 um 12:34 Uhr
>> *Von:* "Steffen \"Daode\" Nurpmeso" <sdaoden_at_gmail.com>
>> *An:* "Jörg Knappen" <jknappen_at_web.de>
>> *Cc:* unicode_at_unicode.org
>> *Betreff:* Re: Do you know a tool to decode "UTF-8 twice"
>> "Jörg Knappen" <jknappen_at_web.de> wrote:
>> | Is there a ready made tool that decodes "UTF-8 twice" while keeping
>> | UTF-8 proper in place?
>>
>> Isn't a shell script with a truly validating iconv(1) enough?
>> This works for me if in utf8.1 there is 'ÄEIÖÜ' in UTF-8 and i run
>>
>> ?0[steffen_at_sherwood tmp]$ iconv -f latin1 -t utf8 < utf8.1 > utf8.2
>>
>> As in
>>
>> for i in utf8.1 utf8.2; do
>> if iconv -f utf8 -t latin1 < ${i} |
>> iconv -f utf8 -t utf8 >/dev/null 2>&1; then
>> echo ${i}: bummer, going home by one
>> iconv -f utf8 -t latin1 < ${i} > ${i}.new 2>&1
>> else
>> echo ${i}: valid UTF-8
>> fi
>> done
>>
>> i'll end up as
>>
>> ?0[steffen_at_sherwood tmp]$ sh utf8dec.sh
>> utf8.1: valid UTF-8
>> utf8.2: bummer, going home by one
>> ?0[steffen_at_sherwood tmp]$
>>
>> Ciao,
>>
>> | --Jörg Knappen
>>
>> --steffen
>>
>
> Jörg: There's no ready-made tool, but it's easy to write in python.
> I'll provide you a well-tested function in a few minutes.
>
>
>
>
Jörg:

Here is my function (also attached):
http://paste.pound-python.org/show/jAfKzb5HEyOeGvyF7O9W/
You can either make a larger python program with it, or expose it directly
to shell scripting.
These are the tests passed:

A latin1-encoded string should become utf8-encoded ... ok
An un-encoded unicode string should just become utf8-encoded ... ok
A utf8-encoded string should be unchanged ... ok
A poorly-encoded utf8+latin1 string should become utf8-encoded ... ok
A string mangled by utf8+latin1 several times should become utf8-encoded
... ok

----------------------------------------------------------------------
Ran 5 tests in 0.001s

OK