On Fri, Feb 15, 2002 at 12:41:02PM -0800, Rick Cameron wrote:
> understanding that MBCS character sets are not uncommon on unix (for
> example, EUC). If my program is running on a unix system where the default
> character set is EUC-JP (as I believe it's called) and it tries to import a
> text file containing UTF-8 without a BOM, how is the program supposed to
> know that the file contains UTF-8 rather than EUC-JP?
Why can't EUC-JP start with EF BB BF? (Why can't UTF-16 start with EF BB
BF?) If a user wants to open a UTF-8 file, he should either tell the
program it's UTF-8 or run in a UTF-8 locale.
> So not a unix problem, but rather a problem with dumb command-line tools. I
> wonder whether the GNU people have thought of making their command-line
> tools aware of UTF-8 & BOMs.
It's impossible. For one thing, the tools work and should continue to
work on binary data, given user care. They must continue to work on
ASCII data, including not introducing spurious non-ASCII data. Also, if
I do "cat a b c >> d", and d doesn't exist, then who is supposted to add
the BOM? And what happens when it turns out that a, b, and c are gzip
archives, and adding the BOM corrupted them? Or d does exist, and is
some funky binary format that doesn't mind extra data at the end (like
some executable formats) but does mind the BOM at the start?
To add the BOM would make text programs much more complex and more
fragile, and in exchange for what? UTF-8 is fairly easy to recognize
from other formats, and it's a problem that people have been dealing
with fairly well for over two decades, for 7/8-bit formats that are
indistiguishable without user input.
-- David Starner / Давид Старнэр - starner@okstate.edu Pointless website: http://dvdeug.dhis.org What we've got is a blue-light special on truth. It's the hottest thing with the youth. -- Information Society, "Peace and Love, Inc."
This archive was generated by hypermail 2.1.2 : Fri Feb 15 2002 - 19:47:37 EST