The standard characterizes it in 5.13, Locating Text Element Boundaries. That section is not normative, but if
people follow it, they will interoperate. If your repertoire is limited to European characters, this means a base
plus any number of combining marks--what people expect.
Note: Unicode 3.0 clarifies this section further, and it is also discussed in TR18. (An updated version of TR18
will be discussed at the UTC next week; it is included in the documents collected by Arnold.)
Mark
Tex Texin wrote:
> Mark,
> Thanks Mark, I am familiar with these. It finesses the
> question though. Where does the standard define (for each locale
> if needed), what is or isn't a character/letter/grapheme?
>
> If my application uses ICU, and some other application doesn't,
> if we interchange data (client/server) will we get the same results
> with the same queries/operations?
>
> Also, I noticed in a few places I wrote "multi-non-spacing characters"
> where I meant "multi-spacing characters". ah well.
>
> tex
>
> "Mark E. Davis" wrote:
> >
> > You raise a very good point.
> >
> > What we use in ICU and in Java is a BreakIterator, that gives you 'character' boundaries (you can also
> > choose word, line and sentence boundaries). Cursor movement is one area where character boundaries are
> > useful; searching is another (if a search matches, you want to ensure that the boundaries of the match are
> > character boundaries.
> >
> > You use getCharacterInstance(Locale) to get the iterator; it will then give you boundaries on text. The
> > character properties of the whole are extrapolated from the first code point.
> >
> > The Java interface is at:
> >
> > http://java.sun.com/products/jdk/1.2/docs/api/java/text/BreakIterator.html
> >
> > The ICU C++ interface is at
> >
> > http://www10.software.ibm.com/developerworks/opensource/icu/project/html/BreakIterator.html
> >
> > It also has the overall documentation.
> >
> > The ICU C interface is in separate routines all starting with "ubrk_", e.g. to open, iterate, and close you
> > would use the following:
> >
> > http://www10.software.ibm.com/developerworks/opensource/icu/project/html/ubrk_open.html
> > http://www10.software.ibm.com/developerworks/opensource/icu/project/html/ubrk_next.html
> > http://www10.software.ibm.com/developerworks/opensource/icu/project/html/ubrk_close.html
> >
> > For random access, you use:
> > http://www10.software.ibm.com/developerworks/opensource/icu/project/html/ubrk_following.html, or
> > http://www10.software.ibm.com/developerworks/opensource/icu/project/html/ubrk_preceding.html
> >
> > Mark
> >
> Tex Texin wrote:
>
> Dear Uni-people,
>
> I am of course a supporter and a benefiter of Unicode
> and its many improvements over legacy encodings. As an application
> implementer, and not a linguist, typographer, or nationalist (IE
> not favoring one language or politics over another),
> I look to Unicode to provide me with standardization so I can
> provide world-wide plain-text support. I like that Unicode
> defines algorithms for bidirectional support, character properties
> and the like, and I am no longer in the business of researching
> both code pages and the algorithms for using those code pages.
> (Well, I do a lot less of it now anyway. ;-) )
>
> As a software designer, I need to understand and rely on some
> basic principles. For example, I have to have a GetNextCharacter
> routine. This list can go on and on (and on...) about abstract
> characters, versus letters, versus graphemes, but I need to
> implement something close to what a user expects, (or what I can
> teach them to work with) and to have
> an element, or basic unit, that I can manipulate and design to use
> in my software.
>
> I can work with 16-bit units as Unicode defines them, and I can
> program to either provide users with behaviors based on these
> units (e.g. cursor-right moves through each unit, i.e. through
> each diacritic, tone mark, etc.) or I can
> provide users with a more complete element that users traditionally
> think of as a character (e.g. cursor-right moves to the next letter).
>
> However, Unicode does not prescribe how I recognize or work with
> with elements like "ch". As an implementer, I thought GetNextCharacter
> would look for a 16-bit abstract character (or multi-16-bit
> surrogate) followed by some non-spacing elements. Apparently, this
> is not sufficient.
>
> I also thought I could look up properties of "characters" but I
> don't know where to look up the properties of "ch".
>
> I imagine some of these "multi-spacing character" elements,
> will cause me to redesign my approach to bidi as well.
>
> Although I recognize the many benefits of Unicode, if I cannot
> understand how to reliably implement GetNextCharacter, and have
> a property table for these "multi-spacing character" elements,
> then many current designs for Unicode applications are now inadequate.
> For developers, the greatest benefit of Unicode was it provided
> standardization for the basic character element.
> I suddenly feel thrown back into the multi-codepage
> quagmire of researching researching, researching, and probably
> continually revamping and re-generalizing my software to accomodate
> new character types as I uncover them.
>
> I believe I do understand the rationales offered for why "ch"
> should not be a character. I would rather the onus be shifted to
> input methods and legacy conversion programs to determine whether
> "ch" should be encoded as a single element or not, rather than
> having all remaining software be continually analyzing this.
>
> I would not like for this note to kick off a repetition of everything
> that has already been said many times. Perhaps it is inevitable.
> I would really like a clear definition for knowing how many bytes
> or 16-bit words to read to get to the end of the current "letter",
> and how to look up the properties of that letter, its case, etc.
> These are the basic elements I need to use in my software.
> The algorithms for doing this, should be able to support Slovak and
> other languages in a uniform way.
>
> tex
> Progress Software: The #1 Embedded Database
> -------------------------------------------------------------------------------------------------------
> Tex Texin Director, International Products
>
> Progress Software Corp. Voice: +1-781-280-4271
> 14 Oak Park Fax: +1-781-280-4949
> Bedford, MA 01730 USA texin@bedford.progress.com
>
> http://www.progress.com http://apptivity.progress.com
> -------------------------------------------------------------------------------------------------------
This archive was generated by hypermail 2.1.2 : Tue Jul 10 2001 - 17:20:54 EDT