Statement of purpose:

- Show the need for a consistent internationalization strategy for Web search engines which work on multilingual corpora;
- Demonstrate sample implementations using the TUSTEP toolbox and outline implementation strategies;
- Show on the fly conversion to Unicode from the local storage format to output on the Web.

Abstract

TUSTEP is a toolbox with building blocks which allow flexible handling of structured texts, especially XML-conformant texts. It started its life in the late 60s and has been enhanced ever since. For some 25 years it is used both in commercial and in scholarly applications.

Building blocks include amongst others:
- Professional typesetting facilities (user base in the publishing industries; several prices for excellence in typography) with strong support for multiscript texts;
- 14651-conformant collation tools;
- Database support for XML databases. Access of these databases via CGI, using TUSTEP's efficient and powerful script language.

The paper will concentrate on the implementation of search engines. It will outline general strategies and desiderata for intelligent fuzzy searching on multilingual corpora (cf. also my report for the European Commission on European requirements in the field of browsing and matching at http://www.stri.is/TC304/Matching). These requirements include dealing with local fallback rules (e. g. Þ [Thorn]/ Th), transliteration, phonetically aware searching, legacy character sets, ordering expectation, etc.

The paper will then proceed to demonstrate exemplary implementations in TUSTEP and explain the decisions behind them. It will be shown how data -- including data in non-Latin scripts -- is converted on-the-fly to UTF-8 for presentation on the web. The same data pool can conveniently be used for high-quality database publishing.

The talk will be divided approximately equally between general strategies and concrete implementations and should therefore be of interest to both programmers and project managers.