ARCH-12 Broaden Your Potential Customer Base Using Unicode

ARCH-12Broaden Your Potential Customer Base Using Unicode

David LundSr. Training Program Manager, Progress

2 © 2005 Progress Software Corporation ARCH-12, Unicode

Broaden Your Potential Customer Base Using Unicode

Unicode is the best way to support multiple languages

A number of recent OpenEdge™ enhancements facilitate Unicode

OpenEdge tools simplify the task


Agenda - Implementing Unicode

Essentials

Migrating a Database

Unicode Client

Sorting

Normalization

Other Areas to Consider


Unicode Essentials

Unicode foundation for creating internationalized and localized applications

Unicode provides a unique number for every character

Lossless round tripping– Mapping from any Unicode coded character

sequence S to a sequence of bytes and back will produce S again


Unicode Essentials

UTF – Unicode Transformation Format– Algorithm for mapping (encoding) Unicode

scalar value to a unique sequence

– 3 formats (mappings) UTF-8, UTF-16, UTF-32

– Formats vary in how they handle mapping Impacts access, storage, and performance


Unicode Essentials

Code Page = table that assigns a numeric value– Letters, numbers, punctuation, control codes, etc.

– prolang\list-cp.p lists code pages in convmap Sample Code Page – IBM850 (partial)

– Character ‘2’ is hex 32

20 30 400 0 @1 ! 1 A2 2 B3 # 3 C


Progress I18N Essentials

Undefined code page– Tells Progress not to do any conversions

when reading or writing data

– For example Sports database uses undefined Can be used with any character set

I18N (Internationalization)



Startup parameters – -cpinternal

Code page used for internal data processing

– -cpstream Code page used for stream files

– Parameter file prolang\UTF\UTF-8.pf

I18N (Internationalization)

-cpinternal utf-8-cpstream utf-8



Performing code page conversions– Progress provides a character set

management facility– Automatically converts data between the

code pages of different data sources and targets

Must be in CONVMAP file

Targets for code page conversion– Memory (-cpinternal)– Streams (-cpstream)– Databases



proutil <dbname> –C CODEPAGE-COMPILER convmap.dat convmap.cp

Referenced code pages must be in CONVMAP

Modifying CONVMAP– Edit convmap.dat

– Compile CONVMAP

– Make convmap.cp available to session Progress installation directory PROCONV environment variable -convmap startup parameter



Converting characters or strings in memory– Specify code page in functions

ASC CHR CODEPAGE-CONVERT

Converting input and output data– Specify code page in statements

INPUT FROM (input source to memory target) OUTPUT TO (memory source to output target)


Fonts for Unicode

Locating fonts on windows– C:\WINDOWS\Fonts

– Control Panel, select Font icon Unicode fonts may need to be purchased Setting Unicode fonts for Progress

– Progress.ini

– Use ini2reg.exe to place in registry


System Resources



Essentials


Unicode Client

Sorting

Normalization



Migrating a Database to Unicode

Two ways to migrate database to Unicode– Dump and Load– Converting the database without doing a

dump and load

Start an OpenEdge session– Use startup parameters

-cpinternal UTF-8 -cpstream UTF-8


Migrating a Database to UnicodeCautions:

Backup your database Dump definitions and data

– Do not do a binary dump and load Binary data is not converted to the code

page of the database when it is loaded

– Always use Data Admin tool Goes through automatic conversion

Using dump and load 1 of 3



Create an empty UTF-8 database– Data Administration tool

Database>Create Database

– Create Database dialog Select radio set to create a copy of some

other database Select an empty database from

prolang/UTF-8– For example empty4.db




Load the Definitions– Load will convert to UTF-8 automatically

Load the Data– Data will be automatically converted to

UTF-8 from the dumped code page when it is loaded




Backup your database Use proutil to convert the database

Load the UTF-8 collation table– prolang/UTF/_tran.df

Assign a word break rules to the database Rebuild the indexes

Converting without a dump and load

proutil <db-name> -C convchar convert UTF-8

proutil <db-name> -C idxbuild



Essentials


Unicode Client

Sorting

Normalization



Benefits of GUI Unicode Client

Multi-lingual– Able to use data from multiple languages in the

same session Fully enables AppBuilder to build multilingual

UTF-8 applications Easier deployment:

– Lower costs, higher ROI– No need to have different configurations using

specific settings per language Increased competitive advantage

– No (or very few changes) required to existing apps to take advantage of GUI Unicode client

Added in OpenEdge 10.0A release


Unicode Editor

RichEdit editor in OpenEdge 10– Supports Unicode

Selecting an editor– Modify UseSourceEditor in progress.ini – Default SlickEdit:

UseSourceEditor=yes– For Unicode use RichEdit:

UseSourceEditor=no


Demonstration

GUI UnicodeClient

MultipleLanguages



Essentials


Unicode Client

Sorting

Normalization



Linguistic Sorting

Language sensitive collations

– Tailor to expectations of locale Language Country

Easy to use– Functions just like any other

collation for 4GL

The goal …


Unicode Sorting

OpenEdge 10.0A supports binary sorting Basic collation support Sorts by value in code page Possible to do user defined sorting

OpenEdge 10.0B also supports linguistic sorting– Supports ICU collations

International Components for Unicode OpenEdge does not support multiple

collations in the database


Binary versus Linguistic Sorting -A Visual

beetcarrotentrytrustzoomécoleçedilla

beetcarrotçedillaécoleentrytrustzoom

Binary Sort Linguistic Sort

English (ICU-en)


Linguistic Sorting

Progress uses collations for:– -cpcoll session startup

parameter

– Database collation

– Collation of database CLOB column

– Argument to COMPARE function COLLATE option of the BY

phrase


Linguistic SortingSupported Collations

OpenEdge supports all ICU collations in the icui18n library– Beyond icui18n one additional

collation is supported Japanese Hiragana Quaternary

as case-sensitive


Linguistic Sorting

4GL Usage - Reference collation by name For example “ICU-fr” for French

Specify using– -cpcoll <table name>

Identifies collation table to use with code page in memory at session startup

<table name> is the collation table in convmap.cp or the name of the ICU collation

– 4GL Statements COMPARE COLLATE


Linguistic Sorting

/* French collation */DISPLAY “ICU-fr = ” + COMPARE("côte", "<", "coté",

"case-insensitive", "ICU-fr")/* Spanish collation */DISPLAY “ICU-es = ” + COMPARE("côte", "<", "coté",

"case-insensitive", "ICU-es")

ICU-fr = yesICU-es = no

Sort order depends on selected collation

Output of above statements


Linguistic SortingExamples 1 of 4

Examples– UTF-8 database with “basic” collation– Names: beet, carrot, çedilla, entry, école,

zoom, trust

FOR EACH words WHERE name < “t”:DISPLAY name.

END.

beetcarrotentry

Output result



FOR EACH words WHERE name >= “t”:DISPLAY name.

END.

trustzoomécoleçedilla

Output result



FOR EACH words WHERE COMPARE(name < “t”,“case-insensitive”,

“ICU-en”):DISPLAY name.

END.

beetcarrotentryécoleçedilla

Output result



FOR EACH words WHERE COMPARE(name < “t”,“case-insensitive”,

“ICU-en”) BY COLLATE(name, “case-insensitive”,

“ICU-en”): DISPLAY name.

END.

beetcarrotçedillaécoleentry

Output result



Essentials


Unicode Client

Sorting

Normalization



Unicode Normalization

Why is this needed?

– Puts in “NCF” format as expected by XML (and other W3C entities)

– Best way to convert from Unicode to other code pages

– Useful when doing tasks such as making comparisons



Unicode has different ways of expressing the same characters– Base letter plus combining marks (accents)

as two Unicode code points Á = composite (composed)

(U+0041, Latin Capital Letter A) + (U+0301, Combining Acute Accent ´)

– Base letter and accents as one Unicode code point

Á = precomposed (U+00C1, Latin Capital Letter A with Acute)

What is normalization?



NORMALIZE– 4GL function new in OpenEdge 10.0B

– Returns either CHAR or LONGCHAR Matches the source string CHAR variable must be UTF-8 LONGCHAR variable any form of Unicode

– UTF-8, UTF-16, UTF-32

result-string = NORMALIZE(source-string, normalization-mode)


Normalization Modes Supported

NFD– Canonical Decomposition

NFC– Canonical Decomposition, followed by Canonical

Composition NFKD

– Compatibility Decomposition NFKC

– Compatibility Decomposition, followed by Canonical Composition

None– No change to source string– Turns off normalization when normalization-mode

is a variable



Essentials


Unicode Client

Sorting

Normalization



Bidi Support

Bi-directional (bidi)– Behavior of individual widgets and/or the complete

window to go from right to left or left to right Supported

– Fill-in widget Can type right to left of left to right

Not-Supported– Whole frame

Cannot switch labels from left side to right side


GB18030 Code Page SupportAdded in OpenEdge 10.0B

New Chinese code page Required for all new

software sold in mainland China as of Jan. 1, 2001


Broaden Your Potential Customer Base Using Unicode

Unicode is the best way to support multiple languages

A number of recent OpenEdge™ enhancements facilitate Unicode

OpenEdge tools simplify the task

In summary


Documentation

OpenEdge Development– Internationalizing

Applications


Unicode Resources

Unicode Home page– http://www.unicode.org

– Unicode Standard, Unicode Consortium

International Components for Unicode– http://www-124.ibm.com/icu/docs/– http://www-124.ibm.com/icu/docs/papers/forms_

of_unicode/


System Resources

Viewing keyboard layoutshttp://www.microsoft.com/globaldev/reference/keyboards.aspx

– Select the language and the keyboard layout is displayed

– Use shift key to toggle to ‘lower/upper’ case characters

– Use MS Internet Explorer to display


Questions?


Thank you for your time!

ARCH-12 Broaden Your Potential Customer Base Using Unicode

Documents

Transcript of ARCH-12 Broaden Your Potential Customer Base Using Unicode