UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and...

23
UNICODE • Character Sets and Coding Standards • Han Unification and ISO10646 • Encoding Evolution and Unicode • Programming Unicode

Transcript of UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and...

Page 1: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

UNICODE

• Character Sets and Coding Standards

• Han Unification and ISO10646

• Encoding Evolution and Unicode

• Programming Unicode

Page 2: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Character Sets

• Character Sets - a complete group of characters for one or more writing system.

• Coded character set – a mapping from a set of abstract characters to a set of integers.

• Character encoding scheme- a mapping from a coded character set to a set of octets.

Page 3: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Standards

• International standards – - RFC 2130, - ISO

• RFC standards come out of the Internet Engineer Task Force(IETF), it stands for Request For Comments.

• ISO is a worldwide federation of national standards bodies such as ANSI, JISC, KISI, CSA(GB} and ECMA.

Page 4: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Coded character sets (1)

• ASCII ( American Standard Code for Information Interchange) or ISO 646 defined by 7-bit

- Latin alphabet;

- Arabic numerals;

- Punctuation marks;

- Some computer control codes;

Page 5: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Coded character sets (2)

• ISO-8859 Standards- ISO-8859-1 to 15• ISO-8859-1(Latin1) - the most widely used

standard, it contains the characters necessary for writing in Western European and Scandinavian languages. It is also an ANSI Standard and known as the ANSI character set.

• The first 128 positions are identical to ASCII• All use 8-bit byte• Windows use ISO-8859-1 as its character set.

Page 6: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Coded character sets (3)

• Asian Language character setsTraditional Chinese(Taiwan)- BIG5: 8-16 bit, 94x157 matrix :13002 characters;- CNS 11643 X5012: 16 bit, 48000 characters.Simplified Chinese(China)- GB2312-80: 8-16 bit, 94x94 matrix: 6763

characters.- GBK (Extended GB2312-80): 8-16 bit, 94x94

matrix, 21003 characters.

Page 7: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Coded character sets (3)

Japanese(Japan)- JIS x0208: 16 bit, 94x94 matrix, 6879 characters.- JIS x0212: 16 bit, 5801 supplemental kanji.- JIS x0201: 7 bit, JIS-Roman plus half-width

katakana.Korean(Korea)- KSX-1001(KSC 5601): 8-16 bit, 94x94 matrix,

4888 hanja.- KSC-5636: 7 bit Korean version of ASCII.

Page 8: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Coded character sets (4)• EUC (Extended UNIX Code);• ISO 2022 Escape sequence• Windows Code pages: 8-16 bit,

- cp1252 English is ISO8859-1 or Latin1or Windows ANSI;- cp1200 Unicode code page;- cp932 Japanese is shift-JIS;- cp936 Simplified Chinese is close to GB2312-80;- cp949 Korean is in KSC 5601-1992 order;- cp950 Traditional Chinese is the same as Big 5.

Page 9: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Han Unification

• The Han standards consist of :JIS X0208(6349 Kanji), GB 2312-80 (6763 Hanzi), CNS 11543 (13951 Hanzi) and KSC 5601 (4888 Hanji). All characters in these standards must be included in 10646

• Without unification, more than 100,000 characters are separately encoded

• Characters from these standards should be unified in ISO 10646 that is identical characters from two or more of these standards may have the same code point in 10646. The unification of Han characters allows for approximately 40,000 characters.

Page 10: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Coded character sets (5)• Unicode: 16 bit, most languages are covered, developed by the

Unicode Consortium.• Unicode 1.0 Unicode 2.0 contains 38,885 characters. It will be

Unicode 3.0 in this year.• Unicode is similar to ISO 10646, the UCS(Universal Character Set)

encoding, - UCS-2: i.e.Unicode, 16 bit, a fixed with 2 byte format scheme; - UCS-4: also called ISO 10646, 32 bit, 4 byte format, 32,000 planes each with 65,000 characters capacities, for tall 2,080 million characters. The 1st plane is in use(that’s Unicode)- UTF-7: 7-40 bit, UCS Transformation Format, a Unicode character encoding scheme using a 7 bit.- UTF-8: 7-48 bit, UCS Transformation Format, a Unicode character encoding scheme using a 8 bit.- UTF-16(UCS-2E): 16-32 bit.

• Windows NT, use Unicode, has a single character set.

Page 11: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

• UCS Transformation Format;• A variable-width or multi-byte encoding format;• In UTF-8, the standard ASCII characters occupy

only one byte, other Unicode characters occupy two or three bytes.

Start Character End Character Required Data Bits Binary Byte Sequence (x = data bits)

\u0000 \u007F 7 0xxxxxxx

\u0080 \u07FF 11 110xxxxx 10xxxxxx

\u0800 \uFFFF 16 1110xxxx 10xxxxxx 10xxxxxx

Table The UTF-8 Encoding

UTF-8

Page 12: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Unicode

ISO 10646 (UCS2)

16 bits

8 bits

EUC

PC CPs

Shift-JIS

ISO2022

ISO8859

EBCDIC

Apple CPs

7 bits ASCIIGB JIS KSC

Encoding Evolution

Page 13: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Programming Unicode

• How does Win32 support it

• How to write apps for Unicode

• How to be backward compatible

Page 14: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Win32 Server

Windows NT Base

Win32 App(Unicode)

Win32 AppNon-Unicode

ANSI to Unicode Conversion

Win32 Client

Client-Server Boundary

System fully Unicode internally

Unicode in Win32

Page 15: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Unicode in Win32

• Separate Unicode Datatype

- Wide character: Unicode

- 8-bit char:ANSI, DBCS

• Parallel Unicode and ANSI APIs

- Unicode and ANSI windows classes

- Implicit code conversion

• Resources always in Unicode

Page 16: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Programming Unicode

• Basic techniques

- How to migrate existing code base• Special topics

- Interaction with non-Unicode apps

- Untyped file system• Foundation for script/language specific

functionality• Migration example: Win32

Page 17: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Windows Unicode Programming

CodeConversion

DataExchange

Common Source

Win32 EXENon-Unicode

Win32 EXEUnicode

Page 18: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Generic data types

TCHAR LPTSTR

wchar_t wchar_t *char char *

Explicit data types

WCHAR CHAR LPWSTR LPSTR

wchar_t char wchar_t * char *

Generic data types in C

Page 19: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Macros and literals

• String literals

TEXT(“hello”);

“hello”; L”hello”;

• numeric equivalence

‘A’ = 0x41 0x0041 = L’A’

Page 20: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Dual function prototypes

SetWindowText (HWND, LPTSTR) ;

SetWindowTextA (HWND, LPSTR) ;

SetWindowTextW (HWND, LPWSTR) ;

#ifdef UNICODE

• Generic API prototypes

• Resolved to explicit prototypes

Page 21: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Basic conversion steps

• Use generic data types:

TCHAR, LPTSTR for Text

• Use explicit types for BYTE pointers (data buffers)

• Use TEXT() macro for literal constants

• Adjust pointer arithmetic

• Use generic function prototypes

Page 22: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Conversion metrics

• About 10% of source lines need to be modified by global replace

LPSTR LPTSTR

strstr() wcswcs()• About 2-5% need small modification

lstrlen(s) lstrlen(s)/sizeof(TCHAR)• Less than 1% need revised algorithm

OpenFile() SearchPath(); CreateFile()

Page 23: UNICODE Character Sets and Coding Standards Han Unification and ISO10646 Encoding Evolution and Unicode Programming Unicode.

Summary: Migrating Windows-based Program to Unicode

1. Modify your code to use generic data types2. Modify your code to use generic function prototypes3. Surround any character or string literal with the TEXT macro.4. Create generic version of your data structures5. Change your make process.6. Adjust pointer arithmetic.7. Check for any code that assumes a character is always 1 byte long.8. Add Unicode-specific code if necessary.9. Add code to support special Unicode characters.10. Determine how using Unicode will affect file I/O.11. Double check the way in which you retrieve command line

arguments.12. Debug your port by enabling your compiler’s type-checking