SAP UNICODE

Unicode Enabling of ABAPUnicode Enabling of ABAP

ContentsContents

• Unicode Overview• What is Unicode• Need for Unicode• Unicode Character formats

• Unicode @ SAP• SAP Pre-Unicode solutions• Why SAP adopted Unicode

• Impact of Unicode on ABAP• Overview• Concepts and Conventions• Restrictions in Unicode• New ABAP Features• Tools for Unicode enabling

What is Unicode?What is Unicode?

Unicode OverviewUnicode Overview

What Is Unicode?What Is Unicode?

• Fundamentally, computers store letters and other characters by assigning a number for each one.

• Unicode provides a unique number (Code point) for every character, no matter what the platform, no matter what the program, no matter what the language.– Notation U+nnnn (where nnnn are hexadecimal digits)

Character encoding schemafor (nearly) all characters used world wide

What is Unicode? (contd.)What is Unicode? (contd.)

• Unicode = universally encoded character set to store information from any language

• The Unicode standard primarily encodes scripts rather than languages

• Scripts comprise several languages that historically share the same set of symbols

• In many cases a script may serve to write dozens of languages (e.g. the Latin script)

• In other cases one script complies to one language (e.g. Hangul)

• Additionally it also includes punctuation marks, diacritics, mathematical symbols, technical symbols, musical symbols, arrows, etc.

• In all, the Unicode Standard comprises >95.000 characters, ideograph sets, symbols (version 4.0)


• Unicode Standard:The Unicode Standard is a character coding system designed to support the worldwide interchange, processing and display of written text of the diverse languages and technical disciplines of the modern world.In addition, it supports classical and historical texts of many written languages.

• The Unicode Consortium:The Consortium cooperates with W3C

ISOand has liaison status "C" with ISO/IEC/

JTC1/SC2/WG2, which is responsible for in refining the specification and expanding the character set of ISO/IEC 10646


• Unicode: The last character set?

– It is an open character set, which means that it keeps growing and adding less frequently used characters.

– The standard assigns numbers from 0 to 0x10FFFF, which is more than a million possible numbers for characters.

– 5% of this space is used, 5% is in preparation, about 13% is reserved for private use, and about 2% is reserved not for use

– The remaining 75% is open for future use but not by any means expected to be filled up and finally there is a character set with plenty of space!

Need for UnicodeNeed for Unicode


Need For UnicodeNeed For Unicode

• Hundreds of encodings have been developed, each for small groups of languages and special purposes.

• There is no single, authoritative source of precise definitions of many of the encodings and their names.

• No single encoding could contain enough characters: for example, the European Union alone requires several different encodings to cover all its languages.

• Even for a single language like English no single encoding was adequate for all the letters, punctuation, and technical symbols in common use.

• Incompatibilities between different code pages

Need For Unicode (contd.)Need For Unicode (contd.)

• These encoding systems conflict with one another. That is, two encodings can use the same number for two different characters, or use different numbers for the same character.

• Any given computer (especially servers) needs to support many different encodings; yet whenever data is passed between different encodings or platforms, that data always runs the risk of corruption.

• Programs are written to either handle one single encoding at a time and switch between them, or to convert between external and internal encodings

Need For Unicode (contd.)Need For Unicode (contd.)

• Languages & Code Pages

Unicode, One Code Page For All ScriptsUnicode, One Code Page For All Scripts

Unicode Character FormatsUnicode Character Formats


Unicode Character FormatsUnicode Character Formats

Representation of Unicode characters:

UTF-16: Unicode Transformation Format, 16 bit Encoding

- Fixed Length, 1character = 2 bytes

- Platform dependent BYTE order

- 2 BYTE alignment restriction


- Variable length, 1character = 1 to 4 BYTES

- Platform Independent

- No alignment restriction

- 7 BIT US ASCII compatible

Unicode Character Formats (contd.)Unicode Character Formats (contd.)


• For single characters, 32-bit integer variables are most appropriate for the value range of Unicode.

• For strings, however, storing 32 bits for each character takes up too much space, especially considering that the highest value 0x10FFFF, takes up only 21 bits. 11 bits are always unused in a 32-bit word storing a Unicode code point.

• Therefore, you will find that software generally uses 16-bit or 8-bit units as a compromise, with a variable number of code units per Unicode code point.

Unicode Character Formats (contd.)Unicode Character Formats (contd.)

SAP Pre-Unicode solutionsSAP Pre-Unicode solutions

Unicode @ SAPUnicode @ SAP

SAP Pre-Unicode SolutionsSAP Pre-Unicode Solutions

• Single Code Page SystemSystem using one standard code page which can support a specific

set of languages.

• Blended Code Page System (Release 3.0D)Multi byte blended code pages, which contain characters out of

several standard code pages. Blended code pages are not standard code pages, but SAP-customized code pages that were devised to support an increased number of possible language combinations in a singlecode page.

a) Ambiguous Blended Code Page System: Two characters can share the same code point.

b) Unambiguous Blended Code Page System: Each code point refers exactly to one character.

SAP Pre-Unicode Solutions (contd.)SAP Pre-Unicode Solutions (contd.)

• MDMP System Configuration (Release 3.1I)

Multiple Display/Multiple Processing.

System using more than one system code page on the application server.

Allows languages to be used together in one system although the characters of those languages are not in the same code page.


• Language Combinations Before Unicode

• Each user can only access one code page at a time: a user who logs in as a Japanese user cannot enter German characters ,and all German characters in the database will not be correctly displayed.


• It is possible for a user to log on with German and then manipulate the character set and font settings so that he can enter what appear to be Japanese characters; these characters will not be correctly stored in the database and this data will be corrupt

• If a user wants to enter f.i. Japanese, he/she must log on in Japanese• To insure that no data corruption occurs, the following restrictions must be

followed:Global data must contain only 7-bit ASCII characters, which are in all

code pagesUsers may use only the characters of their log-in language or 7-bit

ASCIIBatch processes must be assigned with the correct user ID and

languageEBCDIC code pages are not supported

View in Different Code PagesView in Different Code Pages

Recommendations From SAP (Pre-Unicode)Recommendations From SAP (Pre-Unicode)

• In general, using a single standard code page for new installations and upgrades is the optimal decision

• If additional languages or language combinations are needed, SAP recommends Unambiguous Blended Code Pages for new installations and MDMP for existing installations.

• Unambiguous Blended Code Pages only support certain language combinations and therefore an MDMP setup may be the only possibility for new installations as well.

Why SAP adopted Unicode?Why SAP adopted Unicode?

Unicode @ SAPUnicode @ SAP

Why SAP adopted Unicode?Why SAP adopted Unicode?

• Globalization = Internationalization + Localization

• The Unicode Standard has already been adopted by industry leaders as Apple, HP, IBM, JustSystem, Microsoft, Oracle, Sun, Sybase, Unisys and many others.

• Unicode is required by modern standards such as XML, Java, ECMAScript (JavaScript), LDAP, CORBA 3.0, WML, etc.,

• It is the official way to implement ISO/IEC 10646.

Why SAP adopted Unicode? (contd.)Why SAP adopted Unicode? (contd.)

• Allows text data from different languages to be stored in one repository

• Enable a single set of source code to be written to process data in virtually all languages

• Simplifies addition of new language support to an e-business application since character processing and storage remains unchanged

• Lowers cost of implementation

• Faster speed to market

• Better customer satisfaction

Why SAP adopted Unicode? (contd.)Why SAP adopted Unicode? (contd.)

View In UNICODE SystemView In UNICODE System

Unicode - SAPUnicode - SAP

Platforms supported by SAP for Unicode systems Platforms supported by SAP for Unicode systems

• The following indicates the current development status and availability different OS and Database combinations for Unicode-based mySAP technical components:

OverviewOverview

Unicode & ABAPUnicode & ABAP

OverviewOverview

• Non-Unicode Versions of SAP :- Versions prior to 4.7

• Unicode Versions of SAP :- Version 4.7 & above

• Each character mapped using 16 bits (= 2 bytes) which offers a maximum of 216 bit combinations

• Affects any older ABAP program in which an explicit or implicit assumption is made about the internal length of a character

Overview (contd.)Overview (contd.)

Character Expansion Model

- Separate Unicode & Non-Unicode versions of R/3

- No explicit Unicode Data type in ABAP

- Single ABAP source for Unicode & Non-Unicode systems

OVERVIEW (contd.)OVERVIEW (contd.)

• Program attributes ‘Unicode checks’ active

- Required to run on a Unicode System

• If attribute is set additional restrictions:

- Apply at run time and compile time

- Apply in Unicode as well as Non-Unicode System.

- Ensure that program will run on both US & NUS with identical behavior

Non-Unicode System

Unicode System

Attribute Set (Unicode enabled)

Ok Ok

Attribute not Set (not Unicode enabled)

Ok Not allowed

Overview (contd.)Overview (contd.)

Screen Shot of Program Attribute ScreenScreen Shot of Program Attribute Screen

Concepts & ConventionsConcepts & Conventions


Concepts & ConventionsConcepts & Conventions

• Data Types• Data Layout of Structures• Unicode Fragment Views

Data TypesData Types

The following data types can be interpreted as Character Type in a Unicode program

C: Character N: Numeric character D: Date T: Time STRING: String Character-type structures: Structures which either directly or in

substructures contain only fields of types C, N, D or T.

Byte Type : X, Xstring (for bit operations)

Data Layout of StructuresData Layout of Structures

• For several data types (like F and I) the memory address should start with multiples of 4 or 8 and for Character-type it should start with multiples 2 or 4 depending on the Unicode representation.

• Within structures, bytes are inserted before or after components with alignment requirements to achieve the necessary alignment.

• Examples:

BEGIN OF struc1, a(1) TYPE X, b(1) TYPE X, c(6) TYPE C,

END OF struc1.

For struc1 there is no alignment gap for NUS or US.

Data layout of structures (Contd..)Data layout of structures (Contd..)

• Examples:

BEGIN OF struc2, a(1) TYPE X,

BEGIN OF struc3, b(1) TYPE X, c(6) TYPE C,

END OF struc3, d TYPE I,

END OF struc2.

Unicode Fragment ViewUnicode Fragment View

• BEGIN OF struc, Unicode Fragment Views: a(2) TYPE C, F1, F2, F3, F4,F5, F6 b(4) TYPE N,

c TYPE D, d TYPE T, e TYPE F, f(2) TYPE X, g(4) TYPE X, h(8) TYPE C, i(8) TYPE C,END OF struc.

Restrictions in UnicodeRestrictions in Unicode


OverviewOverview

• Access Using Offset & Length Addressing

• Assignments

• Casting Data Objects

• Processing Strings in UNICODE

• Determining Length & Distance

• Specifying Key for Table Access

• Database Operations

• Structure Enhancement

• Other changes

Access Using Offset & Length AddressingAccess Using Offset & Length Addressing

• Accessing Single fields:

- Offset or Length based access supported for single fields of Character Type, strings, X, Xstrings

• Accessing Structure:

- This access type results in errors if both character type and non-character type fields are present in the area addressed by the offset and the length

• Passing parameters to subroutines:

- Passing parameters using perform with offset and length specification beyond field boundaries is not allowed in Unicode programming

Access Using Offset & Length Addressing (contd.)Access Using Offset & Length Addressing (contd.)

• Accessing Field Symbols:

- Offset- or length-based access with ASSIGN is only permitted within a predefined range

- ASSIGN feld [+off(len)]TO <f>. : Range corresponds to the field boundaries in case of elementary fields or, in case of flat structures, to the purely character-type starting fragment

- ASSIGN <g>[+off(len)] TO <f>. : Range of the target FS = Range of the Source FS

AssignmentsAssignments

• Conversion between Flat Structures (MOVE):

- The fragments of both structures up to the second-last fragment of the shorter structure are identical

- The last fragment of the shorter structure must be of character or byte-type group

- The corresponding fragment of the longer structure is a character or byte type group with a greater length

Assignments (contd.)Assignments (contd.)

• Conversion between internal tables

- Tables can be converted if their row types are convertible. The restrictions described for structures also apply for the conversion of tables.

• Implicit Conversions:- The same rules also apply to all ABAP statements that use implicit conversions according to the MOVE semantics. APPEND wa TO itab.INSERT wa INTO itab.MODIFY itab FROM wa.READ TABLE itab ...INTO wa. LOOP AT itab INTO wa.

Processing StringsProcessing Strings

• Statements for processing strings:CLEAR ... WITH CONCATENATE CONDENSE CONVERT TEXT ... INTO SORTABLE CODE FIND OVERLAY REPLACE SEARCH SHIFT SPLIT TRANSLATE ... TO UPPER/LOWER CASE TRANSLATE ... USING

• FROM CODEPAGE and FROM NUMBER FORMAT are not allowed with TRANSLATE • The arguments must be single fields of type C, N, D, T, STRING or purely

character-type structures• CONCATENATE a x b INTO c statement is not possible if a, b, and c are character-

type but x has type X.

Processing Strings (contd.)Processing Strings (contd.)

• Comparison operators for processing byte and character strings:

- Operators require single fields of type C, N, D, T, STRING or purely character-type structures as arguments

• CO BYTE-CO CN BYTE-CN CA BYTE-CA

NA BYTE-NA CS BYTE-CS NS BYTE-NS CP BYTE-CP NP BYTE-NP


• Functions for processing byte and character strings:

STRLEN – only for character-type fields and returns the length in characters

XSTRLEN – for finding length of byte strings

NUMOFCHAR - returns the number of characters in a character-type field


• Output in fields and lists:

If the source field is a flat structure in a WRITE statement, it must have character type only, in a Unicode system. This affects the following statements:

WRITE f. WRITE f TO g[+off][(len)].. WRITE (name) TO g.. WRITE f TO itab[+off][(len)] INDEX idx.. WRITE (name) TO itab[+off][(len)] INDEX idx..

Establishing Length & DistanceEstablishing Length & Distance

• NUS – DESCRIBE DISTANCE BETWEEN….. US - Addition of IN BYTE MODE / IN CHARACTER MODE

• NUS – DESCRIBE FIELD….. LENGTH

US - Addition of IN BYTE MODE / IN CHARACTER MODE

ABAP File InterfaceABAP File Interface

• Opening Files:

The OPEN DATASET dsn ... statement must include at least one of the following additions:

... IN TEXT MODE ENCODING ... (UTF-8)

... IN BINARY MODE ... (UTF-8 )

... IN LEGACY TEXT MODE ...(Non Unicode Format)

... IN LEGACY BINARY MODE ... (Non Unicode Format)

ABAP File Interface (contd.)ABAP File Interface (contd.)

• Reading & Writing Files:

READ DATASET dsn INTO f : For Reading File

TRANSFER f TO dsn : For Writing into File

• If the file is opened in TEXT MODE, f must be a character-type field ( C, N, D, or T), a string, or a purely character-type structure

• If the file is opened in LEGACY TEXT MODE, or LEGACY BINARY MODE, conversion errors may occur

Other Changes relating to UnicodeOther Changes relating to Unicode

• Bit Statements:

SET BIT i OF f [TO g]. GET BIT i OF f [INTO g].

f O x, f Z x, and f M x .

The field f must be either X or XSTRING in all the above bit operations.

Other Changes relating to Unicode (contd.) Other Changes relating to Unicode (contd.)

• ADD n1 THEN n2 UNTIL nz [ ACCORDING TO sel ] GIVING m [RANGE str ].

ADD n1 THEN n2 UNTIL nz TO m [ RANGE str ]. Operands n1, n2, and nz must be of the same type and length. The distance between nz and n1 must be an integral multiple of the distance

between n2 and n1. The fields n1, n2, and nz must be in one structure. If this is not statically

recognizable, you must use the RANGE str addition to explicitly specify a structure as a valid area.

If it is discovered that the addressed fields are not within the structure specified using the RANGE addition, a syntax or runtime error occurs.

Other Changes relating to Unicode (contd.)Other Changes relating to Unicode (contd.)

• Loops with VARY & varying Additions:

DO ... VARYING f FROM f1 NEXT f2 [ RANGE f3 ]. WHILE ... VARY f FROM f1 NEXT f2 [ RANGE f3 ].

• The fields f, f1, and f2 must be type-compatible with one another • RANGE for valid accesses must be implicitly or explicitly implemented


• Generating Subroutine:

GENERATE SUBROUTINE POOL itab NAME name

The generated program inherits the contents of the Unicode flag of the generating program.

• Saving Programs: INSERT REPORT prog FROM itab. This statement now includes a new addition, UNICODE ENABLING uc, with which the Unicode flag of the inserted report is given the value of uc


• Types and GET/SET PARAMETER:

GET PARAMETER ID pid FIELD f

SET PARAMETER ID pid FIELD f

The field f must be of character type. For non-character type we use IMPORT & EXPORT statements.

Specifying the Key for Tables Accesses Specifying the Key for Tables Accesses

• A syntax or runtime error occurs when you access the database with a generic key unless the key is purely character-type. This affects the following commands:

READ TABLE dbtab ...SEARCH GKEQ ... READ TABLE dbtab ...SEARCH GKGE ... LOOP AT dbtab ...

REFRESH itab FROM TABLE dbtab.

Please note that this statement is Obsolete and should no longer be used.

Database OperationsDatabase Operations

SELECT * FROM dbtab ... INTO wa / INTO TABLE itab ... FETCH NEXT CURSOR c ... INTO wa / INTO TABLE itab.INSERT INTO dbtab ... FROM wa / FROM TABLE itab.UPDATE dbtab ... FROM wa / FROM TABLE itab.MODIFY dbtab ... FROM wa / FROM TABLE itab.DELETE dbtab FROM wa / FROM TABLE itab.

• The fragment views of the work area and the database table must be identical with regard to the length of the database table.

• If the work area is a single field, the field must be character-type and the database table purely character-type

Storing Data Clusters in Database Tables Storing Data Clusters in Database Tables

• Data clusters are not converted when they are migrated from a non-Unicode database to a Unicode system. For this reason, there may be ABAP cluster tables in a Unicode system that contain non-Unicode characters. These characters are automatically converted during each import.

• When the data is exported, any Unicode characters that may be contained in the data objects are stored in a platform-specific format in the Unicode system.

Structure EnhancementsStructure Enhancements

• Problems caused by structure enhancements:

Enhancements change the fragment views and hence affect the checks for assignments and comparisons

• Enhancement classification in ABAP Dictionary:

Structure Enhancements (contd.)Structure Enhancements (contd.)

• Enhancement handling in program check:

Unicode & RFCUnicode & RFC

New ABAP FeaturesNew ABAP Features


Assignments to Field SymbolsAssignments to Field Symbols

• Range:

ASSIGN feld1 TO <fs> RANGE feld2.

Sets the range limits, making it possible to define addresses past field limits

• Increment:

ASSIGN field INCREMENT n TO <fs>

First the range for the access is defined from the length of field and the INCREMENT definition of the range for the access as defined by ASSIGN fld+n*sizeof[field] (sizeof[fld]) TO <f>.

If sy-subrc > 0 then no increment is made to the field symbol

Assignments to Field Symbols (contd.)Assignments to Field Symbols (contd.)

• Casting:

ASSIGN field TO CASTING. ASSIGN field TO <fs> CASTING TYPE type. ASSIGN field TO <fs> CASTING TYPE (typename). ASSIGN field TO <fs> CASTING LIKE fld. ASSIGN field TO <fs> CASTING DECIMALS dec.

Provides different views on a structure with casts on different types.

Treats the contents of a field as a value of another type using a field symbol

field must be at least as long as the type that was assigned to the field symbol, <fs>

If the field symbol type is a deep structure the system checks for the offset and reference component match of the area covered by <fs>

Includes with Group NamesIncludes with Group Names

Creating Data Objects DynamicallyCreating Data Objects Dynamically

• Data Objects:

Creating and Accessing data objects on the heap

Storing Data ClustersStoring Data Clusters

• Using fields of type XSTRING as data containers

Writing data to an XSTRING (EXPORT):

Storing Data Clusters (contd.)Storing Data Clusters (contd.)

• Reading Data from an XSTRING (IMPORT):

• Automatic Conversion of data during IMPORT.

Generic Types for Field Symbols & ParametersGeneric Types for Field Symbols & Parameters

New Classes For UNICODENew Classes For UNICODE

• Character Utilities:

Class CL_ABAP_CHAR_UTILITIES

New Classes For UNICODE (CONTD.)New Classes For UNICODE (CONTD.)

• Conversion Classes:

Tools for Unicode EnablingTools for Unicode Enabling


ABAP Unicode Scan Tool UCCHECKABAP Unicode Scan Tool UCCHECK

• Transaction UCCHECK is used to examine a Unicode program set for syntax errors without having to set the program attribute "Unicode checks active" for every individual program.

• From the list of Unicode syntax errors, one can go directly to the affected programs and remove the errors.

• It is also possible to automatically create transport requests and set the Unicode program attribute for a program set.

Transaction UCCHECKTransaction UCCHECK

ABAP Coverage AnalyzerABAP Coverage Analyzer

• Coverage Analyzer (transaction SCOV):

• Persistently traces the execution of all program objects within one system.

• Traces all processing blocks i.e. forms, methods, modules and ABAP events.

• Collects information– number of calls– number of runtime errors– number of program changes

Transaction SCOVTransaction SCOV

THANK YOUTHANK YOU

SAP UNICODE

Documents

Transcript of SAP UNICODE