Converting the Oracle Database Character Set

Information Technology Services DBA-OLTP

CATALINA MARKETING ▪ 200 CARILLON PARKWAY ▪ ST. PETERSBURG FL 33716

Converting the Oracle Database Character Set

Using CSSCAN and CSALTER Tools

Gus A. Miklos

Senior Database Administrator

Information Technology Services


Catalina Marketing Corp. June 5, 2015

Table of Contents Overview ....................................................................................................................................................... 4

Installing the CSSCAN Tool ............................................................................................................................ 4

Uninstall CSSCAN ...................................................................................................................................... 5

Verify CSSCAN Installation ........................................................................................................................ 5

Verify Current Character Set ..................................................................................................................... 5

Character Set Conversion Process ................................................................................................................ 6

Process Overview ...................................................................................................................................... 6

Understanding CSSCAN Output ................................................................................................................ 7

Tablespace Expansion ........................................................................................................................... 7

Database Scan Summary ....................................................................................................................... 7

Data Dictionary Conversion Summary .................................................................................................. 8

Distribution of Convertible, Truncated and Lossy Data by Table ......................................................... 8

Application Data Details ........................................................................................................................ 9

Pre-CSSCAN Clean Up Process ...................................................................................................................... 9

Clean Up Invalid Objects ........................................................................................................................... 9

Drop Orphaned Data Pump Jobs ............................................................................................................ 11

Purge the DBA Recyclebin ....................................................................................................................... 11

Remove Sample and/or Demo Schemas and Objects ............................................................................ 11

Run CSSCAN ................................................................................................................................................ 12

Clean Up Data Dictionary Objects ............................................................................................................... 13

SYS.COM$(COMMENT$) ......................................................................................................................... 13

SYS.HISTGRM$(EPVALUE) ....................................................................................................................... 14

SYS.REG$(SESSION_KEY) ......................................................................................................................... 15

SYS.SOURCE$(SOURCE) ........................................................................................................................... 15

Run CSSCAN for Final Check of Data Dictionary ..................................................................................... 16

Clean Up Application Data .......................................................................................................................... 17

Modify Columns That Will Expand on Conversion .................................................................................. 17

Clean Up Rows/Columns That Contain Lossy Data ................................................................................. 18

Run Final CSSCAN Check ............................................................................................................................. 19

Generate List of Tables to be Exported ...................................................................................................... 20

Export Convertible Table Data .................................................................................................................... 22



Dealing with XML Datatypes ................................................................................................................... 22

Run Data Pump Export ............................................................................................................................ 24

Disable Constraints and Triggers ................................................................................................................ 24

Create Schema List Table ........................................................................................................................ 25

Disable Foreign Key Constraints ............................................................................................................. 25

Disable Primary (and Unique) Key Constraints ....................................................................................... 25

Disable Triggers on Exported Tables ....................................................................................................... 26

Truncate the Data in the Problematic Tables ............................................................................................. 26

Run CSSCAN for Final Check ........................................................................................................................ 26

Run CSALTER to Convert Character Set ...................................................................................................... 26

Run Post-Conversion Checks ....................................................................................................................... 27

Run Post-Conversion Query .................................................................................................................... 27

Rebuild Catalog and Compile Objects ..................................................................................................... 28

Perform Data Pump Import ........................................................................................................................ 29

Errors During Import ............................................................................................................................... 30

Re-Enable Constraints, Triggers, and Final Clean-Up .................................................................................. 30

Enable Primary Key Constraints .............................................................................................................. 30

Enable Foreign Key Constraints .............................................................................................................. 30

Enable Triggers ........................................................................................................................................ 31

Reset Run Stats History Retention and Run Database Statistics ............................................................ 31

Reset Column Comments........................................................................................................................ 31

Recreate Special Objects ......................................................................................................................... 32

Drop Intermediate Tables ....................................................................................................................... 32

Conclusion ................................................................................................................................................... 32

Appendix A .................................................................................................................................................. 34



Overview This document covers the steps required to convert an Oracle database from the Western European (or

any) character set to AL32UTF8. AL32UTF8 is a more universal character set that supports most

languages of the world including, for example, Kanji.

Appendix A lists the pertinent Oracle Support document IDs used to perform all of the conversion steps.

Please refer to this documentation, since various databases will have much different data and,

therefore, will require modifications to the overall conversion process covered in this document.

NOTE: Changing the Oracle character set requires a scheduled outage of the database. The length of the outage will depend on the amount of data that must be converted. For best results, run CSSCAN with the database open in RESTRICTED mode. This will produce a more accurate scan as well as ensuring that other sessions are not modifying data that will then later become a problem during conversion. In order to run the actual CSALTER procedure, the database must be in RESTRICTED mode and the only active session must be SYS (as SYSDBA). To place the database in restricted mode, perform a shutdown/startup. Once the database is back up: ALTER SYSTEM ENABLE RESTRICTED SESSION.

TIP: The entire set of SQL and shell scripts needed to perform all steps for character set migration are included on the DBA-OLTP SharePoint site in the Oracle -> Scripts section. They are contained in a Unix TAR archive, so that you can copy and extract them for use on any Oracle Database server. Obviously, you can modify these scripts as needed, based on the requirements for each database on which character set conversion is being performed.

Installing the CSSCAN Tool For Oracle 11g, the CSSCAN tool must be run as SYSDBA using SQL*Plus. The self-contained install script

is in the ORACLE_HOME path: @?/rdbms/admin/csminst.sql

Due to some required modifications, however, there is a version of the install script included in the TAR

file that accompanies this document. The modifications are as follows:

1. Create a dedicated tablespace called CSMIG. It should be approximately 4-8 GB, depending on

the size of the database being converted.

2. The CSMINST script should use the new, dedicated tablespace and not the SYSTEM tablespace:

So after extracting the TAR file to your local directory, simply execute the following to install CSSCAN:

$ sqlplus /nolog

SQL> conn / as sysdba

SQL> set TERMOUT ON

SQL> set ECHO ON

SQL> spool csminst.log

SQL> @csminst.sql



Check the installation log for any errors: > grep –i ora- csminst.log

There should be no errors (of course, you can also view the log to make sure there were no issues).

Uninstall CSSCAN Uninstalling CSSCAN is a simple two-step process (log-in as SYSDBA):

drop user CSMIG cascade;

drop tablespace CSMIG including contents and datafiles;

Verify CSSCAN Installation After installing CSSCAN, run a check to make sure the tool is working properly. This is done by running a

scan on one table as shown below (from the OS prompt):

>csscan TABLE='(SYS.SQL_VERSION$)' FROMCHAR=US7ASCII TOCHAR=US7ASCII LOG=dbchk

CAPTURE=N PROCESS=1 ARRAY=1024000

You will be prompted for the SYS password. Below is the entire output:

Character Set Scanner v2.2 : Release 11.2.0.3.0 - Production on Mon Apr 27 15:17:48

2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

Username: SYS as sysdba

Password:

Connected to:

Oracle Database 11g Release 11.2.0.3.0 - 64bit Production

Enumerating table to scan...

. process 1 scanning SYS.SQL_VERSION$[AAAAHXAABAAAA1JAAA]

Creating Database Scan Summary Report...

Creating Individual Exception Report...

Scanner terminated successfully.

As long as you see the last highlighted line above, then CSSCAN is properly installed and ready for use.

Verify Current Character Set Before beginning the process of conversion, verify the current character set for the database by logging

in as SYSDBA and executing the following command:

SQL> SELECT value FROM NLS_DATABASE_PARAMETERS WHERE parameter='NLS_CHARACTERSET';

VALUE

----------------------------------------

WE8MSWIN1252

So for this document, you will convert the database from Western European (WE8MSWIN1252) to

AL32UTF8.



Character Set Conversion Process Character set conversion is largely dependent on the column and data types found within the database.

Most of the columns (and data) may be converted using the CSALTER tool, but some of the columns and

data will not be convertible and will result in either data loss or truncation (or both). The CSSCAN tool

identifies the schemas and tables that are not directly convertible by listing these as either truncation or

lossy. The problem tables may contain either data dictionary or application data, and both types must

be corrected prior to running the CSALTER tool.

Process Overview The following is a summary of the steps required to perform character set migration:

1. CSSCAN prerequisite clean up

a. Check for invalid objects and recompile if necessary

b. Drop invalid synonyms or other objects that cannot be compiled and/or are not being

used

c. Remove orphaned Data Pump master tables

d. Check for leftover temporary tables

e. Purge the recyclebin

f. Remove unneeded sample/demo schemas (SCOTT, etc.)

2. Run the CSSCAN tool and identify problematic objects for both the data dictionary and

applications

3. Clean up the data dictionary objects first (fix lossy or truncation issues)

4. Clean up any identified application data that is flagged as lossy or truncation

5. Run CSSCAN again to make sure the data dictionary and application data are 100% convertible

6. Perform a Data Pump data-only export on all application tables flagged as convertible

7. Disable key constraints and any triggers for the problematic tables

8. Truncate the data in the problematic tables

9. Run the CSALTER tool to migrate the character set

10. Run post-conversion checks and steps

11. Perform a Data Pump data-only import from the dump file in Step 7

12. Re-enable referential constraints and triggers, run database statistics, and perform other clean-

up as needed



Understanding CSSCAN Output CSSCAN will produce three output files. The base file name is specified by the name entered in the LOG

parameter in the CSSCAN command. So in this example, the output files will be:

dbcheck.txt

dbcheck.out

dbcheck.err

The file of primary interest is dbcheck.txt. This file will contain a summary of all of the database objects

that will be convertible, need to be truncated, or experience data loss on conversion. Let’s look at each

section of CSSCAN output.

Tablespace Expansion

The process of character set conversion may increase the column sizes of various tables, so the first

section of the dbcheck.txt file shows how each tablespace may be affected by the column expansion.

Below is a fragment of this section of CSSCAN output.

[Database Size]

Tablespace Used Free Total Expansion

------------------------- --------------- --------------- --------------- ---------------

SYSTEM 654.13M 54.88M 709.00M 2.42M

UNDOTBS_1 9.94M 704.06M 714.00M .00K

TEMP .00K .00K .00K .00K

TOOLS 23.28M 80.72M 104.00M .00K

USERS 7,053.17M 351.91M 7,405.09M 71.36M

AUDIT_TS 1,388.38M 86.63M 1,475.00M 932.00K

.

.

.

CSMIG 22.38M 8,169.63M 8,192.00M .00K

------------------------- --------------- --------------- --------------- ---------------

Total 123,176.91M 20,760.93M 143,937.84M 706.98M

As you can see, the USERS tablespace will grow by 71.36 MB. In this case, there are 351.91 MB of free

space available, so this tablespace will handle the character set migration. There may, however, be

other tablespaces without enough free space to allow for the expansion. In that case, you’ll need to

allocate the additional space needed prior to executing CSALTER.

Database Scan Summary

The next few sections of CSSCAN output will summarize the findings. First, as shown below, it verifies

the parameters being used for CSALTER conversion.

[Database Scan Parameters]

Parameter Value

------------------------------ ------------------------------------------------

CSSCAN Version v2.1

Instance Name dbatest

Database Version 11.2.0.3.0

Scan type Full database



Scan CHAR data? YES

Database character set WE8MSWIN1252

FROMCHAR WE8MSWIN1252

TOCHAR AL32UTF8

Scan NCHAR data? NO

Array fetch buffer size 1024000

Number of processes 4

Capture convertible data? YES

------------------------------ ------------------------------------------------

The Scan Summary below shows the most critical information. In order to run CSALTER for both

dictionary and application objects, CSSCAN must report that all character type data is convertible. As

you can see below, this is not the case, so it will be necessary to perform all of the steps outlined earlier.

[Scan Summary]

Some character type data in the data dictionary are not convertible to the new character set

Some character type application data are not convertible to the new character set

Data Dictionary Conversion Summary

In the next section, CSSCAN shows the percentage of data dictionary data types and their corresponding

state for character conversion. In order for CSALTER to work, there must be zero Truncation or Lossy

columns. At the end of this section, CSSCAN reports this fact.

[Data Dictionary Conversion Summary]

Data Dictionary Tables:

Datatype Changeless Convertible Truncation Lossy

--------------------- ---------------- ---------------- ---------------- ----------------

VARCHAR2 14,886,618 203 12 2

CHAR 17,970,479 0 0 0

LONG 208,837 0 0 0

CLOB 65,980 6,716 0 0

VARRAY 14,151 0 0 0

--------------------- ---------------- ---------------- ---------------- ----------------

Total 33,146,065 6,919 12 2

Total in percentage 99.979% 0.021% 0.000% 0.000%

The data dictionary cannot be safely migrated using the CSALTER script

Distribution of Convertible, Truncated and Lossy Data by Table

The final section of CSSCAN output provides all of the details for convertible, truncated, or lossy columns

by table. Below is a fragment of this output.

[Distribution of Convertible, Truncated and Lossy Data by Table]


USER.TABLE Convertible Truncation Lossy

-------------------------------------------- ---------------- ---------------- ----------------

SYS.COM$ 1 0 0

SYS.EXTERNAL_TAB$ 2 0 0

SYS.HISTGRM$ 76 0 0

SYS.METASTYLESH 179 0 0

SYS.REG$ 0 0 1



SYS.RULE$ 1 0 0

SYSTEM.MASTER_CLU_T_TEST 41 12 0

.

.

.

The first highlighted row above is showing one column in SYS.REG$ that contains lossy data. The second

highlighted row will be truncated upon conversion. Both of these findings must be repaired before

conversion can continue.

TIP: Notice that the second highlighted row appears to be an application test table created in the SYSTEM schema. This table very likely may be safely dropped. Wherever possible, these types of objects (e.g., backup, test, or temporary) that CSSCAN is reporting as problematic for conversion may be dropped. Check with the application team(s) before dropping any objects. (Always ensure you have a recent database backup before dropping objects and also, just prior to running CSALTER.)

Application Data Details

This section of the output is very similar to the data dictionary section. It shows all of the application

tables that contain problem columns.

Application Data:

USER.TABLE Convertible Truncation Lossy

--------------------------------------------- ---------------- ---------------- ----------------

AS_SCHEDULER.ASAPP_CBD_PARMS 27 0 0

.

.

.

.

MASTER.UPC_DEL_T 0 16 0

MASTER.UPC_MKT_T 6 0 0

MASTER.UPC_T 307 450 0

MASTER.UPC_T_BKP 446 514 0

MASTER.USER_T 22 0 13

Again, this output shows tables with both truncation and lossy issues. Also note, as the tip above

indicated, there is one table that appears to be a backup and very likely may be dropped.

As noted in the Process Overview section above, application tables that show truncation or lossy

columns must be corrected before using Data Pump (data-only) export. In other words, all application

data must be identified as convertible by CSSCAN.

Pre-CSSCAN Clean Up Process The following sections cover the step-by-step procedures to clean up the source database in preparation

to run CSALTER character set conversion.

Clean Up Invalid Objects Run chk_invalid_objects.sql and you will get output similar to the following:



Owner Obj Name Obj Type STATUS

-------------------- ------------------------------ ------------------------------ -------

BATCHODM TBLCON SYNONYM INVALID

BATCHODM TBLRESCHAIN SYNONYM INVALID

BATCHODM RESET_SEQUENCES PROCEDURE INVALID

C2C_APP PRICING_PKG_20141125 PACKAGE BODY INVALID

C2C_APP INVOICE_PKG_20140402 PACKAGE BODY INVALID

C2C_BATCH PRICING_PKG SYNONYM INVALID

C2C_BATCH BUS_SEG_CURRENT_V SYNONYM INVALID

.

.

.

PUBLIC PATH SYNONYM INVALID

PUBLIC ABSPATH SYNONYM INVALID

PUBLIC ALL_PATH SYNONYM INVALID

PUBLIC CDS_PKG SYNONYM INVALID

RADAR_APP TBLCON SYNONYM INVALID

ROBER003 TBLRESCHAIN SYNONYM INVALID

ROBER003 BILL_T SYNONYM INVALID

ROBER003 TBLCON SYNONYM INVALID

The first step is to recompile or drop any invalid synonyms. You may use the gen_compile_syn.sql

script, which will attempt to compile both application and PUBLIC synonyms. After running this, you may

still see some that are invalid. This is most likely because the objects referred to have errors, for

example:


-------------------- ------------------------------ ------------------------------ -------

PUBLIC CMS_NDCTGT_V SYNONYM INVALID

PUBLIC CMS_NDCTGT_NDC_V SYNONYM INVALID

DBATEST> select * from CMS_NDCTGT_V;

select * from CMS_NDCTGT_V

*

ERROR at line 1:

ORA-04063: view "MASTER.CMS_NDCTGT_V" has errors

To the extent possible, investigate these objects and see if they can be fixed or dropped (i.e., not in use).

Check with the application team(s) and see if the objects (a view in this case) may be fixed or dropped.

For this document, the synonyms for the invalid views above have been dropped. So now, the

chk_invalid_synonyms.sql script returns no rows.

Finally, if there are some other types of objects that are invalid and can’t be dropped, you may still

proceed with character conversion. In some cases, however, a package or package body may not be

convertible. In that case, you will need to save it off and recreate it after character conversion. This will

be covered in a later section.



Drop Orphaned Data Pump Jobs Now, run chk_orphaned_dp_jobs.sql. This should return no rows, but if it does, drop the orphaned

tables. The SQL script will show the owner and the table name. The owner is usually (but not always)

SYS.

Purge the DBA Recyclebin Finally, clear out the DBA_RECYCLEBIN:

DBATEST> select count(*) from dba_recyclebin;

COUNT(*)

----------

76

DBATEST> purge dba_recyclebin;

DBA Recyclebin purged.

DBATEST> select count(*) from dba_recyclebin;

COUNT(*)

----------

0

Remove Sample and/or Demo Schemas and Objects By default, Oracle installs a set of schemas that may not be in use or not needed depending on your

application requirements. Run the script chk_demo_schemas.sql to see if you have any of these on

the source database.

DBATEST> @chk_demo_schemas.sql

USERNAME

------------------------------

DBSNMP

OUTLN

PERFSTAT

ANONYMOUS

DIP

XDB

Obviously, the schemas listed are valid Oracle-owner schemas and should not be dropped.

The link below is an excellent source that tells you what schemas can be safely dropped.

http://www.orafaq.com/wiki/List_of_default_database_users

If possible, drop or remove any unneeded or unused schemas. You should also check the DBA_REGISTRY

to find out what has been installed:

http://www.orafaq.com/wiki/List_of_default_database_users



DBATEST> select COMP_NAME,VERSION,STATUS from dba_registry;

COMP_NAME

--------------------------------------------------------------------------------------

--------------------------------------------

VERSION STATUS

------------------------------ -----------

Oracle XML Database

11.2.0.3.0 VALID

Oracle Database Catalog Views

11.2.0.3.0 VALID

Oracle Database Packages and Types

11.2.0.3.0 VALID

If there are any installed components that are invalid, you should check them, reinstall, or remove them,

again, depending on requirements.

Run CSSCAN Once CSSCAN has been properly installed and the prerequisites above have been done , the first step is

to perform an initial run on the source database. Depending on the size of the database, CSSCAN may

take a long time to run. So it may be best to run it in the background via a shell script. Below is an

example.

#!/usr/bin/ksh

cd $HOME/gmiklos/csscan/bin

csscan \"sys/<password> as sysdba\" FULL=Y FROMCHAR=WE8MSWIN1252 TOCHAR=AL32UTF8

ARRAY=1024000 LOG=dbcheck CAPTURE=Y PROCESS=4

print "$ORACLE_SID Scan Complete" | mailx -s "CSSCAN"

[email protected]

exit

TIP: The PROCESSES parameter specifies how many parallel processes are used for the scan. It is CPU and memory dependent. If you have enough memory and CPUs, you can increase the PROCESSES parameter to speed up the CSSSAN check.

As explained in an earlier section, once you have the CSSCAN output file, check the tablespace section

and allocate additional space as needed.



Clean Up Data Dictionary Objects Run chk_data_dict.sql to see a condensed list of problematic data dictionary objects. Sample

output is shown below.

DBATEST> @chk_data_dict.sql

NOTHANDLEDDATADICTCOLUMNS

-------------------------------------------------------------------------------------

SYS.COM$(COMMENT$) - VARCHAR2 - CONVERTIBLE

SYS.HISTGRM$(EPVALUE) - VARCHAR2 - CONVERTIBLE

SYS.REG$(SESSION_KEY) - VARCHAR2 - DATA_LOSS

SYS.SOURCE$(SOURCE) - VARCHAR2 - CONVERTIBLE

SYS.WRI$_OPTSTAT_HISTGRM_HISTORY(EPVALUE) - VARCHAR2 - CONVERTIBLE

SYS.WRI$_OPTSTAT_HISTGRM_HISTORY(EPVALUE) - VARCHAR2 - DATA_LOSS

SYSTEM.MASTER_CLU_T_TEST(AD_MSG_1_TXT) - VARCHAR2 - CONVERTIBLE

SYSTEM.MASTER_CLU_T_TEST(AD_MSG_1_TXT) - VARCHAR2 - EXCEED_SIZE

SYSTEM.MASTER_CLU_T_TEST(AD_MSG_2_TXT) - VARCHAR2 - CONVERTIBLE

SYSTEM.MASTER_CLU_T_TEST(MCLU_DESCR) - VARCHAR2 - EXCEED_SIZE

As you can see, this shows all dictionary objects that will either exceed column size or suffer data loss

(convertible data may be ignored; for 10g and above, the CSALTER procedure will handle these). Objects

that exceed side or show data loss need to be corrected. Note also, as mentioned earlier, that the

highlighted table above appears to be a test table in the SYSTEM schema that can be dropped.

So as a first step, let’s drop this table.

DBATEST> drop table SYSTEM.MASTER_CLU_T_TEST cascade constraints purge;

Table dropped.

NOTE: CSSCAN will scan any objects in the DBA recyclebin. So you must use the purge command when dropping problematic tables (or other objects).

Now, let’s see how to solve the remaining data dictionary issues. For this, we need to refer to the Oracle

Support documents (specifically, Doc ID 258904.1).

SYS.COM$(COMMENT$) The first problem table in the CSSCAN list is SYS.COM$. This system table contains column comments on

various tables that are not convertible to the new character set. Run chk_sys_comments.sql to

identify the tables on which column comments will need to be cleared (this query may take a while,

since it must check every column by row id).

DBATEST> @chk_sys_comments.sql

OWNR ONAME

------------------------- ------------------------------------------------------------

MFGAWDS_APP INCENTIVE_BLIND_T



Obviously, you will want to save off the comments and add them back to the table after migration. You

can do this using the gen_save_sys_col_comments.sql script. You must pass in the schema owner and

table name. This will generate a SQL script that will reset the comments after conversion. It appends the

name of the table to the generated SQL script. So, for example:

DBATEST> @gen_save_sys_col_comments.sql MFGAWDS_APP INCENTIVE_BLIND_T

This will produce a new script called add_comments-INCENTIVE_BLIND_T.sql that you can use after

conversion to add the column comments back to the table.

Now that the column comments for this table are saved off, clear the column comments using the

gen_clear_sys_col_comments.sql script.

DBATEST>@gen_clear_sys_col_comments.sql MFGAWDS_APP INCENTIVE_BLIND_T

Do this for all of the tables and columns listed in the initial check.

SYS.HISTGRM$(EPVALUE) For issues with this dictionary item, perform the following:

Check the current retention of statistics

The default value is 31 select systimestamp - dbms_stats.get_stats_history_availability from dual;

Now disable statistics retention exec dbms_stats.alter_stats_history_retention(0);

Remove all statistics exec DBMS_STATS.PURGE_STATS(systimestamp);

Check there are no more left select count(*) from SYS.WRI$_OPTSTAT_HISTGRM_HISTORY;

This should clear SYS.WRI$_OPTSTAT_HISTGRM_HISTORY(EPVALUE)

After the character set change you will set the retention back to the original value exec dbms_stats.alter_stats_history_retention(31);

Now check if there are any table histogram statistics that are not convertible by running

analyze_histgrm.sql. If this returns rows for either CONVERTIBLE DATA or EXCEPTIONAL DATA, then

you must delete the histogram statistics for each table. Use the gen_del_hist_stats.sql script to

generate a dynamic script that will delete the statistics for these problem tables

(run_clr_tbl_histgrm.sql).

Re-run the analyze script and it should now return no rows.

DBATEST> @analyze_histgrm.sql

Table: SYS.HISTGRM$

Error: CONVERTIBLE DATA



no rows selected

Table: SYS.HISTGRM$

Error: EXCEPTIONAL DATA

no rows selected

NOTE: After running CSALTER to migrate to the new character set, you must gather database statistics: EXEC DBMS_STATS.GATHER_DATABASE_STATS(DEGREE=> 'x') -- where x is the number of CPU cores. Also, be sure to reset the stats history retention to the original value (31).

SYS.REG$(SESSION_KEY) If this item shows up in the CSSAN report, it is most likely due to a session being reported other than

SYSDBA. As mentioned previously, CSSCAN should be run in restricted mode (as SYS only). You can check

the type of connection that was registered by running the chk_sys_reg.sql query. For example:

DBATEST> @chk_sys_reg.sql

SUBSCRIPTION_NAME LOCATION_NAME usr# namespace

-------------------- ------------------------------ ----- ---------

REG_TIME

---------------------------------------------------------------------------

"SYS"."ALERT_QUE":"N net8://(ADDRESS=(PROTOCOL=tcp) 19 1

GTESTCM4_CA_3872_ITO (HOST=10.22.3.139)(PORT=47632)

CUSQA3" )?PR=0

24-APR-15 10.58.40.556612 AM -04:00

In this case, there is only one row and, by querying DBA_USERS on user_id=19, it is an old entry (see the

timestamp) from a connection by the OEM agent:

DBATEST> select username from dba_users where user_id=19;

USERNAME

------------------------------

DBSNMP

NOTE: If the OEM agent is running on the server on which the target database resides, then if possible, stop the agent until character set migration is complete. If you cannot shut down the OEM agent, then you must remove the database from the OEM targets until character set conversion has been successfully completed.

In our example, since the database was shut down and then restarted in restricted mode, this table can

be safely truncated as this connection is no longer valid.

SYS.SOURCE$(SOURCE) This item is referencing an object (usually a package, package, body, procedure, etc.) that cannot be

converted to the new character set (usually attributed to the fact that there are literals being used in the



object that are not covertible). So first, find the row id in the dbcheck.err file for the object. Use the vi

(or other) Unix editor and search for SOURCE$. You should see a section similar to the following:

User : SYS

Table : SOURCE$

Column: SOURCE

Type : VARCHAR2(4000)

Number of Exceptions : 0

Max Post Conversion Data Size: 4000

ROWID Exception Type Size Cell Data(first 30 bytes)

------------------ ------------------ ----- ------------------------------

AAAABHAABAAAHh+ABn convertible 'Expédition des

AAAABHAABAAAHi3AA/ convertible || 'Le numéro d

AAAABHAABAAAHi3AA2 convertible || 'Nous avons

Now run the chk_src_rowid.sql query passing in the first row id and this will give the owner, object

name and object type:

DBATEST> @chk_src_rowid.sql AAAABHAABAAAHh+ABn

75452

OWNR ONAME OTYPE STATUS

-------------------- ------------------------------ -------------------- -------

CIMS_APP BATCH_FILE_PKG PACKAGE BODY VALID

As you can see, this is a package body in the CIMS_APP schema. Check with the applications team(s) to

find out if this package is in use or if it can be dropped. Alternatively, use TOAD or SQL*Developer to

save off the source script, so you can recreate the object after character set conversion.

The final step is to check for data dictionary (source) corruption. Run the

chk_orphaned_obj_source.sql script. This should return no rows. If this query returns any rows then

there are entries in SOURCE$ that are not mapped to an existing object. In other words, the obj# in

SYS.SOURCE$ contains object id's that do not exist in DBA_OBJECTS. Double check that the obj# is really

not known in DBA_OBJECTS and log an Oracle SR for the database "Dictionary Corruption" team to solve

the orphaned rows.

Run CSSCAN for Final Check of Data Dictionary Once the above steps have been completed, run another scan to check dictionary objects. If the above

steps have been successful, you should then see the following in the CSSCAN output:

[Scan Summary]

All character type data in the data dictionary are convertible to the new character set

Some character type application data are not convertible to the new character set






--------------------- ---------------- ---------------- ---------------- ----------------

VARCHAR2 2,899,270 0 0 0

CHAR 158,624 0 0 0

LONG 208,147 0 0 0

CLOB 404 443 0 0

VARRAY 14,151 0 0 0

--------------------- ---------------- ---------------- ---------------- ----------------

Total 3,280,596 443 0 0


The data dictionary can be safely migrated using the CSALTER script

NOTE: For Oracle 10g and up no further action is required on the data dictionary for convertible CLOB data. The CSALTER procedure will successfully convert CLOB columns to the new character set.

So now that the data dictionary is ready for conversion, you may proceed to prepare application data for

conversion.

Clean Up Application Data The primary goal of this step is to reduce the number of tables that will need to be exported and

imported for character set conversion.

Modify Columns That Will Expand on Conversion The first step is to identify and modify table columns that will exceed their size when the character set is

converted. Once you identify the problematic columns, you can modify them and thus eliminate the

need to export the owning table.

Run the chk_column_expansion.sql script to identify the columns that need expansion. A portion of

the output is shown below.

DBATEST> @chk_column_expansion.sql

output is:

owner.table_name column_name - column_type - max expansion in Bytes.

....................................................................

CHKDIR.MTRX_BRND_T (BRND_DESCR) - VARCHAR2 - 51 Bytes .

CHKDIR.MTRX_T (CAT_SEG_DESCR) - VARCHAR2 - 34 Bytes .

CHKDIR.MTRX_T (CLIENT_NM) - VARCHAR2 - 32 Bytes .

CHKDIR.MTRX_T (MTRX_DESCR) - VARCHAR2 - 68 Bytes .

CHKDIR.MTRX_T (SPCL_INSTRUCT_DESCR) - VARCHAR2 - 66 Bytes .

CMCPORTAL.RETAILPROGRAMENTRYDATA_T (APPENDED_DESCR) - VARCHAR2 - 22 Bytes .

CRMS.TBLCALL (NOTES) - VARCHAR2 - 257 Bytes .

CRMS.TBLCOUP (DESCR) - VARCHAR2 - 32 Bytes .

CRMS.TBLERROR (ERROR) - VARCHAR2 - 257 Bytes .

So if you check the current column size for the CAT_SEG_DESCR column of CHKDIR.MTRX_T, you see

that it is currently 30 bytes. And CSSCAN is showing that this needs to be expanded to 34 bytes.



So now, for all the columns listed in the above output, you need to modify each column to the

appropriate size.

You can use the gen_modify_column_size.sql script to create dynamic SQL that will perform the

required column modifications.

Clean Up Rows/Columns That Contain Lossy Data Now you must identify rows (by rowid) that will suffer data loss on conversion. The rows are identified

in the dbcheck.err output file produced by CSSCAN. This is usually a large file, since it also includes all

convertible data. To first get an idea of how many rows are lossy, perform a grep operation as follows:

(dbatest)> grep -i "lossy conversion" dbcheck.err

AAAJ3SABsAAAAL/AAx lossy conversion 93

AAAJ3SABsAAAAL/ABo lossy conversion ( (ÈU

AAAJ3SABsAAAAMBABv lossy conversion "!AAJ3SABsAAAAMAABK lossy conversion 39 /ßRå

AAAJ3SABsAAAAMCABa lossy conversion 3#s conversion &.

AAAJ3SABsAAAAMEABP lossy conversion 93 0014

AAAJ3SABsAAAAL/AAf lossy conversion "3#Ê

AAAJ3SABsAAAAL/ABo lossy conversion (

AAAJ3SABsAAAAMAABK lossy conversion =PV

AAAJ3SABsAAAAMCAA0 lossy conversion !

In this case, there are not a lot of rows involved, so it may be possible to convert the problem columns in

these rows to the new character set. Use the gen_convert_lossy_rows.sql script to convert the

problem columns. A sample of the run script is shown here:

update "MASTER".USER_T set PREV_PSWD_TXT = (select convert

(PREV_PSWD_TXT,'WE8MSWIN1252','AL32UTF8') from "MASTER".USER_T where rowid =

'AAAJ3SABsAAAAL/AAf') where rowid ='AAAJ3SABsAAAAL/AAf';

This should clean up the majority of lossy rows. In some cases, however, the data in a particular row

may not be convertible to the new character set:

DBATEST> update "MASTER".USER_T set CURR_PSWD_TXT = (select convert

(CURR_PSWD_TXT,'WE8MSWIN1252','AL32UTF8') from "MASTER".USER_T where rowid =

'AAAJ3SABsAAAAMCAAJ') where rowid ='AAAJ3SABsAAAAMCAAJ';

update "MASTER".USER_T set CURR_PSWD_TXT = (select convert

(CURR_PSWD_TXT,'WE8MSWIN1252','AL32UTF8') from "MASTER".USER_T where rowid =

'AAAJ3SABsAAAAMCAAJ') where rowid ='AAAJ3SABsAAAAMCAAJ'

*

ERROR at line 1:

ORA-12703: this character set conversion is not supported

There are not many options at this point. The first is to check the data and see if the row can be deleted

or the lossy column modified. In this particular case, we see the following data for this rowid:

DBATEST> select * from master.user_t where rowid='AAAJ3SABsAAAAMCAAJ';

USER_CD USER_LAST_NM USER_FIRST_NM A

------------------------------ ------------------------------ ------------------------------ -



EMAIL_ADDR COUNTRY_NBR CURR_PSWD_TXT

PREV_PSWD_TXT PSWD_CHG_

------------------------------------------------------------ ----------- -------------------- ---

----------------- ---------

LOCALE_ID_NBR

-------------

chasek Kearl Chase N

Chase Kearl 1 &?.?/?R?

"?/?>EF 26-FEB-01

1033

In this case, it appears that this is a password field. But note that it has not changed since Feb 2001. So it

may be possible to simply delete the row or update the column. In either case, this data will be lost on

conversion, so you may need to investigate further and engage application team(s) to make an informed

decision on next steps. For the purposes of this document (and since this is a test database), we have set

the column to a NULL value.

Run Final CSSCAN Check Once all of the lossy application data has been dealt with, run CSSCAN again to be sure that the

database is now 100% convertible to the new character set. (It may be necessary to re-run some of the

previous steps if you still see lossy or truncation issues—primarily, this is usually due to not finding all

columns that may need to be expanded. )

The final CSSCAN run is now showing exactly what we need to see in order to run the CSALTER

procedure.

[Database Scan Parameters]

Parameter Value

------------------------------ ------------------------------------------------

CSSCAN Version v2.1

Instance Name dbatest

Database Version 11.2.0.3.0

Scan type Full database

Scan CHAR data? YES

Database character set WE8MSWIN1252

FROMCHAR WE8MSWIN1252

TOCHAR AL32UTF8

Scan NCHAR data? NO

Array fetch buffer size 1024000

Number of processes 8

Capture convertible data? YES

------------------------------ ------------------------------------------------

[Scan Summary]

All character type data in the data dictionary are convertible to the new character set

All character type application data are convertible to the new character set






--------------------- ---------------- ---------------- ---------------- ----------------

VARCHAR2 2,897,421 0 0 0

CHAR 158,712 0 0 0

LONG 208,112 0 0 0

CLOB 720 816 0 0

VARRAY 14,151 0 0 0

--------------------- ---------------- ---------------- ---------------- ----------------

Total 3,279,116 816 0 0


The data dictionary can be safely migrated using the CSALTER script

As the CSSCAN report shows, all remaining data dictionary and application data can be safely migrated

using CSALTER. Nevertheless, it is still necessary to export all of the data that is convertible and then

import it back into the database once the character set migration is completed. As stated in the Oracle

documentation:

It is MANDATORY to export and truncate/delete ALL "Convertible" User / Application Data.

Generate List of Tables to be Exported In order to get the list of tables that need to be exported, execute the create_export_tbl.sql script.

This will create an export_list table in the SYS schema that you will use for other steps in this

document. Now, generate a list of all tables that need to be exported (export_tbl.lst) in a format that

you can use in the export dump parfile. Execute gen_export_tbl_list.sql to produce the table list.

Cut and paste this output into the parfile. Below is a fragment of the final parfile.

USERID='/ as sysdba'

TABLES=(AS_SCHEDULER.ASAPP_CBD_PARMS,

AS_SCHEDULER.GJF_CHERYL1,

AS_SCHEDULER.NTR_IMPORT_MFG_EMAILS,

AS_SCHEDULER.PMDB_PROJECTS,

AS_SCHEDULER.PMDB_PROJECTS_0917,

AS_SCHEDULER.PRD_PROGRAMMCLUS,

AS_SCHEDULER.PRD_PROGRAMMCLUS_PREFIX,

AS_SCHEDULER.PRD_PROGRAMMCLUS_PREFIXM,

AS_SCHEDULER.PRD_PROGRAMNOTES,

.

.

.

RADAR.MS_RPT_OFFER_MSG_T,

RADAR.RADAR_RPT_NM_T,

RTLDIR.MATRIX_BRAND_T,

RTLDIR.MATRIX_T)

EXCLUDE=STATISTICS

CONTENT=DATA_ONLY

DIRECTORY=CSSCAN_DIR

DUMPFILE=expdp_csscan_tbls.dmp

JOB_NAME=expdp_csscan

LOGFILE=expdp_csscan_tbls.log



Closely examine the export table list for tables that can be dropped. Also, note that the export table list

is generated by querying the CSMIG.CSMV$ERRORS table. Below are some examples of tables that

probably can be dropped (and thus, do not need to be exported).

AS_SCHEDULER.PRD_PROGRAMS_BKP050914

C2C_APP.CON_HIST_T_BAK

MASTER.MEDIA_MSG_AUDIT_T,

MASTER.MEDIA_MSG_AUDIT_T_b35b6adc

The first two tables above are obviously backups. You may want to check with the application team(s) to

see if these can be dropped. If so, there is obviously no need to export them. The highlighted table is

also suspect due to the unusual naming convention. Using SQL*Plus, first check the row count. But note

that depending on how the table was created you may get something like this:

DBATEST> select count(*) from MASTER.MEDIA_MSG_AUDIT_T_b35b6adc;

select count(*) from MASTER.MEDIA_MSG_AUDIT_T_b35b6adc

*

ERROR at line 1:

ORA-00942: table or view does not exist

If you get this error, it’s mostly likely because the table was created using double-quotes and the

DBA_TABLES view won’t find it. (And in fact, Data Pump export will not find it either, so you won’t be

able to export the data). If you specify the owner and table name in double quotes, then it works.

DBATEST> select count(*) from "MASTER"."MEDIA_MSG_AUDIT_T_b35b6adc";

COUNT(*)

----------

104

Finally, check the creation date and/or the last time any DDL changes were made. The DBA_OBJECTS

view will be able to find the object without using double-quotes.

DBATEST> select created, last_ddl_time from dba_objects where owner='MASTER' and

object_name='MEDIA_MSG_AUDIT_T_b35b6adc';

CREATED LAST_DDL_

--------- ---------

23-AUG-11 23-AUG-11

As you can see, this table was created in 2011, so it’s very likely a candidate to be dropped. If, however,

the application team(s) say that the table is in use, then the only alternative is either to rename the

table or use CTAS to recreate the table with proper naming, so that you can export the data. If the table

needs to be renamed or recreated, keep in mind the following:

Oracle Database automatically transfers integrity constraints, indexes, and grants on the old

object to the new object.



Oracle Database invalidates all objects that depend on the renamed object, such as views,

synonyms, and stored procedures and functions that refer to a renamed table.

So use caution with this approach. Once the table is correctly renamed, include it in the parfile for Data

Pump export. Likewise, if the table was dropped, remove it from the parfile.

Export Convertible Table Data It is now time to export the data that CSSCAN is flagging as convertible during character set conversion.

There are, however, some issues that must be handled before running the export.

Dealing with XML Datatypes Oracle 11g has a known bug for tables containing XML datatypes (and in conjunction with character set

conversion). The bug will not appear during the export process, but it may cause the following error

when using Data Pump Import:

ORA-31693: Table data object "MASTER"."MCLU_XML_T" failed to load/unload and is being

skipped due to error:

ORA-29913: error in executing ODCIEXTTABLEFETCH callout

ORA-00600: internal error code, arguments: [klxpr_73], [3], [0], [1], [10025], [101],

[], [], [], [], [], []

NOTE: Bug 8528039 - IMPDP errors / conversion problems importing XMLType data with NLS conversion (Doc ID 8528039.8)

The work-around for this bug is to use the old (9i) exp/imp commands. So first, let’s find the tables from

our export list that contain XML datatype columns. Run the chk_xml_columns.sql script to identify

tables with XML columns

OWNER TABLE_NAME COLUMN_NAME

------------------------------ ------------------------------ ------------------------------

MASTER IMAGE_T IMAGE_XML

MASTER AWARD_XML_T XML_DATA

MASTER OFFER_XML_T XML_DATA

MASTER MCLU_XML_T XML_DATA

MASTER MCLU_LAYOUT_XML_T MCLU_XML

MFGAWDS_APP GPS_BUS_RULES_JOB_T JOB_XML

CDS CDS_JOB_T OUTPUT_XSD

CDS CDS_JOB_T INPUT_XSD

CDS CDS_JOB_REQ_T PARMS_XML

Now run gen_xml_exp_list.sql and this will give you the list of tables (xml_export_tbl.lst) with

XML data types that you can cut and paste into the parfile. You will need to export these tables using

the original exp function using a parfile with the “.dat” extension as follows:

USERID="/ as sysdba"

TABLES=(MASTER.IMAGE_T,

MASTER.AWARD_XML_T,



MASTER.OFFER_XML_T,

MASTER.MCLU_XML_T,

MASTER.MCLU_LAYOUT_XML_T,

MFGAWDS_APP.GPS_BUS_RULES_JOB_T,

CDS.CDS_JOB_T, CDS.CDS_JOB_REQ_T)

ROWS=Y

GRANTS=N

INDEXES=N

CONSTRAINTS=N

STATISTICS=NONE

FILE=exp_xml_tbls.dmp

LOG=exp_xml_tbls.log

Notice that since we’re only interested in the data, we have not exported grants, indexes, constraints, or

statistics. Below is the sample output (you should get no errors).

About to export specified tables via Conventional Path ...

Current user changed to MASTER

. . exporting table IMAGE_T 22835 rows exported

. . exporting table AWARD_XML_T 2860 rows exported

. . exporting table OFFER_XML_T 305 rows exported

. . exporting table MCLU_XML_T 0 rows exported

. . exporting table MCLU_LAYOUT_XML_T 576 rows exported

Current user changed to MFGAWDS_APP

. . exporting table GPS_BUS_RULES_JOB_T 2599 rows exported

Current user changed to CDS

. . exporting table CDS_JOB_T 12 rows exported

. . exporting table CDS_JOB_REQ_T 3872 rows exported

Export terminated successfully without warnings.

After character set migration, you should be able to import the tables using the imp function, again,

using a parfile as follows:

USERID="/ as sysdba"

FROMUSER=MASTER

TOUSER=MASTER

TABLES=(IMAGE_T,

AWARD_XML_T,

OFFER_XML_T,

MCLU_XML_T,

MCLU_LAYOUT_XML_T)

FROMUSER=MFGAWDS_APP

TOUSER=MFGAWDS_APP

TABLES=(GPS_BUS_RULES_JOB_T)

FROMUSER=CDS

TOUSER=CDS

TABLES=(CDS_JOB_T, CDS_JOB_REQ_T)

DATA_ONLY=Y

FILE=exp_xml_tbls.dmp

LOG=imp_xml_tbls.log

And again, the import should show no errors:



Export file created by EXPORT:V11.02.00 via conventional path

import done in US7ASCII character set and AL16UTF16 NCHAR character set

import server uses AL32UTF8 character set (possible charset conversion)

. importing MASTER's objects into MASTER

. . importing table "IMAGE_T" 22835 rows imported

. . importing table "AWARD_XML_T" 2860 rows imported

. . importing table "OFFER_XML_T" 305 rows imported

. . importing table "MCLU_XML_T" 0 rows imported

. . importing table "MCLU_LAYOUT_XML_T" 576 rows imported

. importing MFGAWDS_APP's objects into MFGAWDS_APP

. . importing table "GPS_BUS_RULES_JOB_T" 2599 rows imported

. importing CDS's objects into CDS

. . importing table "CDS_JOB_T" 12 rows imported

. . importing table "CDS_JOB_REQ_T" 3872 rows imported

Import terminated successfully without warnings.

TIP: The gen_export_tbl_list.sql script will not include tables with XML columns, so can use the export_tbl.lst directly in the Data Pump parfiles (for export and import steps).

Run Data Pump Export Find a landing zone on the server for the export dump file(s) and create a directory (if necessary). For

example,

DBATEST> create directory CSSCAN_DIR as ‘/testcm_export/csscan’;

DBATEST> grant read, write on directory CSSCAN_DIR to SYS;

Since you don’t know the exact size of the export, nor how long it will take, you can use a shell script to

run the export in the background. So from the Unix prompt:

>nohup expdp_csscan.sh &

Once the export is completed, check the log for any errors.

TIP: For all succeeding steps in this document, be sure to check the output logs. Each of the scripts provided are spooled off. So for a quick check you can execute the following Unix command: >grep –i ora- <filename.log>

And, of course, you can visually check the logs for errors.

Disable Constraints and Triggers Strictly speaking, only constraints on the problematic tables (and triggers) need to be disabled. But

finding all of the referential dependencies may be time-consuming and cumbersome. The goal here is to

be able to truncate (and later, import) the data into the problematic tables (without running into

integrity errors). For this process, a broad-brush approach is used.



Create Schema List Table Most of the remaining scripts used to prepare the database for character set conversion rely on a

generated table created in the SYS schema that contains all of the application owners that contain table

objects. As SYSDBA, run the create_schema_list.sql script. The main body of this script is as follows:

drop table schema_list purge;

create table schema_list as

select distinct owner from dba_objects

where object_type='TABLE'

order by 1

/

This creates an ordered table (in SYS) of all schemas that own table objects. Obviously, you must now

delete the Oracle-owned schemas from the schema_list table. You want to narrow this down only to

application owners that contain tables. So you must execute a step similar to what is shown below to

delete owners/schemas that are not relevant to the character set migration.

DBATEST> delete from schema_list where owner in (

2 'APPQOSSYS',

3 'CSMIG',

4 'DBSNMP',

5 'OUTLN',

6 'PERFSTAT',

7 'SYS',

8 'SYSTEM',

9 'XDB');

8 rows deleted.

DBATEST> commit;

Commit complete.

Disable Foreign Key Constraints Before you can truncate the tables in the export list, you must disable all referential constraints.

Execute the gen_disable_global_fk.sql script to disable all foreign keys based on the schema list

generated above. By using this broad approach (disabling all constraints for all table object owners), you

will eliminate errors like the following in the next step:

ORA-02297: cannot disable constraint (C2C_APP.CON_T_PK) - dependencies exist

Disable Primary (and Unique) Key Constraints After you have disabled the FK constraints, run the gen_disable_global_pk.sql script to do the same

with the primary (and unique) key constraints. As usual, check the logs for any errors and deal with them

as appropriate. In the process of documenting character set conversion, each of these scripts have been

tested and by using this database-wide approach, the character set migration went smoothly.



Disable Triggers on Exported Tables In this case, it is not necessary to disable all triggers in the database (as it was with the constraints). You

only need to disable the triggers that are related to the owners/tables from the export list table you

created in Step 6. Run the gen_disable_trig.sql script to disable these triggers.

Truncate the Data in the Problematic Tables Run the gen_trunc_tbls.sql to truncate the data in the tables that were marked as convertible by

CSSCAN and that you exported in Step 6. Again, be sure to check the log file for any errors or issues.

Run CSSCAN for Final Check You are now ready to run CSSCAN on final time using the steps outlined in a previous section. Once the

CSSCAN tool has finished, run the csscan_final_chk.sql script. This should now return no rows.

DBATEST> @csscan_final_chk.sql

no rows selected

The database is now ready for character set conversion.

Run CSALTER to Convert Character Set The CSALTER procedure is located in the ORACLE_HOME path at:

@?/rdbms/admin/csalter.plb

The procedure is self-contained, so you can copy it to the directory of your choice (it is included in the

TAR file that comes with this document). To run CSALTER, do the following:

1. SHUTDOWN IMMEDIATE

2. STARTUP

3. You must log-in using the actual SYS password. Do not connect using:

>sqlplus / as sysdba This will not work and CSALTER will fail, because this actually produces

two SQL*Plus sessions. Only one session must be running when CSALTER is executed, so you

must connect as follows:

>sqlplus sys/******** as sysdba

4. ALTER SYSTEM ENABLE RESTRICTED SESSION

5. SQL> @csalter.plb

CSALTER will prompt you to verify that you want to proceed to migrate the character set:



This script will update the content of the Oracle Data Dictionary.

Please ensure you have a good database backup before initiating this procedure.

Would you like to proceed (Y/N)?Y

It will begin by migrating the data dictionary objects and some other output will be displayed. Finally,

you should see the following:

PL/SQL procedure successfully completed.

Alter the database character set...

CSALTER operation completed, please restart database

At this point, you may shutdown and restart the database. When the database is back online, connect as

SYSDBA and verify the character set:

DBATEST> SELECT value FROM NLS_DATABASE_PARAMETERS WHERE parameter='NLS_CHARACTERSET';

VALUE

----------------------------------------------------------------------------------------------------------------------------------

AL32UTF8

So character set conversion is successfully completed. Now you must perform the post-conversion

steps.

Run Post-Conversion Checks First check the Oracle alert log for any unusual errors or messages related to CSALTER. What you should

see is something like the following:

ALTER SYSTEM SET aq_tm_processes=0 SCOPE=MEMORY;

OS Pid: 54657126 executed alter system set events '22838 TRACE NAME CONTEXT LEVEL 1,

FOREVER'

alter database character set internal_use AL32UTF8

Updating character set in controlfile to AL32UTF8

Synchronizing connection with database character set information

Thu May 21 14:18:30 2015

Refreshing type attributes with new character set information

Completed: alter database character set internal_use AL32UTF8

If you see any other messages that indicate a problem, you will need to refer to the Oracle

Documentation referenced in Appendix A.

Run Post-Conversion Query Now run the post_cs_mig.sql script, which will run a final check on all column data types. You should

see the following output: DBATEST> @post_cs_mig.sql



CHARACTERSET TYPES_USED_IN

---------------------------------------- --------------------------------

AL16UTF16 NCHAR

AL16UTF16 NCLOB

AL16UTF16 NVARCHAR2

AL32UTF8 CHAR

AL32UTF8 CLOB

AL32UTF8 LONG

AL32UTF8 VARCHAR2

7 rows selected.

This query should return exactly 7 rows. If this select returns more than 7 rows (for example, 2 different

character sets for VARCHAR2) then there is something wrong and you will have errors in the alert log

during the conversion. Again, in this case, you will need to refer to the Oracle documentation.

NOTE: Even though the output shows AL16UTF16 for N-datatypes, this is correct and expected for AL32UTF8 conversion.

Rebuild Catalog and Compile Objects Character set conversion is basically an upgrade to the database. You must, therefore, rebuild the

catalog, rebuild Oracle packages and procedures, and recompile. Do the following:


2. STARTUP UPGRADE

3. From $ORACLE_HOME/rdbms/admin, run the following (be sure to spool the output): catalog.sql

catproc.sql

utlrp.sql

4. Check the output logs for any ORA- errors. There should be none.


6. STARTUP

Now verify that everything in the DBA_REGISTRY is showing as VALID:

DBATEST> SELECT r.comp_name, r.version, r.status FROM dba_registry r;

COMP_NAME

--------------------------------------------------------------------------------------

--------------------------------------------

VERSION STATUS

------------------------------ --------------------------------------------

Oracle XML Database

11.2.0.3.0 VALID

Oracle Database Catalog Views

11.2.0.3.0 VALID



Oracle Database Packages and Types

11.2.0.3.0 VALID

Finally, run the chk_invalid_objects.sql script and you should not see any SYS or XDB invalid objects:

DBATEST> @chk_invalid_objects.sql


-------------------- ------------------------------ ------------------------------ -------

BATCHODM RESET_SEQUENCES PROCEDURE INVALID

C2C_APP CTRL_PROP_PKG PACKAGE BODY INVALID

C2C_APP PRICING_PKG_20141125 PACKAGE BODY INVALID

C2C_APP INVOICE_PKG_20140402 PACKAGE BODY INVALID

CIMS_APP CRTRDG_ORD_STAT_UPD_TRG TRIGGER INVALID

CMCPORTAL RTLCATAVAILABILITY_P PACKAGE BODY INVALID

MASTER MCLU_SBE_TEST VIEW INVALID

ORDERS_APP SYS_PLSQL_78791_68_1 TYPE INVALID



SSR SYS_PLSQL_129399_271_1 TYPE INVALID

Some of the above objects were invalid before the character set migration and some will become valid

when you complete the post-migration clean-up steps. You may want to point these out to the

application team(s). Otherwise, the database is now prepared for you to run the Data Pump import.

Perform Data Pump Import The scripts that are included with this document contain an import parfile for the dump file from Step 7.

Obviously, you may need to modify this based on the database, but here is a fragment from our test

case:

USERID='/ as sysdba'

TABLES=(AS_SCHEDULER.ASAPP_CBD_PARMS,

AS_SCHEDULER.GJF_CHERYL1,

AS_SCHEDULER.NTR_IMPORT_MFG_EMAILS,

AS_SCHEDULER.PMDB_PROJECTS,

AS_SCHEDULER.PMDB_PROJECTS_0917,

AS_SCHEDULER.PRD_PROGRAMMCLUS,

.

.

.

CONTENT=DATA_ONLY

TABLE_EXISTS_ACTION=TRUNCATE

DIRECTORY=CSSCAN_DIR

DUMPFILE=expdp_csscan_tbls.dmp

JOB_NAME=impdp_csscan

LOGFILE=impdp_csscan_tbls.log

Since we truncated these tables in an earlier step, the TABLE_EXISTS_ACTION parameter may not be

necessary, but it’s included here as best practice. There is a corresponding shell script with which you

can perform the import.



> nohup impdp_csscan.sh &

Errors During Import Although CSSCAN is a good tool, it is not 100% foolproof. Particularly, for data where column expansion

was required. So you may see the following type of errors in the Data Pump import log:

KUP-11007: conversion error loading table "MASTER"."LOCALE_FORMAT_T"

ORA-12899: value too large for column CURRENCY_SYMBOL_TXT (actual: 7, maximum: 5)

KUP-11009: data for row: CURRENCY_SYMBOL_TXT : 0X'2080202020'

As the error implies, CSSCAN missed this table column for column expansion. The solution is:

1. Truncate the table

2. Alter the table and expand the column size at least to the actual size specified

3. Re-run the Data Pump import for this table

There should not be many of these. For our test database (which is a clone of TOCUSQA3), there were

only two tables that had this issue.

Once the Data Pump import has completed successfully, you must then run the additional import using

the old imp command to reload the XML data, as we covered in Step 7: Export Convertible Table Data.

Re-Enable Constraints, Triggers, and Final Clean-Up This is the final phase of character set conversion. You must now re-enable foreign and primary key

constraints and triggers and perform some final clean-up steps.

Enable Primary Key Constraints You must enable primary keys first, before you can enable foreign keys, due to validation. To enable the

primary keys that were disabled in Step 8, run the gen_enable_global_pk.sql script. This will enable

all primary (and unique) keys for all tables based on the schema list created in an earlier step. This may

take a while to run, so use the gen_enable_global_pk.sh shell script to run this in the background.

Once this has completed, check the log for errors (there should be none).

Enable Foreign Key Constraints Once the primary (and unique) key constraints are enabled, run the chk_fk_constraints.sql script to

look at the list of all FK constraints (based on schema_list) and their status. You will see that all FK

constraints are DISABLED and NOT VALIDATED.

Now run gen_enable_global_fk.sql to enable all FK constraints. This could take a while since there

will be many foreign keys and also because Oracle will be validating the FK data. So you may want to run

this in the background using the gen_enable_global_fk.sh shell script.



Once this has completed, execute: grep –i ora- enable_global_fk.log and you may see that some

FK’s could not be validated due to non-existent parent keys:

ORA-02298: cannot validate (C2C_APP.BUS_SEG_T_ACCT_T_FK) - parent keys not found

ORA-02298: cannot validate (C2C_APP.CONTACT_T_CON_CONTACT_T_FK) - parent keys not found

ORA-02298: cannot validate (C2C_APP.BUS_SEG_T_DOC_TYP_T_FK) - parent keys not found

ORA-02298: cannot validate (C2C_APP.BILL_BRND_DESCR_PROD_CD_FK) - parent keys not found

ORA-02298: cannot validate (C2C_APP.BILL_T_SLS_HIER_T_FK) - parent keys not found

ORA-02298: cannot validate (MASTER.SCHEDULE_REQ_ID_NBR_FK) - parent keys not found

ORA-02298: cannot validate (MASTER.STR_INI_IMPORT_STR_NBR_FK) - parent keys not found

In this case some of these FKs should be enabled with the NOVALIDATE option. Run the

gen_enable_global_fk_novalid.sql script to enable FKs without further validation. Then, run the

check script again. You will now see that all FKs are ENABLED, but not all are VALIDATED.

Again, depending on the database, you may want to engage the application team(s) to clean up the

orphaned keys (particularly, if this is Production).

Enable Triggers Now enable the triggers for the tables that were exported/imported for convertible character set data.

Run the gen_enable_trig.sql script to enable the triggers. As always, check the output log for errors

(there should be none).

Reset Run Stats History Retention and Run Database Statistics After character set conversion, set the statistics history retention back to the original value:

EXEC DBMS_STATS.ALTER_STATS_HISTORY_RETENTION(31);

The final step for character set conversion is to the gather database statistics using the DBMS package:

EXEC DBMS_STATS.GATHER_DATABASE_STATS(DEGREE=> 'n');

Where n is the CPU count. Since this can be time-consuming, use the gather_db_stats.sh shell script

to run this in background using nohup.

Reset Column Comments Using the script(s) generated from an earlier step, add the comments back to the respective tables

where comments were cleared. For example:

DBATEST> @add_comments-INCENTIVE_BLIND_T.sql

Comment created.



Recreate Special Objects In an earlier step, you saved off any packages, package bodies, procedures, etc. that would not covert

properly. You must now recreate these objects. For example, we saved off the

CIMS_APP.BATCH_FILE_PKG package, so now run the batch_file_pkg.sql to recreate it. For example:

DBATEST> @batch_file_pkg.sql

Package body created.

DBATEST>

DBATEST> GRANT EXECUTE ON CIMS_APP.BATCH_FILE_PKG TO CIMS_BATCH;

Grant succeeded.

DBATEST>

DBATEST> spool off

DBATEST> quit

TIP: Be sure to SET DEFINE OFF when recreating these objects.

Drop Intermediate Tables The final step is to drop the intermediate tables created in SYS that were used to run the various scripts

for character set conversion.

DBATEST> drop table schema_list purge;

Table dropped.

DBATEST> drop table export_list purge;

Table dropped.

Uninstall CSSCAN Uninstalling CSSCAN is a simple two-step process (log-in as SYSDBA):

drop user CSMIG cascade;

drop tablespace CSMIG including contents and datafiles;

Conclusion Character Set conversion can be a time-consuming and cumbersome process. In addition, the level of

complexity is dependent on the size and type of the data needing conversion. The Oracle tools (and

attached scripts) should make this process somewhat easier. But there will be cases in which you will

need to rely on the documents in Appendix A. The Oracle documentation is very good, but rather dense.

That is why we’ve tried here to capture a step-by-step approach.



Appendix A

Oracle Support Doc ID 745809.1: Installing and configuring CSSCAN in 10g and 11g

Oracle Support Doc ID 260192.1: Changing the NLS_CHARACTERSET to AL32UTF8 / UTF8 (Unicode) in 8i,

9i , 10g and 11g

Oracle Support Doc ID 258904.1: Solving Convertible or Lossy data in Data Dictionary objects reported

by CSSCAN when changing the NLS_CHARACTERSET

Converting the Oracle Database Character Set

Documents

Transcript of Converting the Oracle Database Character Set