8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

download 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

of 100

Transcript of 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    1/100

    Alcatel-Lucent8965

    Convergent ChargingCollection Solution

    Administ rat ion Guide

    Written by :

    C3S team

    Visa and date :

    06/ 07/ 10

    Verif ied by : Visa and date :

    Approved by :

    A. Potier

    Visa and date :

    03/ 01/ 11

    Approved by : Visa and date :

    3AT 21188 AWAA TNZZA 01

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    2/100

    3AT 21188 AWAA TNZZA 2/100

    HISTORY

    Version Date Authors Modif icat ion Pages

    modified added deleted

    01 PR01 06/ 07/ 10 Dupety Antoine Cret ion of document forrelease 4.5

    All

    01 PR02 02/ 08/ 10 Dupety Antoine Update C3S direct ories 2.1.6.X01 03/ 01/ 11 Adolphe Vronique Document approval 1-2

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    3/100

    3AT 21188 AWAA TNZZA 3/100

    Contents

    1 INTRODUCTION TO ADMINISTRATION ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .. 81.1 Presentation.........................................................................................................81.2 Main domains............. ............. .............. ............. .............. ............. ............. ........... 8

    2 C3S CONFIGURATION....... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... . 92.1 C3S server ............ ............. .............. ............. ............. .............. ............. ............. .. 9

    2.1.1 Automatic start-up of Oracle environment at boot time... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 92.1.2 Automatic start-up of C3S server at boot time ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .122.1.3 rsh / rlogin / rexec configuration.... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ..... .... 132.1.4 SNMP configuration..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... ... 172.1.5 C3S server directories.... .... .... ..... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .182.1.6 C3S server logs configuration.. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .. 202.1.7 SFTP configuration .... .... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... 252.1.8 HTTP & HTTPS configuration .... .... .... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... ..262.1.9 C3S server parameters.... .... .... .... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .272.1.10 ASN1 configuration ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... ... 282.1.11 Containers configuration .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... ..... .... .282.1.12 IPSec configuration... .... .... ..... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..... .... .... 292.1.13 Diameter TLS configuration.. .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... 30

    2.2 C3S cl ient .............. ............. ............. .............. ............. ............. .............. ............ 312.2.1 C3S server hostname configuration.. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .312.2.2 HTTP & HTTPS port configuration ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .312.2.3 C3S client logs management .... .... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ...312.2.4 C3S client reconnection ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... ..32

    3 C3S DATABASE MANAGEMENT ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 333.1 Enterprise Manager ............ ............. .............. ............. ............. .............. ............. .. 33

    3.1.1 Enterprise Manager Database Control status.............................................................333.1.2 Enterprise Manager Database Cont roller GUI .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 34

    3.2 Monitoring Oracle logs .............. ............. ............. .............. ............. ............. .......... 353.2.1 Oracle RDBMS .... .... .... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... .353.2.2 Oracle Warehouse Builder... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... ..... .373.2.3 Oracle Workflow ............. .............. ............. ............. ............. ............. ............. ..38

    4

    C3S SUPERVISION .... ..... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..... .... ..... .... .... ..... .... 39

    4.1 Event management ............ .............. ............. ............. .............. ............. ............. .. 394.1.1 Event description fi les.... .... .... .... ..... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... .... 394.1.2 Alarm l i fe cycle ............. ............. ............. ............. ............. ............. ............. ....39

    4.2 JBoss / C3S server log fi les.... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..... .... ..... .... .... ..... .... . 404.2.1 Log configuration fi les.......................................................................................404.2.2 JBoss log fi les ............. .............. ............. ............. .............. ............. ............. ....404.2.3 C3S server log fi les. .... .... .... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ..... .... 40

    5 CLUSTERING MANAGEMENT ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 415.1 MC Service Guard cluster / Duplex configurati on ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 41

    5.1.1 Repair aft er an automatic switch-over... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .. 415.1.2 Switch-back t o t he nominal node .... .... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... .42

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    4/100

    3AT 21188 AWAA TNZZA 4/100

    5.1.3 Maintenance operations. ..... .... .... ..... .... ..... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... .... 435.2 Red Hat Cluster Suite / Duplex configuration .... .... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... . 44

    5.2.1 Repair aft er an automatic switch-over... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .. 445.2.2 Switch-back t o t he nominal node .... .... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... .455.2.3 Maintenance operations for Red-Hat cluster ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .45

    5.3 JBoss cluster ............ .............. .............. ............. .............. ............. .............. ........ 465.3.1 Configuring C3S servers.... .... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... 465.3.2 JBoss server configuration files ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .475.3.3 Configuring C3S client ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... .... 485.3.4 Repair aft er a JBoss server failure... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .485.3.5 Restart the node... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..48

    6 ADMINISTRATION ACTIVITIES. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 496.1 Start / Stop Linux operating system.... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..... ... 49

    6.1.1 Shutdown / Boot levels..... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... ..496.2 Oracle............. ............. ............. .............. ............. ............. .............. ............. ..... 50

    6.2.1 Simplex configuration..... .... .... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .506.2.2 MC Service Guard cluster configuration... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .576.2.3 JBoss cluster configuration .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ...586.2.4 Check Oracle ............ .............. ............. ............. .............. ............. .............. .....596.2.5 Log on Oracle..................................................................................................61

    6.3 C3S server ............ ............. .............. ............. ............. .............. ............. ............. 626.3.1 Simplex configuration..... .... .... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .626.3.2 MC Service Guard cluster configuration... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .646.3.3 Red Hat cluster configuration .... .... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..... ... 706.3.4 JBoss cluster configuration .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ...74

    7 PERIODIC MAINTENANCE PROCEDURE .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. . 757.1 Daily procedure........................ ............. ............. ............. ............. .............. ......... 75

    7.1.1 C3S application supervision ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 757.1.2 C3S database supervision.... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .. 767.1.3 MC/ SG cluster check .... .... .... ..... .... ..... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... .797.1.4 MC/ SG package switch back t o nominal node ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... .. 807.1.5 Red Hat cluster check.... .... .... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..... .... .... ..817.1.6 Red Hat cluster service switch back to nominal node. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .827.1.7 System monitoring.... .... .... .... .... ..... .... .... ..... .... ..... .... .... ..... .... .... ..... .... .... ..... .... ..83

    8 C3S PLATFORM BACKUP. ..... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..... .... ..... .... .... ..... .... .... 858.1 Operating system backup........ .... ..... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..... ... 858.2 C3S application data backup. ..... .... ..... .... ..... .... ..... .... .... ..... .... ..... .... ..... .... ..... .... ..... . 91

    9 ABBREVIATIONS. .... ..... .... ..... ..... .... ..... .... ..... .... ..... .... ..... .... ..... .... ..... .... ..... .... ..... ..... ... 9610 GLOSSARY ............. ............. .............. ............. .............. .............. ............. .............. . 98

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    5/100

    3AT 21188 AWAA TNZZA 5/100

    PREFACE

    Purpose This guide explains how to administrate the Alcatel Convergent Charging CollectionSolution including:

    C3S configurat ion, Clustering management, Administration activities, Preventive maintenance, C3S backup.

    Audience This guide isfor high-level system engineers.

    Contents This guide isorganized in the following manner:

    Chapter 1 Introduction to administrationprovides an overview of system administration and explains the

    concepts you need to master.

    Chapter 2 C3S configurati ongives the procedure t o configure all the C3S components.

    Chapter 3 C3S database managementexplains how to manage the Oracle environment (database, OWB,OWF) required by C3S applicat ion.

    Chapter 4 C3S super visiondescribes the procedures for the C3S applicat ion supervision.

    Chapter 5 Clusteringmanagement explains how t o manage MC/ SG cluster (High Availabil it y cluster)

    and JBoss cluster.

    Chapter 6 Administ rat ion acti vit iesdescribes the most useful administration activities withprocedures to be applied in order to start / stop the system.

    Chapter 7 Periodic maintenance proceduregives the list of periodic procedures to be applied in order tomaintain t he system operat ional (prevent ive maintenance).

    Chapter 8 C3S backupdetails the list of backup procedures for the C3S system : Linuxoperat ing system and C3S applicat ion.

    Abbreviations lists the extended forms of abbreviations used in this guide.

    Glossary defines key words used in this guide.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    6/100

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    7/100

    3AT 21188 AWAA TNZZA 7/100

    TERMINOLOGICAL CONVENTIONS

    This guide applies standard terminology from:

    Global System for Mobil e communications (GSM), General Packet Radio Service (GPRS),

    Universal Mobile Telecommunicat ions System (UMTS).

    This guide uses acronyms developed from the English extended form when available. Certain acronyms maybe French-based when no English extended form exists.

    This guide uses:

    BSS to indicate Business Support System,

    C3S to indicate Convergent Charging Collection Solution,

    CDR to indicate Call Detail Record,

    DPM to indicate Data Processing Module,

    IMS to indicate IP Mult imedia Sub-system,

    NE to indicate Network Element,

    OSS to indicate Operation Support System.

    The extended forms of all abbreviations used appear at the end of this guide.

    Alcatel-Lucent acknowledges the following trademarks for company names or products mentioned withinAlcatel-Lucent documents:

    Oracle is a registered U.S. trademark of Oracle Corporation, Redwood City, California.

    Windows and Windows NT are U.S. registered t rademarks of Microsoft Corporat ion.

    Linux is a registered trademark of the Open Group.

    JBoss is a registered trademark and service trademark of JBoss, Inc.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    8/100

    3AT 21188 AWAA TNZZA 8/100

    1 INTRODUCTION TO ADMINISTRATION

    1.1 Presentation

    This guide is aimed at help operators to administ rate the A8965 C3S syst em (Convergent ChargingColl ect ion Solut ion), in order t o have a syst em working properl y and up to dat e wit h best perf ormances.

    1.2 Main domains

    Operators need to mast er administ rat ion procedures in order to reduce platform down t ime.

    This administration guide covers the foll owings domains:

    C3S configurat ion,

    Operating system start / stop,

    C3S applicat ion start / stop,

    C3S Oracle database management,

    C3S supervision,

    Clustering management (duplex configurat ion managed by MC/ SG cluster and JBoss cluster),

    Administrative activities,

    Preventive maintenance,

    C3S backup.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    9/100

    3AT 21188 AWAA TNZZA 9/100

    2 C3S CONFIGURATION

    2.1 C3S server

    2.1.1 Automat ic start -up of Oracle environment at boot time

    Server(s) C3S Linux serverFuncti onal Domain/Domain Database Administration

    Operation Reference Applicat ion Start -up at boot t ime

    Funct ional Descript ion Start at boot t ime t he Oracle environment required for C3S applicat ion(Oracle database, Listener, Apache Web server)

    Prerequisite Log on the Linux server. This procedure is not applicable on duplexconfiguration (Service Guard).

    Other DocumentApproximate Duration A few minutes

    Frequency If required

    Steps to proceed: Activate at boot time the automatic restart of C3S Oracle database as detailed below. This procedure

    must be performed under the Unix root user account:

    [root@c3sn1] cd /etc/init.d

    [root@c3sn1 init.d]# chkconfig --list | grep dbora

    [root@c3sn1 init.d]# chkconfig --add dbora

    [root@c3sn1 init.d]# chkconfig dbora on[root@c3sn1 init.d]# chkconfig --list | grep dbora

    dbora 0:off 1:off 2:off 3:on 4:off 5:on 6:off

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    10/100

    3AT 21188 AWAA TNZZA 10/100

    Post procedure: Reboot the Linux server:

    [root@c3sn1] shutdown r now

    Check that the Oracle environment (Oracle database, Listener, Apache Web server) is startedautomatically at boot t ime:

    [oracle@c3sn1 oracle]$ ps -eaf | grep oracle

    oracle 4656 1 0 17:39 ? 00:00:00 /oracle/product/10.2/bin/tnslsnr LISTENER

    -inherit

    oracle 4709 1 0 17:39 ? 00:00:00 ora_pmon_C3S

    oracle 4711 1 0 17:39 ? 00:00:00 ora_psp0_C3S

    oracle 4713 1 0 17:39 ? 00:00:00 ora_mman_C3S

    oracle 4715 1 0 17:39 ? 00:00:00 ora_dbw0_C3S

    oracle 4717 1 0 17:39 ? 00:00:00 ora_lgwr_C3S

    oracle 4719 1 0 17:39 ? 00:00:00 ora_ckpt_C3S

    oracle 4721 1 0 17:39 ? 00:00:00 ora_smon_C3S

    oracle 4723 1 0 17:39 ? 00:00:00 ora_reco_C3S

    oracle 4725 1 0 17:39 ? 00:00:00 ora_cjq0_C3S

    oracle 4727 1 0 17:39 ? 00:00:00 ora_mmon_C3S

    oracle 4729 1 0 17:39 ? 00:00:00 ora_mmnl_C3S

    oracle 4733 1 0 17:39 ? 00:00:00 ora_qmnc_C3S

    oracle 4778 1 0 17:39 ? 00:00:00 /bin/sh

    /oracle/owb_10.2/owb/bin/unix/run_service.sh -manual 1 /oracle/owb_10.2 C3SOWBRUN

    LOCALHOST 1521 C3S

    oracle 4780 4778 0 17:39 ? 00:00:05 ../../../jdk/jre/bin/java -Xmx768M -

    Djava.awt.headless=true -DORACLE_HOME=/oracle/owb_10.2 -DOCM_HOME=/oracle/owb_10.2 -

    DOCM_ORACLE_HOME=/oracle/owb_10.2 -classpath

    ../admin:../admin/launcher.jar:Personalties.jar:: -DOWB_HOME=/oracle/owb_10.2 -

    DJDK_HOME=../../../jdk -DORA_HOME=/oracle/owb_10.2 -DOEM_HOME=/oracle/owb_10.2 -

    DIAS_HOME=/oracle/owb_10.2 Launcher ../admin/owb.classpath

    oracle.wh.runtime.platform.service.Service -manual 1 C3SOWBRUN LOCALHOST 1521 C3Soracle 4794 1 0 17:39 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 4796 1 0 17:39 ? 00:00:00 ora_q000_C3S

    oracle 4798 1 0 17:39 ? 00:00:00 ora_q001_C3S

    oracle 4834 1 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/opmn/bin/opmn -d

    oracle 4836 4834 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/opmn/bin/opmn -d

    oracle 4855 4836 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/Apache/Apache/bin/httpd -d

    /oracle/product/10.2/companion/Apache/Apache -U 700579922

    oracle 4858 4855 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/Apache/Apache/bin/rotatelogs

    /oracle/product/10.2/companion/Apache/Apache/logs/error_log 43200oracle 4859 4855 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/Apache/Apache/bin/fcgi- -d

    /oracle/product/10.2/companion/Apache/Apache -U 700579922

    oracle 4860 4855 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/Apache/Apache/bin/rotatelogs

    /oracle/product/10.2/companion/Apache/Apache/logs/access_log 43200

    oracle 4861 4855 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/Apache/Apache/bin/httpd -d

    /oracle/product/10.2/companion/Apache/Apache -U 700579922

    oracle 4863 4855 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/Apache/Apache/bin/httpd -d

    /oracle/product/10.2/companion/Apache/Apache -U 700579922

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    11/100

    3AT 21188 AWAA TNZZA 11/100

    oracle 4864 4855 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/Apache/Apache/bin/httpd -d

    /oracle/product/10.2/companion/Apache/Apache -U 700579922

    oracle 4866 4855 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/Apache/Apache/bin/httpd -d

    /oracle/product/10.2/companion/Apache/Apache -U 700579922

    oracle 4868 4855 0 17:39 ? 00:00:00/oracle/product/10.2/companion/Apache/Apache/bin/httpd -d

    /oracle/product/10.2/companion/Apache/Apache -U 700579922

    oracle 4869 4855 0 17:39 ? 00:00:00

    /oracle/product/10.2/companion/Apache/Apache/bin/httpd -d

    /oracle/product/10.2/companion/Apache/Apache -U 700579922

    oracle 9394 1 0 17:41 ? 00:00:00 ora_p000_C3S

    oracle 9396 1 0 17:41 ? 00:00:00 ora_p001_C3S

    oracle 9398 1 0 17:41 ? 00:00:00 ora_p002_C3S

    oracle 9400 1 0 17:41 ? 00:00:00 ora_p003_C3S

    oracle 14303 4855 0 17:43 ? 00:00:00

    /oracle/product/10.2/companion/Apache/Apache/bin/httpd -d

    /oracle/product/10.2/companion/Apache/Apache -U 700579922

    oracle 18254 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)oracle 18256 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18262 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18264 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18266 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18272 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18274 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18278 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18282 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18287 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18293 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18297 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18303 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18310 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18312 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18318 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18324 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    oracle 18327 1 0 17:46 ? 00:00:00 oracleC3S (LOCAL=NO)

    Check that it is possible to connect to the C3S Oracle database through the listener:

    [root@c3sn1 valide]# su - oracle

    [oracle@c3sn1 ~]$ sqlplus c3sowbtar/c3sowbtar@c3s

    SQL*Plus: Release 10.2.0.1.0 - Production on Thu Apr 24 17:44:42 2008

    Copyright (c) 1982, 2005, Oracle. All rights reserved.

    Connected to:

    Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production

    With the Partitioning, OLAP and Data Mining optionsSQL> exit

    Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 -

    Production

    With the Partitioning, OLAP and Data Mining options

    [oracle@c3sn1 ~]$

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    12/100

    3AT 21188 AWAA TNZZA 12/100

    2.1.2 Automat ic start -up of C3S server at boot t ime

    Server(s) C3S Linux server

    Functi onal Domain/Domain C3S Administ rat ion

    Operation Reference Applicat ion Start -up at boot t ime

    Funct ional Descript ion Start at boot time the C3S server

    Prerequisite Log on the Linux server. This procedure is not applicable on duplexconfiguration (Service Guard).The Automatic start-up of Oracle environment at boot time has beenconfigured.

    Other DocumentApproximate Duration A few minutes

    Frequency If required

    Steps to proceed: Activate at boot time the automatic restart of C3S server as detailed below. This procedure must be

    performed under the Unix root user account:

    [root@c3sn1] cd /etc/init.d

    [root@c3sn1 init.d]# chkconfig --list | grep C3Sctrl

    [root@c3sn1 init.d]# chkconfig --add C3Sctrl

    [root@c3sn1 init.d]# chkconfig C3Sctrl on

    [root@c3sn1 init.d]# chkconfig --list | grep C3Sctrl

    C3Sctrl 0:off 1:off 2:off 3:on 4:off 5:on 6:off

    Post procedure:

    Reboot the Linux server: [root@c3sn1] shutdown r now

    Check that the C3S server Oracle is started automatically at boot time:

    [oracle@c3sn1 oracle]$ ps -eaf | grep java | grep c3s

    c3s 18201 18196 8 17:46 pts/3 00:00:28 /usr/java/jdk1.6.0_03/bin/java

    -server -Xms400M -Xmx1024M -DC3S_HOME=/home/c3s/C3S_Supervision/

    -DC3S_SERVER_HOME=/home/c3s/C3S_Supervision/ -DDF_ROOT=/home/c3s/C3S_Dataflow

    -DLOG_ROOT=/home/c3s/C3S_Log

    -Dc3s.server.properties=/home/c3s/C3S_Supervision//conf/c3s.server.properties

    -DASN1_HEADER_PROPERTIES=

    Check that it is possible to connect to the C3S server through the C3S client GUI.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    13/100

    3AT 21188 AWAA TNZZA 13/100

    2.1.3 rsh / rlogin / rexec confi guration

    Server(s) C3S Linux server

    Functi onal Domain/Domain C3S Administ rat ion

    Operation Reference Remote Application Start-up

    Funct ional Descript ion Start the C3S server in remote access from the Alcatel CMC (CentralManagement Center)

    Prerequisite Linux installed

    Other DocumentApproximate Duration A few minutes

    Frequency If required

    Steps to proceed:This configurati on has to be perf ormed when it is required t o start /stop the C3S server remotely f rom a

    CMC (Central Management Center).

    In order to run an rsh, rlogin, and rexec server, the rsh-server package is required on the Linux server. Itmay be installed from your installation CDs using the rpm command:

    rpm -ivh rsh-server-.rpm

    or using the Red Hat Network Web site.

    Check that the package related to the rsh server is installed on the Linux server: rsh-server-0.17-17.6.The release 17.17.6 or later is required on Red Hat release 3 to work properly. The packages rsh clientand rsh server must have the same release. See below:

    [root@c3sn1 root]# rpm -qa | grep rshrsh-server-0.17-17.6

    rsh-0.17-17.6

    [root@c3sn1 root]#

    Enable the rexec, rsh, rlogin services.

    For security reasons, rexec, rsh, and rlogin are not enabled by default.

    Once you install the rsh-server, the rsh, rlogin, and rexec services are launched by xinetd.

    To enable the services, the redhat-config-services program in the X window system, or the ntsysv programmay be used. Once enabled, the xinetd service must be restarted using the service command:

    service xinetd restart

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    14/100

    3AT 21188 AWAA TNZZA 14/100

    Check that these services are enabled.

    [root@c3sn1 root]# chkconfig --list

    [root@c3sn1 root]# xinetd based services:

    krb5-telnet: off

    rsync: off

    eklogin: off

    gssftp: off

    klogin: off

    chargen-udp: off

    kshell: off

    auth: on

    chargen: off

    daytime-udp: off

    daytime: off

    echo-udp: off

    echo: off

    services: off

    time: off

    time-udp: off

    cups-lpd: off

    sgi_fam: on

    ktalk: off

    swat: off

    rexec: onrlogin: on

    rsh: on

    Edit the files/etc/xinetd.d/rsh, /etc/xinetd.d/rlogin, /etc/xinetd.d/rexecand verify that the fielddisable = no.

    Here is the rsh file as it looks by default :

    # default: on# description: The rshd server is the server for the rcmd(3) routine and, \

    # consequently, for the rsh(1) program. The server provides \

    # remote execution facilities with authentication based on \

    # privileged port numbers from trusted hosts.

    service shell

    {

    disable = no

    socket_type = stream

    wait = no

    user = root

    log_on_success += USERID

    log_on_failure += USERID

    server = /usr/sbin/in.rshd}

    If not, change "disable = yes" to "disable = no".

    The same must be done if required to the rlogin and rexec config fil es to enable these services. If one ofthese fil es has been modif ied, restart the xinetd service.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    15/100

    3AT 21188 AWAA TNZZA 15/100

    To allow root to use these tools, add the names of t he tools you wish to allow to t he / etc/ securet ty. Ifyou want to enable root login using rexec, rsh, and rlogin, add the following lines to/etc/securet ty:

    rexec

    rsh

    rlogin

    For a remote rsh/rexec connect ion, configure the file .rhosts of the c3s user account on each side, addingthe hostname and the Unix user used for the remote connection.

    Example:Consider a user call ed cmc_adm running a rexec command from the server ming.cmc21.tmntest .sel.alcatel.de (HP-UX 11i) towards the server c3sn1 where the C3S server isrunning (Linux Red Hat).

    The .rhosts file of the c3s user account on the C3S Linux server looks like:

    [c3s@c3sn1 c3s]$ pwd

    /home/c3s

    [c3s@c3sn1 c3s]$ cat .rhosts

    149.204.191.231 cmc_adm[c3s@c3sn1 c3s]$

    Check that the .rhosts f il e permissions are set to 755

    [c3s@c3sn1 c3s]$ ls -lrt .rhosts

    -rwxr-xr-x 1 c3s c3s 94 Feb 22 18:08 .rhosts

    [c3s@c3sn1 c3s]$

    On the C3S Linux server, the fi le / etc/ hosts must be updated wit h the IP address and the hostname ofthe server source of the rexec command:

    127.0.0.1 localhost.localdomain localhost

    172.26.116.150 c3sn1

    # Test rexec IMS149.204.191.231 ming.cmc21.tmntest.sel.alcatel.de

    The .rhosts file of the cmc_adm user account on the HP-UX server looks like:

    ming:/home/cmc_adm $ cat .rhosts

    172.26.116.150 c3s

    ming:/home/cmc_adm $

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    16/100

    3AT 21188 AWAA TNZZA 16/100

    Post procedure: Test locally the rsh command as follows:

    [root@c3sn1 xinetd.d]# rsh localhost -l c3s

    Password:

    Last login: Wed Mar 29 12:15:06 from localhost.localdomain[c3s@c3sn1 c3s]$

    Test locally the rexec command as follows:

    [root@c3sn1 root]# rexec -l c3s -p c3sc3s localhost ls /tmp

    PserverC3S_stderr.log

    hsperfdata_c3s

    kde-c3s

    kde-root

    ksocket-c3s

    ksocket-root

    mcop-c3s

    mcop-root

    orbit-c3sorbit-oracle

    orbit-root

    persistent_state

    ssh-GDB28214

    ssh-QGga9544

    toto

    [root@c3sn1 root]#

    Test remotely the rexec command between the HP-UX server and the Linux server as follows:

    c3s@c3sn1 c3s]$ rlogin 149.204.191.231 -l cmc_adm

    ming:/home/cmc_adm $ rexec 172.26.116.150 -l c3s -n ls /tmp

    Password (172.26.116.150:c3s):PserverC3S_stderr.log

    hsperfdata_c3s

    kde-c3s

    kde-root

    ksocket-c3s

    ksocket-root

    mcop-c3s

    mcop-root

    orbit-c3s

    orbit-oracle

    orbit-root

    persistent_state

    ssh-GDB28214

    ssh-QGga9544

    toto

    ming:/home/cmc_adm $

    Start the C3S server remotely wit h the rexec command from CMC environment instal led on the HP-UXserver:

    rexec 172.26.116.150 -l c3s -n /home/c3s/C3S_Supervision/run/C3Sctrl start -d

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    17/100

    3AT 21188 AWAA TNZZA 17/100

    2.1.4 SNMP configurati on

    Server(s) C3S Linux server

    Functi onal Domain/Domain C3S Administ rat ion

    Operation Reference SNMP Management

    Funct ional Descript ion SNMP agent that binds to a port and await s requests from SNMPmanagement software. Upon receiving a request, it processes therequest (s), collects the requested information and/ or perf orms therequested operation(s) and returns the informat ion to the sender

    Prerequisite Linux installed, C3S installed

    Other DocumentApproximate Duration A few minutes

    Frequency NA

    Steps to proceed:File to modif y: / home/ c3s/ C3S_Supervision/ conf/ alarmExt.propert ies

    After modif icat ion of this f il e you must restart C3S applicat ion.

    To activate alarm externalization, set parameter snmpAgent to true:

    snmpAgent=true

    To add or modify alarm manager modify managers entry:

    manager1.ip=172.26.10.20

    manager1.port=162

    manager1.community=public

    manager2.ip=

    manager2.port=

    manager2.community=

    manager3.ip=

    manager3.port=

    manager3.community=

    manager4.ip=

    manager4.port=

    manager4.community=

    manager5.ip=

    manager5.port=

    manager5.community=

    You can create up to 5 managers. For each one, specify its IP address, listening port, and community.

    The snmpd configurat ion f ile is / etc/ snmp/ snmpd.conf.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    18/100

    3AT 21188 AWAA TNZZA 18/100

    2.1. 5 C3S server directories

    All t he C3S direct ories are created under C3S_HOME=/ home/ c3s/ .

    Five directories exist :

    C3S_Supervision containing allthe applicat ion data

    C3S_Dataflow containing all the CDR files treated or waiting for treatment. This directory containsone directory per dataflow named with this dataflow name. Each dataflow name directory containsone subdirectory per DPM named wit h the DPM name. Finally, each DPM subdirect ory cont ains 2other subdirectories described below:

    o Data -> contains all the CDR f iles to be treated or already treated

    o Error -> contains all the erroneous CDR f iles

    Ex: / home/ c3s/ C3S_Dataflow/ IMS_FLOW/ IMS_PP_ASN1/ Data/

    C3S_AVR containing all the CDR files treated or waiting for treatment. This directory contains onedirectory per dataflow named with this dataflow name. Each dataflow name directory contains onesubdirectory per DPM named with the DPM name. Finally, each DPM subdirectory contains 4 othersubdirectories described below:

    o ToBeArchived -> contains all the CDR f il es archived by a user t hrough the C3S supervision

    o ToBeRecovered -> contains all the CDR f iles recovered by a user t hrough the C3S supervision

    o ToBeReprocessed -> contains all the CDR files able to be reprocessed

    o View -> contains all the CDR f iles the C3S user wants to view

    Ex: / home/ c3s/ C3S_AVR/ IMS_FLOW/ IMS_PP_ASN1/ View/

    C3S_Log containing all the applicat ion logs. This directory contains:

    o the following files:

    the current c3s_daily.log Current and previous c3s_all. logx dataflow.log

    o one directory per dataflow named with this dataflow name containing: one subdirectory DAILY_LOG one subdirectory DPM containing one log f il e per DPM

    C3S_OSSdirect ory contains one directory per dataf low named wit h this dataflow name. Eachdataflow name directory contains one subdirectory per Distributor LocalCopy DPM named with theDPM name. Finally, each DPM subdirectory contains all distributed CDR files.

    C3S_FTPdirectory is a specific directory that can be used for FTP collect.

    C3S_ImportExport is used for archiving/ reprocessing features on C3S database tables (f il e .dmp).

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    19/100

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    20/100

    3AT 21188 AWAA TNZZA 20/100

    2.1.6 C3S server logs configurati on

    2.1.6.1 Kind of log files used

    The C3S Server has to log several kinds of information in the system:

    Daily log:

    It is a daily log that contains informat ion of each t icket being t reated in the dataflow. Each DPM logsdail y information to indicate a t icket has been successfully processed in t he fl ow.

    DPM log:

    It is the detail of the treatment of each DPM. This log is useful to understand what is going on in anident if ied DPM.

    Other logs:

    c3s_all log f il e is a common log that contains all t he information that are not needed in the dail ylog or in the DPM log.

    dataflow log fi le is a log for t he dataf low object, explaining when a start or a stop is performed onthe fl ow.

    c3s_license log fi le is a log for l icense management.

    2.1.6.2 Log fi les locat ion

    There is an environment variable named LOG_ROOT defined in the $C3S_HOME/ run/ environment.sh

    shell script which indicates where all the log fil es are located.

    By default it is the / home/ c3s/ C3S_Log/ directory.

    Under this directory, the following structure is available:

    c3s_license.log

    c3s_daily.log

    c3s_daily.2004-06-22

    c3s_daily.2004-06-23

    _FLOW/dpm_.log

    The DPM directory contains the log of the DPM composing the dataflow.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    21/100

    3AT 21188 AWAA TNZZA 21/100

    2.1.6.3 Log confi guration

    The API used for t he log is the LOG4J standard. It s configurat ion is made through a unique configurat ionfi le named log4j .xml .

    This configuration f ile is located in the $JBOSS_HOME/ server/ default / conf/ directory.

    2.1.6.3.1 Appender definition

    An appender is a log reposit ory or an output dest ination for t he log. It can be a fil e, a database or anythingelse. For t he C3S application, all the LOG4J appenders are f il es.

    2.1.6.3.2 Category definition

    A Category (or Logger), is a hierarchical view that matches one or several appenders. Most of the time, acategory is the name of a package or the full name of a class.

    Thus, a category can be defined by the name of a package as fr. alcatel or by the name of a class as fr.alcatel.cit .products.c3s.dataf low.dpm.Processor .

    2.1.6.3.3 Level definit ion

    The goal of a LOG4J level is to log a type of informat ion from the less important to the most important .

    The order of the level is the following one: OFF, FATAL, ERROR, WARN, INFO, DEBUG and ALL.

    Level name Descript ion

    OFF The OFF level has the highest possible rank and is intended to turn off logging.

    FATALThe FATAL level defines very severe error events that will presumably lead the applicationto abort.

    ERRORThe ERROR level defines error events that might still allow the application to continuerunning.

    WARN The WARN level defines potent iall y harmful situations.

    INFOThe INFO level defines informational messages that highlight the progress of the applicationat coarse-grained level.

    DEBUGThe DEBUG level defines fine-grained informational events that are most useful to debug anapplication.

    ALL The ALL level has the lowest possible rank and is int ended to turn on all logging.

    File Appender A

    Database A ender B

    Category 1

    Category 2

    The Category 1 is logging information to Appender A and B,

    whereas Category 2 is only logging information to Appender B.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    22/100

    3AT 21188 AWAA TNZZA 22/100

    2.1.6.3.4 Log4j swit ching

    Each DPM uses two categories for logging informat ion:

    a category that inserts information in the DPM log

    a category that inserts information in the Daily log and also in the DPM logWhen a DPM is logging information in the Daily log, it is also logged in the log of the DPM.

    Daily Log Appender

    DPM A Log Appender

    DPM Daily Category

    DPM Category

    The Daily Log Appender is ref erenced by two categories of DPMs.

    DPM B Log Appender

    DPM Daily Category

    DPM Category

    DPM A

    DPM B

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    23/100

    3AT 21188 AWAA TNZZA 23/100

    This configuration is defined in an XML file. Here is an example for a DPM A:

    DPM appender definition:

    The appender named FILE_GPRS_A is a roll ing f ile appender, defined by a maximum f ile size of 10 000 Kband a maximum backup index of 1.

    The roll ing fi le i s named dpm_GPRS_A.log . In the root log directory, we wil l have:

    dpm_GPRS_A.log for the current DPM log,

    dpm_GPRS_A.log.1 for the previous DPM log (only 1 previous file, as the maximum backup index isdefined to 1).

    The default size of this file is 10 000 Kb. To modify this value, open the$C3S_HOME/ etc/ resources/ SystemConfigSt rings.propert ies and change the value ofMoUtil.maxLogSizekey.

    Daily appender definition:

    The appender named FILE_DAILY is a daily rol l ing fil e appender, defined by a date patt ern.

    The daily roll ing f ile is named c3s_dail y.log . In the root log directory, we wil l have:

    c3s_daily.log for the current daily log,

    c3s_daily. log 2004-06-25, c3s_dail y. log 2004-06-26, , for t he previous daily logs.

    The name of each previous f ile is defined by the date patt ern.

    Note: The daily log file is deleted when the maximum authorized number of files is reached (see 2.1.6.4Maximum number of daily logs).

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    24/100

    3AT 21188 AWAA TNZZA 24/100

    DPM A categories definit ion:

    The f irst category named fr.alcatel.cit .products.c3s.dataFlow.dpm.Scheduler.GPRS_A is linked t o t he DPMappender FILE_GPRS_A . Its level is DEBUG . This category is useful for logging information in the log ofthe DPM.

    The second category named fr. alcatel. cit .products.c3s.dataFlow.dpm.Scheduler.DAILY_GPRS_A is li nkedto the common daily log appender FILE_DAILY and also to the DPM appender FILE_GPRS_A . It s level i s INFO , meaning a dailylog.debug() log4j call wil l not log information to that f ile. This category is useful forlogging information in the daily log (also automatically logged in the DPM log).

    Note on the category name: t he LOG4J standard uses the name of a package or a class to name a category.For t he C3S needs, this standard has been slight ly modif ied for t he DPM, to be able t o log information foreach INSTANCE of a class.

    This is why the category uses the name of the Scheduler class suffixed by the name of the DPM. The

    fr. alcatel. cit .products.c3s.dataFlow.dpm.Scheduler.DAILY_GPRS_A name is used for each java class of aDPM, especially all the classes of the GPRS_A j ava instance.

    The informat ion used by a java instance to identif y the correct category is the name of t he DPM, here GPRS_A .

    2.1.6.4 Maximum number of daily logs

    Because a dail y log is created every day, a process has been included to delete the older daily log fil e whenthe number of files reaches the allowed maximum number. By default, this number is set to 30 days.

    To modify t his value, open the $C3S_HOME/ etc/ resources/ SystemConfigSt rings.propert ies and change t he

    value of C3S.maxNumberDailyLog key.

    The control is made every day.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    25/100

    3AT 21188 AWAA TNZZA 25/100

    2.1. 7 SFTP configuration

    2.1.7.1 Automat ic start -up of SFTP server at boot t ime

    Server(s) C3S Linux server

    Functi onal Domain/Domain C3S configurat ion

    Operation Reference Secure FTP server start-up at boot time

    Funct ional Descript ion Start at boot time the secure FTP server required for C3S application

    Prerequisite Log on the Linux server

    Other DocumentApproximate Duration A few minutes

    Frequency If required

    Steps to proceed:

    Activate at boot time the automatic restart of FTP server as detailed below. This procedure must beperformed under the Unix root user account:

    [root@c3sn1] cd /etc/init.d

    [root@c3sn1 init.d]# chkconfig --list | grep vsftpd

    [root@ c3sn1 init.d]# chkconfig --add vsftpd

    [root@ c3sn1 init.d]# chkconfig vsftpd on

    [root@ c3sn1 init.d]# chkconfig --list | grep vsftpd

    vsftpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

    [root@c3sn1 init.d]# ./vsftpd start

    [root@c3sn1 init.d]# ps -eaf | grep vsftp

    root 1682 1 0 2005 ? 00:00:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf

    Post procedure: Reboot the Linux server:

    [root@c3sn1 init.d]# shutdown r now

    Check that the Secure FTP server is started automatically at boot time:

    [root@c3sn1 init.d]# ps -eaf | grep vsftp

    root 1682 1 0 2005 ? 00:00:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    26/100

    3AT 21188 AWAA TNZZA 26/100

    2.1.8 HTTP & HTTPS configurati on

    Server(s) C3S Linux server

    Functi onal Domain/Domain C3S configurat ion / Client-server communication

    Operation Reference HTTP & HTTPS conf igurat ion

    Funct ional Descript ion HTTP & HTTPS configuration required for C3S application (client-servercommunication)

    Prerequisite All the platform (all the clients & servers) must be running either in HTTPor HTTPS but never a mix of these 2 protocols

    Other DocumentApproximate Duration Less than a minute

    Frequency If required

    The C3S client-server communicat ion can be done through HTTP or HTTPS protocol.

    HTTP & HTTPS port s are def ined in $C3S_HOME/ etc/ resources/ ServerConfigSt rings.propert ies:HTTP port : GSEM.serverPort=8080HTTPS port : GSEM.ht tpsServerPort=8443

    In a TOMCAT specif ic f il e $JBOSS_HOME/ server/ default (and all)/ deploy/ JBoss-web.deployer/ server. xml:

    HTTP port :

    HTTPS port :

    For HTTPS, the keystoreFile f ield contains the path and t he name of the cert if icate and keystorePassthe password of this cert if icate you can find under $JBOSS_HOME/ .

    These 2 xml description must be uncommented together to switch from HTTP to HTTPS without changingthis fi le.

    HTTP/ HTTPS configurat ion:

    The used protocol depends of the presence of 2 fil es in t he $C3S_HOME/ conf/ directory:

    HTTP: presence of allow.http & allow.noSecurity f iles (t hese f il es can be empty).

    HTTPS: these files are not present.

    To take into account this modification, the server has to be restarted.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    27/100

    3AT 21188 AWAA TNZZA 27/100

    2.1. 9 C3S server parameters

    The following list describes the C3S server parameters defined in the$C3S_HOME/etc/resources/SystemConfigStrings.properties file.

    Key Default Value Descript ion

    user.winscp c3swinscp User name used by WinSCP to launch asession when t he Purge f il es recoveringfunction is launched

    c3s.user

    c3s.password

    c3s

    c3sc3s

    User name used by WinSCP to launch asession when the choose data function islaunched from the container menu

    HistoryPath.loggedEvents

    HistoryExtension.loggedEvents

    etc/ log/

    .txt

    Resources for events history logging

    Welcome.Image etc/ SystemIcons/information.gif

    Resource used for the welcome banner

    WinSCP.LaunchWinSCP C:\Program Files\winscp350\winscp350.exe

    WinSCP pat h

    FlowDetails.Path / run/ runFlowDetails.bat Flow detail s path

    JBOSS.defaultDeployPath / usr/ local/ JBoss/ server/default / deploy/

    JBoss path in mono-server configurat ion

    JBOSS.allDeployPath / usr/ local/ JBoss/ server/ all/deploy/

    JBoss path in multi-server configuration

    GSEM.nbMaxConnectedClient 30 Number of cl ient that must be connectedin the same time, on the same server

    MoLoggedEvent .AutomaticPurgeThreshold

    2500 Define a threshold which will cause anautomatic purge of history events whenover passed

    MoLoggedEvent .

    AutomaticPurgeDelay

    10080 Define a t ime threshold (in minutes) which

    wil l cause an automat ic purge of historyevent s when over passed

    MoLoggedEvent .AutomaticFileName

    AutoPurge Name of the f ile automat ically generated

    MoLoggedEvent .nbrPurgeFilesAuthorized

    5 Number of f iles in the purge directory

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    28/100

    3AT 21188 AWAA TNZZA 28/100

    2.1. 10 ASN1 configurat ion

    For each C3S map, a li st of ASN1 grammars list is defined in t he$C3S_HOME\ioStudio\pr ototypingArea\dpmCodec\gener icCodecAsn1Ber\config\Asn1GrammarsList.properties file.

    It is automatically generated during the map generation within the C3S Studio.

    2.1.11 Containers configuration

    The C3S Container configuration is available through the C3S client GUI. For details, refer to the Alcatel8965 Convergent Charging Coll ect ion Solut ion (C3S) Operat or Guide [R1].

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    29/100

    3AT 21188 AWAA TNZZA 29/100

    2.1.12 IPSec configuration

    Server(s) C3S Linux server

    Functi onal Domain/Domain Linux server configuration

    Operation Reference IPSec configuration

    Funct ional Descript ion IPSec can be used to secure data collect ion

    Prerequisite Package ipsec-tools, release 0.3.3-6 or later must be installed

    Other DocumentApproximate Duration Few minutes

    Frequency If required

    2.1. 12.1 Racoon configuration

    Add in / etc/ init tab the following line (in bold):

    # System initialization.

    si::sysinit:/etc/rc.d/rc.sysinit

    7:2345:respawn:/usr/sbin/snmpd f

    8:2345:respawn:/usr/sbin/racoon -F

    l0:0:wait:/etc/rc.d/rc 0

    l1:1:wait:/etc/rc.d/rc 1

    l2:2:wait:/etc/rc.d/rc 2

    Modify the f ile / etc/ racoon/ racoon.conf as follows:

    path include "/etc/racoon";

    path pre_shared_key "/etc/racoon/psk.txt";path certificate "/etc/racoon/certs";

    sainfo anonymous

    {

    pfs_group 2;

    lifetime time 1 hour ;

    encryption_algorithm 3des, blowfish 448, rijndael ;

    authentication_algorithm hmac_sha1, hmac_md5 ;

    compression_algorithm deflate ;

    }

    remote anonymous

    {

    exchange_mode aggressive, main;

    doi ipsec_doi;

    my_identifier address;

    proposal {

    encryption_algorithm 3des;

    hash_algorithm sha1;

    authentication_method pre_shared_key;

    dh_group 2 ;

    }

    }

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    30/100

    3AT 21188 AWAA TNZZA 30/100

    2.1. 12.2 Pre-shared key configuration

    In the fi le / etc/ racoon/ psk.txt , add all pre-shared key used by ipsec, add one li ne for each dif ferent NE IPaddress that may connect to C3S using IPSec:

    2.1. 12.3 Racoon restart

    For the first configuration, if racoon process is running, kill it. Then execute:

    telinit q

    If you just add a new pre-shared key execute:

    racoonctl reload-config

    2.1. 13 Diameter TLS configuration

    Server(s) C3S Linux serverFuncti onal Domain/Domain Linux server configuration

    Operation Reference Diameter TLS configurat ion

    Funct ional Descript ion TLS is used to secure diameter data collection

    PrerequisiteOther DocumentApproximate Duration Few minutes

    Frequency If required

    You must install server and client certificates in the keystore

    / home/ c3s/ C3S_Supervision/ etc/ resources/ diameter/ resource/ t ls/ ccserver. auth f il e.

    This fil e can be created wit h the keytool command, using the default format (j ks).

    Set the used password to create the keystore in the/ home/ c3s/ C3S_Supervision/ etc/ resources/ diameter/ resource/ t ls/ t ls.propert ies fil e.

    server.authFilePassword=

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    31/100

    3AT 21188 AWAA TNZZA 31/100

    2.2 C3S client

    2.2.1 C3S server hostname configuration

    The hostname where the client is connected is defined in the

    %C3S_HOME%/ etc/ resources/ ServerConfigSt rings.propert ies fi le:GSEM.serverAddress=HOSTNAME

    2.2.2 HTTP & HTTPS port configurati on

    The C3S client-server communicat ion can be done through HTTP or HTTPS protocol.

    The Client HTTP or HTTPS port used for the communication with the C3S server is defined in the%C3S_HOME%/ etc/ resources/ ServerConfigSt rings.propert ies fi le:

    For HTTP: GSEM.cl ientSideServerPort=8088For HTTPS: GSEM.ht tpsClientSideServerPort=8444

    And in t he TOMCAT specif ic %C3S_HOME%/ tools/ tomcat/ cong/ GSEMClient.xml f ile:

    HTTP port :

    HTTPS port : For HTTPS, the keystore f ield contains the path and the name of t he cert if icate and keyspass thepassword of this cert if icate you can find under %C3S_HOME%/ run/ .

    C3S client checks the protocol used from server and switches from HTTP to HTTPS automatically.

    2.2.3 C3S client logs management

    Logging inf ormation in the C3S client is not mandatory but it can be done easil y with LOG4J using the samemechanism detailed previously for the server.

    There is a java environment variable named log4j .propert ies defined when calli ng the

    C3S_Client_Console.bat script indicating where the LOG4J client configurat ion is located.

    By default , it s value is %C3S_HOME%\conf\c3s_client_log4j .xml f il e.

    This configuration file defines only one appender for logging all the informat ion.

    This file i s named %C3S_HOME%\log\c3s_client .log where C3S_HOME = C:\C3S_Client .

    The category associated to the client is in DEBUG level.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    32/100

    3AT 21188 AWAA TNZZA 32/100

    2.2. 4 C3S client reconnect ion

    When a C3S client is connected to a server and this one is stopped (or crashed), a popup is displayed to askthe user if he wants to be reconnected to another server. In this case a new C3S supervision login window isdisplayed and the user may connect to another server: the detailed server list gives the state and theversion of each server. Otherwise, the client is stopped.

    During installation of the C3S client, the installer has to define the servers list (hostname and correspondingIP address) where this client can be connected. This list can be dynamically modified at any time after theinstallation through the GUI for adding, updating or removing C3S server.

    Another way to add a server is to modif y the $C3S_HOME/ etc/ resources/ serverList.propert ies f il e on theWindows client PC by adding or removing some server hostname. Only 10 servers max can be defined.

    Moreover, the hosts file must be modified to assign the corresponding IP address to the hostname. This fileis under c: / winnt / System32/ drivers/ etc/ .

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    33/100

    3AT 21188 AWAA TNZZA 33/100

    3 C3S DATABASE MANAGEMENT

    3.1 Enterpr ise Manager

    The C3S database management can be performed using the Enterpr ise Manager Database control ler.

    This tool can be used only i f the Enterpr ise Manager Repository has been created in the C3S database.

    3.1.1 Enterpr ise Manager Database Cont rol status

    The start / stop of t he Enterprise Manager Database Control is performed running below commands:

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    34/100

    3AT 21188 AWAA TNZZA 34/100

    3.1.2 Enterpr ise Manager Database Cont roller GUI

    The Enterprise Manager Database Controller GUI is reachable using a Web interface with thehttp://:5500/em URL where is the IP address or the hostname of the server where theC3S database is located.

    Logon and home screens of the Enterpr ise Manager Database Control are displayed below:

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    35/100

    3AT 21188 AWAA TNZZA 35/100

    3.2 Monit ori ng Oracle logs

    This section gives a general description of all the log files related to C3S application based on the followingcomponents:

    Oracle RDBMS,

    Oracle Warehouse Builder,

    Oracle Workflow.

    3.2.1 Oracle RDBMS

    3.2.1.1 RDBMS kernel parameters

    The array below gives the list of Oracle parameters related to traces.

    Parameter name Parametervalue

    Recommendedvalue

    Description / Log file

    sql_trace TRUE/ FALSE FALSE After set t ing sql_trace to t rue, a trace file will bewritten. When one of the Oracle backgroundprocesses (such as dbwr, lgwr, pmon, smon, )encounters an exception, it wil l wr it e a trace file.These trace files are also recorded in the alert.log.Trace files are also created for diagnost ic dumpevents.user_dump_dest=/ dbora/ admin/ C3S/ udump

    trace_enabled TRUE/ FALSE FALSE Used for debugging. Controls t racing of the execut ionhistory or code path of Oracle

    t racefil e_identif ier st ring None Specif ies a custom identif ier that becomes part of theOracle trace file name

    How to change trace value of the previous parameters:

    SQL> ALTER SYSTEM SET TRACE_ENABLED = TRUE;

    SQL> ALTER SESSION SET SQL_TRACE = TRUE;

    SQL> ALTER SESSION SET TRACEFILE_IDENTIFIER = "traceid

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    36/100

    3AT 21188 AWAA TNZZA 36/100

    3.2.1.2 Log files

    The array below gives main Oracle log files location. Note that some of these log files do not exist in the C3Senvironment.

    Component DefaultFormat Description / Location

    All user session tracing will be located in user_dump_dest orbackground_dump_dest for background traces. These t races are enabled wi ththe parameter sql_trace and t race_enabled . Oracles naming convent ionis platform dependant, but should have the spid in it returned from thesession SQL above. An example would be {ORACLE_SID}_ora_{spid}.t rc(orcl_ora_1389.t rc). You can also use the TRACEFILE_IDENTIFIER before youstart the t racing, to help ident if y your t race f iles.background_dump_dest = / dbora/ admin/ C3S/ bdumpuser_dump_dest = / dbora/ admin/ C3S/ udumpcore_dump_dest = / dbora/ admin/ C3S/ cdump

    / dbora/ admin/ C3S/ udump/ c3s_ora_.t rc

    RDBMS Text

    / dbora/ admin/ C3S/ bdump/ alert _C3S.log/ dbora/ admin/ C3S/ bdump/ c3s_xxx.t rc

    Listener TextORACLE_HOME/ network/ log/ listener.log is the listener's logORACLE_HOME/ network/ log sqlnet . log is a client sqlnet log

    EnterpriseManager

    Text ORACLE_HOME/ sysman/ log

    HTTPServer/ Apache

    Text ORACLE_HOME/ companion/ Apache/ Apache/ logs/ error_log.t ime

    Log Loader ODL ORACLE_HOME/ diagnostics/ logs

    OPMN TextORACLE_HOME/ opmn/ logsORACLE_HOME/ opmn/ logs/ component_type~

    Universal Installer Text ORACLE_HOME/ cfgtool logs/

    In the previous array, the value of ORACLE_HOME is / oracle/ product / 10.2.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    37/100

    3AT 21188 AWAA TNZZA 37/100

    3.2.2 Oracle Warehouse Builder

    3.2.2.1 Runtime log file

    File name File system location Descri pt ion

    C3sowbrun.log.xx / oracle/ owb_10.2/ owb/ log Traces of the OWB Runtime service

    Oracle Warehouse Builder Runtime Platform Service (RTP) does generate a log fi le t hat should reside int othe Warehouse_Builder_home/ owb/ log.

    The log file contains information about what the service is doing and can become very huge.

    At each startup of the OWB RTP service a new log file is created. The naming isRUNTIME_OWNER_NAME.LOG.xx (where xx is a sequence number).

    [oracle@c3svalid log]$ pwd

    /oracle/owb_10.2/owb/log

    [oracle@c3svalid log]$oracle@c3svalid log]$ ls -lrt

    total 38308

    -rw-r--r-- 1 oracle dba 2579 Jan 10 10:56 C3SOWBDES.log.10

    -rw-r--r-- 1 oracle dba 1939 Jan 17 11:48 C3SOWBDES.log.11

    -rw-r--r-- 1 oracle dba 2579 Dec 20 18:36 C3SOWBDES.log.9

    -rw-r--r-- 1 oracle dba 237331 Apr 24 16:44 C3SOWBRUN.log.63

    -rw-r--r-- 1 oracle dba 49478 Apr 24 17:32 C3SOWBRUN.log.64

    -rw-r--r-- 1 oracle dba 47688 Apr 24 17:45 C3SOWBRUN.log.65

    There is no possibility to reduce the logging of the runtime service. And there is also no specific procedurefor delet ing/ purging the log fil es. You can just delete the log fi les you want except t he current one.

    If you have a short period of time where the service can be stopped and restarted, then you could write andschedule a procedure that will stop the service, delete or archive the log file and start the service.

    Note that this log f il e cannot be deleted whi le t he OWB RTP service is running.

    The only way is to stop/ restart the service. This wil l generate a new log fi le. The old one can bedeleted/ archived. The default number of f il e is 5 wit h 40000 messages by fil e. This can be change using theshell set_c3s_owb_rt _parameters.sh int o / home/ c3s/ C3S_Supervision/ db/ config_db/

    There are scripts (start_service.sql and stop_service.sql under the / oracle/ owb_10.2/ owb/ rt p/ sql directory)that can be used for the OWB start / stop. Those procedures wil l let the service startup in automatic status,meaning that if for any reason the database instance is stopped, t he OWB RTP service wil l restart

    automatically after the database restart.

    It is recommended to stop the service when there is no activity running.

    If this is the case, the runtime platform service should try to restart the activities that were in busy statuswhen the service was stopped.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    38/100

    3AT 21188 AWAA TNZZA 38/100

    The restart means, complete restart of the activity from the beginning.

    oracle@c3sn1 sql]$ sqlplus c3sowbrun/c3sowbrun

    SQL*Plus: Release 10.2.0.1.0 - Production on Thu Apr 24 18:08:53 2008

    Copyright (c) 1982, 2005, Oracle. All rights reserved.

    Connected to:

    Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production

    With the Partitioning, OLAP and Data Mining options

    SQL> start service_doctor.sql

    All PL/SQL packages and functions are valid

    Platform properties have been loaded correctly

    Platform location has been seeded correctly

    NLS messages have been loaded correctly

    The platform service is available

    Service script is accessible to the database server

    Connection information stored within the repository is correct

    PL/SQL procedure successfully completed.

    SQL> start stop_service.sql

    Not Available

    PL/SQL procedure successfully completed.

    SQL> start start_service.sql

    Available

    PL/SQL procedure successfully completed.

    SQL> start service_doctor.sql

    All PL/SQL packages and functions are valid

    Platform properties have been loaded correctly

    Platform location has been seeded correctly

    NLS messages have been loaded correctly

    The platform service is available

    Service script is accessible to the database server

    Connection information stored within the repository is correct

    PL/SQL procedure successfully completed

    3.2.2.2 Audits

    Audits information is stored into tables in the runtime owner schema in the Oracle database.

    Those can be purged using the Warehouse Buil der Runtime Audit Browser or using the purge purge_audit_template.sql script .

    3.2.3 Oracle Workf low

    3.2.3.1 Workflow logs

    There is no available log file generated in the OWF file systems.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    39/100

    3AT 21188 AWAA TNZZA 39/100

    4 C3S SUPERVISION

    4.1 Event management

    4.1.1 Event descripti on fi les

    The configuration files for the event generation are alarm.txt & log.txt. They are located in the$C3S_HOME/ etc directory where C3S_HOME=/ home/ c3s/ C3S_Supervision. These fil es contain t he foll owingevent information:

    Event Id Event type Severi ty (only for alarm) Description Cause

    Action Addit ional informat ion

    To change the value of the descript ion, cause or action, the/ home/ c3s/ C3S_Supervision/ etc/ resources/ LocalSt rings.propert ies fi le must be changed. In fact, these 3fields are only keys of the file.

    4.1.2 Alarm lif e cycle

    The alarms are displayed in the [Alarm Events] tab list. Initials designate the alarm states:P: PendingA: Acknowledged

    T: TerminatedC: Cleared

    The state is or changes from .. . When .. .

    P An alarm is received and has been formatted.

    P to A An operator acknowledges the alarm manually. This action means thatthe operator has taken the alarm into account.

    P to T The end of alarm trap is received.

    A to C The end of alarm is received.

    A to C An operator performs a manual clear action.

    T to C The operator performs a manual acknowledgement action.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    40/100

    3AT 21188 AWAA TNZZA 40/100

    4.2 JBoss / C3S server log f i les

    JBoss and C3S servers can be supervised through several log fi les located on f ile systems on Linux server .

    4.2.1 Log configurati on fi les

    The configurat ion f il e for log generation for j boss and C3S servers is log4j.xml . This file is located in the$JBOSS_HOME/ server/ default / conf directory, where JBOSS_HOME=/ usr/ local/ j boss.

    For a jboss cluster configurat ion, this f il e is located in $JBOSS_HOME/ server/ all / conf.

    4.2.2 JBoss log f il es

    File name File system location Descript ion

    boot.log $JBOSS_HOME/ server/ default / log Start ing up t races of JBoss server

    server.log $JBOSS_HOME/ server/ default / log JBoss and C3S servers t races at run t ime.

    4.2. 3 C3S server log f iles

    See 2.1.6.3 Log configurat ion for a detailed descript ion of the C3S log fi les configuration.

    File name File system location Descript ion

    c3s_license.log / home/ c3s/ C3S_log t races related to li cense management

    .log / home/ c3s/ C3S_log/ / t races related to a specif ic DPM of a dataf low

    c3s_dail y.log / home/ c3s/ C3S_log daily t races related to the C3S application

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    41/100

    3AT 21188 AWAA TNZZA 41/100

    5 CLUSTERING MANAGEMENT

    5.1 MC Service Guard cluster / Duplex confi gurati on

    A MC Service Guard cluster (High Availability cluster) is made up of N active nodes and one passive node. Aduplex configuration is restricted to 2 nodes (one active and one passive).

    The passive node is a stand-by node, ready to t ake over f rom the node that f ails.

    The High Availabil it y soft ware used for implementing the duplex configuration is MC/ServiceGuard forLinux.

    5.1.1 Repair aft er an automat ic swit ch-over

    When a problem occurs on a node within the cluster, the C3S functions running on the active node are

    act ivated automat ically on the spare node of t he cluster: swit ch over mechanism.

    The previous cluster node has to be repaired before performing a switch back of the C3S functions.

    How to know that a fail ure happened:

    Mono server case

    In this case, no event is generated when the failure happened. The only information is thegeneration of an historic event on the C3S client when the C3S server is started back.

    Mult i-server case

    In this case, when a server is stopped, an historic event is generated to inform every clients, nomatter on which server they are connected, the specific server is stopped.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    42/100

    3AT 21188 AWAA TNZZA 42/100

    How t o analyze the issue:

    Cluster behavior traces are stored in the/var/ log/messages log file.

    MC/ SG package behavior t races are stored in the/usr/ local/ cmcluster /packages/pkg_c3s/pkg_c3s.sh. log file.

    Thanks to these log fi les, t he operator detects the node where the problem occurs and can see the log andthe alert describing the origin of the failure (disk, memory, IO card, ).

    List of t he possible causes:

    The C3S package is switched-over from t he act ive node t o the stand-by node when there is a hardwareproblem on the active node of t his kind:

    CPU failure, Memory failure, SCSI interface failure, LAN failure.

    There is no package switch over due to software problem. In the current configuration, the C3S serverswitch over is restricted to hardware failures.

    5.1.2 Swit ch-back to the nominal node

    Once the failed server has been repaired, the switch back is not automatic. The system administrator has toperform t his procedure manually.

    See 5.1.3 Maintenance operations.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    43/100

    3AT 21188 AWAA TNZZA 43/100

    5.1. 3 Maintenance operat ions

    Maintenance operat ions related to duplex can be performed using textual commands on each server of thecluster.

    Main useful procedures are detailed in this document:

    In the 6.3. 2 MC Service Guard cluster configuration, f or the C3S package start / stop procedure and thepackage manual switch-over.

    In the 7.1.3 MC/SG cluster check, for checking the location of the C3S package (to know on whichserver the package is running).

    In the 7.1.4 MC/SG package switch back to nominal node, for switching back the C3S package fromthe secondary node to the primary node.

    When C3S is delivered in a duplex configuration, the clustering architecture allows upgrading a node

    (hardware or software upgrade) while the other node continues to be operational.

    This hardware configuration offers the advantage to reduce the down time and not to loose charging eventscoming from Network Elements.

    Hardware upgrade:

    The operational server has to be upgraded (for example, adding 1 Gb of memory).

    Following procedures have to be performed on the duplex:

    Stop the C3S package on the operat ional server ,

    Switch over manually the C3S package from the active node to the stand-by node,

    Perform the hardware upgrade on the old operational server.

    See previous sect ions.

    Sof t ware upgrade:

    For software upgrades, a specific document will be delivered describing all the upgrade steps to perform.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    44/100

    3AT 21188 AWAA TNZZA 44/100

    5.2 Red Hat Cluster Suite / Duplex configurat ion

    Red Hat Cluster Suite (High Availability cluster) is made up of N active nodes and one passive node. A duplexconfiguration is restricted to 2 nodes (one active and one passive).

    The passive node is a stand-by node, ready to t ake over f rom the node that f ails.

    The High Availabil it y soft ware used for implementing the duplex configuration is Red Hat Cluster Suite(RHCS).

    5.2.1 Repair aft er an automat ic swit ch-over

    When a problem occurs on a node within the cluster, the C3S functions running on the active node areact ivated automat ically on the spare node of t he cluster: swit ch over mechanism.

    The previous cluster node has to be repaired before performing a switch back of the C3S functions.

    How to know that a fail ure happened:

    Mono server case

    In this case, no event is generated when the failure happened. The only information is thegeneration of an historic event on the C3S client when the C3S server is started back.

    Mult i-server case

    In this case, when a server is stopped, an historic event is generated to inform every clients, nomatter on which server they are connected, the specific server is stopped.

    How t o analyze the issue:

    Cluster behavior traces are stored in the/var/ log/messages log f ile. In some cases Red-Hat Clustert races could have been stored into a dif ferent f ile, t ry grep local4 / etc/ syslog.conf to know witch.

    Thanks to this log file, the operator detects the node where the problem occurs and can see the log and the

    alert describing the origin of the failure (disk, memory, IO card, ).

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    45/100

    3AT 21188 AWAA TNZZA 45/100

    List of t he possible causes:

    The C3S package is switched-over from t he act ive node t o the stand-by node when there is a hardwareproblem on the active node of t his kind:

    CPU failure,

    Memory failure, SAS interface failure, LAN failure.

    There is no package switch over due to software problem. In the current configuration, the C3S serverswitch over is restricted to hardware failures.

    5.2.2 Swit ch-back to the nominal node

    Once the failed server has been repaired, the switch back is not automatic (like with Service Guard). Thesystem administrator has to perform this procedure manually.

    See 5.2.3 Maintenance operati ons for Red-Hat cluster.

    5.2. 3 Maintenance operat ions for Red-Hat cluster

    Maintenance operations related to RHCduplex can be performed using textual commands on each server ofthe cluster.

    Main useful procedures are detailed in this document:

    In the 6.3.3 Red Hat cluster configuration, f or the C3S service start / stop procedure and the servicemanual swit ch over.

    In the 7.1.5 Red Hat cluster check, for checking the location of the C3S service (to know on whichserver the C3S application is running).

    In the 7.1.6 Red Hat cluster service switch back to nominal node, for switching back the C3S servicefrom the secondary node to the primary node.

    Clustering architecture on Red Hat cluster allows upgrading a node (hardware or software upgrade) whilethe other node continues to be operational (l ike MC/ SG).

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    46/100

    3AT 21188 AWAA TNZZA 46/100

    5.3 JBoss cluster

    JBoss cluster i s made up of N act ive nodes. These nodes can be int egrated in a duplex.

    JBoss takes care of active nodes only. Each node implements a specific data flow collection (or several data

    flows).

    The goal of this architecture is to expand the C3S collection functions in order to increase performances ofthe global mediation function. The supervision client PCs can be connected by default to the server(s) oftheir choice depending on the customer s organization. Through this server t hey are able to administ ratethe other nodes exactly as they were connected directl y to them.

    5.3.1 Configuring C3S servers

    Servers have to be deployed one by one.

    Cluster with one server: apply deployment procedures

    o Once server and clients (1) installed, start data fl ow.o Check map, status (green color), counters, history.o Stop dist ribut ion DPMs.o Create collection instances (prepare the parameters before starting instances creation).o Activate instance collection.o Check files collection, counters progression.o When containers not empty at DPM dist ribution level, act ivate dist ribution.

    Adding a node to a cluster: apply deployment procedures.

    Check that users connected on both servers share the same alarms panel.

    Check that maps of the second server are visible and manageable from the client 1 and vice versa.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    47/100

    3AT 21188 AWAA TNZZA 47/100

    5.3.2 JBoss server configurat ion f il es

    In such a hardware configuration, the C3S client is delivered with a JBoss clustering management tool:JMX-console.

    This tool is used thanks to an Internet browser by typing the ht tp: / / :8080 URL, where

    is the hostname or IP address of the Linux server.

    File Name File Location Descri pt ionFactoryconfiguration

    Siteconfiguration

    JNDIConfigSt rings.properties

    $C3S_HOME/etc/ resources

    HA-JNDI port (j npPort=):

    1099 for mono server 1100 for multi server

    no no

    JBoss-service.xml $JBOSS_HOME/server/ default /conf

    Core Services:

    Listen port for JBoss, JNDI declarat ion for JBoss cluster

    architecture

    yes no

    Log4j .xml Logging Services:

    C3S appenders definition (applicationlog fi les)

    C3S daily (administrative tasks), C3S classes log files, DMP log file, FTP collection, GTP collect ion, C3S server t race level

    yes no

    loggin-config.xml Securit y Services

    Authentication modules :

    Java Authent ication

    Authorizat ion modules

    yes possible

    Web.xml

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    48/100

    3AT 21188 AWAA TNZZA 48/100

    5.3.3 Conf iguring C3S cli ent

    5.3.3.1 C3S server hostname list

    During the C3S client installation, it is possible to define a list of servers where the C3S application couldrun. That means if a server node fails, t he operator has the choice to reconnect to another one defined in

    this list.

    The procedure to apply normall y is the foll owing:

    First client installat ion: only one server node (clustering system or not) is configured.

    Then if a new server is added (cluster JBoss), the system administrator has to decide first if the usercan be connected in case of failure on the second node or not. In order t o do so, include the newserver name on the C3S client in the $C3S_HOME/ etc/ resources/ serverList .propert iesfile.

    He has to introduce the packaging client CD in order to declare the new DPM MIBs. When this newpackage is instal led, the user is able t o manage both servers and to administ rate the C3S applicationdistributed on these servers, whatever is the server on which the system administrator is connected.

    The rights of this user on t he new server can be managed through OAD (Object Access Domain) andFAD (Funct ional Access Domain).

    5.3.4 Repair aft er a JBoss server failure

    When a problem occurs on a node within the JBoss cluster, the C3S functions running on the active node areno longer available.

    The previous JBoss server node has to be repaired before perf orming a restart of the C3S appli cat ion.

    How to know that a fail ure happened:

    When a failure happened on a server, an historic event is generated to inform every cl ient , no mat ter onwhich server they are connected, the specific server is stopped.

    How t o analyze the issue:

    First the server hardware and software must be analyzed.

    If all is OK, the C3S installat ion log files ($C3S_HOME/ C3S_Install_Log/ *.log) must be analyzed.

    List of t he possible causes:

    The JBoss server is unable when there is a hardware problem on the active node of this kind: CPU failure, Memory failure, SCSI interface failure, LAN failure, C3S applicat ion installat ion failure.

    5.3.5 Restart the node

    Once the failed server has been repaired, the restart is not automatic. The system administrator has toperform this procedure manually. It consists only to start the server as indicated in 6.3.1.1.1 C3S serverstart procedure.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    49/100

    3AT 21188 AWAA TNZZA 49/100

    6 ADMINISTRATION ACTIVITIES

    The following administ rat ion act ivit ies are described:

    Start / stop Linux operat ing system, Start / stop / check Oracle environment for a simplex, a duplex, a JBoss cluster configuration, Start / stop / check the C3S server f or a simplex, a duplex, a JBoss cluster configurat ion, Log on the Oracle environment, Check to perform after a duplex package switch over.

    6.1 Start / Stop Linux operati ng system

    6.1.1 Shutdown / Boot levels

    Shutdown:

    init 0shutdown -h now

    -a: Use file /etc/shutdown.allow

    -c: Cancel scheduled shutdown.

    halt -p

    -p: Turn power off after shutdown.

    poweroff

    Reboot :

    init 6

    shutdown -r now

    reboot

    Enter single user mode:init 1

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    50/100

    3AT 21188 AWAA TNZZA 50/100

    6.2 Oracle

    6.2.1 Simplex configurati on

    6.2.1.1 Start Oracle

    Server(s) C3S Linux server

    Functi onal Domain/Domain Database Administration

    Operation Reference Application Start

    Funct ional Descript ion Start the Oracle environment required for the C3S application (Oracledatabase, Listener, Apache Web server)

    Prerequisite To be logged on the Linux server

    Other DocumentApproximate Duration A few minutes

    FrequencyIf required

    Steps to proceed:The required Oracle runtime environment by the C3S application is composed of several items:

    The Oracle database, The l istener, The Apache Web server.

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    51/100

    3AT 21188 AWAA TNZZA 51/100

    6.2.1.1.1 Start Oracle environment

    The C3S Oracle environment can be started in one shot using the system start -up scripts under t he rootuser account:

    [root@c3sn1 ~]# /etc/init.d/dbora start

    Starting Oracle database and listenerProcessing Database instance "C3S": log

    file /oracle/product/10.2/startup.log

    SQL*Plus: Release 10.2.0.1.0 - Production on Thu Apr 24 18:22:16 2008

    Copyright (c) 1982, 2005, Oracle. All rights reserved.

    Connected to:

    Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production

    With the Partitioning, OLAP and Data Mining options

    SQL> Available

    PL/SQL procedure successfully completed.

    SQL> All PL/SQL packages and functions are valid

    Platform properties have been loaded correctly

    Platform location has been seeded correctly

    NLS messages have been loaded correctly

    The platform service is available

    Service script is accessible to the database server

    Connection information stored within the repository is correct

    PL/SQL procedure successfully completed.

    SQL> Disconnected from Oracle Database 10g Enterprise Edition Release

    10.2.0.1.0 - Production

    With the Partitioning, OLAP and Data Mining options

    opmnctl: starting opmn and all managed processes...

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    52/100

    3AT 21188 AWAA TNZZA 52/100

    6.2.1.1.2 Start Oracle listener

    The Oracle listener status can be checked under t he Linux user account oracle using the followingcommand:

    [oracle@c3sn1 ~]$ lsnrctl status

    LSNRCTL for LINUX: Version 10.2.0.1.0 - Production on 24-APR-2008 18:23:34

    Copyright (c) 1991, 2005, Oracle. All rights reserved.

    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=c3sn1)(PORT=1521)))

    STATUS of the LISTENER

    ------------------------

    Alias LISTENER

    Version TNSLSNR for Linux: Version 10.2.0.1.0 - Production

    Start Date 24-APR-2008 18:22:11

    Uptime 0 days 0 hr. 1 min. 23 sec

    Trace Level off

    Security ON: Local OS AuthenticationSNMP ON

    Listener Parameter File /oracle/product/10.2/network/admin/listener.ora

    Listener Log File /oracle/product/10.2/network/log/listener.log

    Listening Endpoints Summary...

    (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=c3sn1)(PORT=1521)))

    Services Summary...

    Service "C3S" has 2 instance(s).

    Instance "C3S", status UNKNOWN, has 1 handler(s) for this service...

    Instance "C3S", status READY, has 1 handler(s) for this service...

    Service "C3S_XPT" has 1 instance(s).

    Instance "C3S", status READY, has 1 handler(s) for this service...

    The command completed successfully

    The Oracle listener is start ed under the Linux user account oracle using the following command:

    [oracle@c3sn1 ~]$ lsnrctl start

    LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 24-APR-2008 18:24:30

    Copyright (c) 1991, 2005, Oracle. All rights reserved.

    Starting /oracle/product/10.2/bin/tnslsnr: please wait...

    TNSLSNR for Linux: Version 10.2.0.1.0 - Production

    System parameter file is /oracle/product/10.2/network/admin/listener.ora

    Log messages written to /oracle/product/10.2/network/log/listener.log

    Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=c3sn1)(PORT=1521)))

    Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=c3sn1)(PORT=1521)))

    STATUS of the LISTENER

    ------------------------

    Alias LISTENER

    Version TNSLSNR for Linux: Version 10.2.0.1.0 - Production

    Start Date 24-APR-2008 18:24:30

    Uptime 0 days 0 hr. 0 min. 0 sec

    Trace Level off

    Security ON: Local OS Authentication

    SNMP ON

    Listener Parameter File /oracle/product/10.2/network/admin/listener.ora

    Listener Log File /oracle/product/10.2/network/log/listener.logListening Endpoints Summary...

    (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=c3sn1)(PORT=1521)))

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    53/100

    3AT 21188 AWAA TNZZA 53/100

    Services Summary...

    Service "C3S" has 1 instance(s).

    Instance "C3S", status UNKNOWN, has 1 handler(s) for this service...

    The command completed successfully

    6.2.1.1.3 Start Oracle database

    Using the Oracle command under t he oracle user account:

    [oracle@c3sn1 ~]$ dbstart

    Processing Database instance "C3S": log file /oracle/product/10.2/startup.log

    Note: This command is operational only when the fl ag indicating if the instance should automat ically start atboot t ime (Y=yes, N=no) is set to Y in the Oracle / etc/ oratab configurat ion fi le as follows:

    C3S:/ oracle/ product / 10.2:Y

    Using the sqlplus interface under t he oracle user account:

    [root@c3sn1 ~]$ su - oracle

    [oracle@c3sn1 ~]$ export ORACLE_SID=C3S

    [oracle@c3sn1 ~]$ sqlplus /nolog

    SQL*Plus: Release 10.2.0.1.0 - Production on Thu Apr 24 18:29:53 2008

    Copyright (c) 1982, 2005, Oracle. All rights reserved.

    SQL> connect sys/sysadm as sysdba

    Connected to an idle instance.

    SQL> startup

    ORACLE instance started.

    Total System Global Area 1375731712 bytes

    Fixed Size 1219328 bytes

    Variable Size 369100032 bytes

    Database Buffers 989855744 bytes

    Redo Buffers 15556608 bytes

    Database mounted.

    Database opened.

    SQL>

    6.2.1.1.4 Start Apache Web server

    The Oracle Apache Web Server status can be checked under t he Linux user account oracle using thefollowing command:

    [root@c3sn1 ~]$ su - oracle

    [oracle@c3sn1 ~]$ cd product/10.2/companion/opmn/bin/[oracle@c3sn1 bin]$ ./opmnctl start

    opmnctl: opmn started

  • 8/2/2019 8965 C3S Administrator Guide 3AT21188AWAATNZZA 01

    54/100

    3AT 21188 AWAA TNZZA 54/100

    Post procedure: Check that Unix processes related to C3S Oracle database are running on the server using the ps

    command:

    [oracle@c3sn1 oracle]$ps -eaf | grep ora_

    oracle 25659 1 0 18:37 ? 00:00:00 ora_pmon_C3S

    oracle 25661 1 0 18:37 ? 00:00:00 ora_psp0_C3S

    oracle 25663 1 0 18:37 ? 00:00:00 ora_mman_C3S

    oracle 25665 1 0 18:37 ? 00:00:00 ora_dbw0_C3S

    oracle 25667 1 0 18:37 ? 00:00:00 ora_lgwr_C3S

    oracle 25669 1 0 18:37 ? 00:00:00 ora_ckpt_C3S

    oracle 25671 1 0 18:37 ? 00:00:00 ora_smon_C3S

    oracle 25673 1 0 18:37 ? 00:00:00 ora_reco_C3S

    oracle 25675 1 0 18:37 ? 00:00:00 ora_cjq0_C3S

    oracle 25677 1 0 18:37 ? 00:00:00 ora_mmon_C3S

    oracle 25679 1 0 18:37 ? 00:00:00 ora_mmnl_C3S

    oracle 25683 1 0 18:37 ? 00:00:00 ora_qmnc_C3S

    oracle 25720 1 5 18:37 ? 00:00:00 ora_j000_C3Soracle 25791 1 0 18:37 ? 00:00:00 ora_q000_C3S

    oracle 25823 1 0 18:37 ? 00:00:00 ora_q001_C3S

    Check that j ava process related to OWB runt ime, the runt ime service, i s running on the server using the ps command:

    [oracle@c3sn1 oracle]$ps -eaf | grep java

    oracle 25723 25721 1 18:37 ? 00:00:00 ../../../jdk/jre/bin/java -Xmx768M -

    Djava.awt.headless=true -DORACLE_HOME=/oracle/owb_10.2 -DOCM_HOME=/oracle/owb_10.2 -

    DOCM_ORACLE_HOME=/oracle/owb_10.2 -classpath

    ../admin:../admin/launcher.jar:Personalties.jar:: -DOWB_HOME=/oracle/owb_10.2 -

    DJDK_HOME=../../../jdk -DORA_HOME=/oracle/owb_10.2 -DOEM_HOME=/oracle/owb_10.2 -

    DIAS_HOME=/oracle/owb_10.2 Launcher ../admin/owb.classpath

    oracle.wh.runtime.platform.service.Service -manual 1 C3SOWBRUN LOCALHOST 1521 C3S