Leading change; planning for success – ESM migration and ... · High availability. What are you...

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Leading change; planning for success – ESM migration and upgrade best practices Archana Bharathidasan, Senior Software Engineer Brian Freedman, Senior Cyber Security Consultant

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 3

This is a rolling (up to three year) Roadmap and is subject to change without notice.

Forward-looking statements

This document contains forward looking statements regarding future operations, product development, product capabilities and availability dates. This information is subject to substantial uncertainties and is subject to change at any time without prior notification. Statements contained in this document concerning these matters only reflect Hewlett Packard's predictions and / or expectations as of the date of this document and actual results and future plans of Hewlett-Packard may differ significantly as a result of, among other things, changes in product strategy resulting from technological, internal corporate, market and other changes. This is not a commitment to deliver any material, code or functionality and should not be relied upon in making purchasing decisions.


This is a rolling (up to three year) Roadmap and is subject to change without notice.

HP confidential information

This Roadmap contains HP Confidential Information. If you have a valid Confidential Disclosure Agreement with HP, disclosure of the Roadmap is subject to that CDA. If not, it is subject to the following terms: for a period of 3 years after the date of disclosure, you may use the Roadmap solely for the purpose of evaluating purchase decisions from HP and use a reasonable standard of care to prevent disclosures. You will not disclose the contents of the Roadmap to any third party unless it becomes publically known, rightfully received by you from a third party without duty of confidentiality, or disclosed with HP’s prior written approval.


Agenda

• Resource migration • Upgrade • Troubleshooting upgrade failures


Resource migration

Includes resource migration features available in ESM Next Beta


Purpose of resource migration

• Migrate resources from existing legacy ESM on Oracle to ESM on CORR-Engine

• Migrate session lists and active lists

• Upgrade schema and content based on source and target versions

Oracle CORR-E


Things to consider

Retention and reporting Who do you report to? • What Polices are you bound by?

– SOX, PCI, PII, HIPPA etc. • Do you support external or

internal customers? – Does your organization act as

an MSSP? – Window for reporting • Weekly? Daily?

Immediately?

High availability What are you responsible for? • Are you obligated by an SLA? • Does your organization have

dispersed teams? – Follow the sun

• Failover capabilities

Network What are your capabilities? • Network bandwidth • Network Latency • Hardware

– Windows/Linux? • Geographic layout


Oracle CORR-E Engine

Data flow overview

• Dictionary is read • Previous data and schema on

MySQL is dropped

• Schema is rebuilt • Data is streamed from Oracle to

MySQL

• Schema is upgraded • Content is upgraded

Extract Load

Source data or schema is never changed


Where we started – two unique ArcSight ecosystems

Network monitoring Type A • Hierarchical ESM structure • All connectors feeding one or both ESM’s depending

on correlation need or data value • Limited by hardware • 30 day retention • Used to collect all network related data

– Firewall – Proxy – VPN – Etc.

Case: Large federal contractor


Where we started – two unique ArcSight ecosystems

User and account monitoring Type B • Single ESM with Logger Tier • All connectors fed into 1 of 8 Loggers, events of

value were then forwarded to ESM • 60 day retention

– Compliance retention done in ESM • Used to collect data from WUCs and Unix servers • Limited by event volume

Case: Large federal contractor


Merge and migration strategy

Multiple ESM 5.2 instances to a singular ESM 6.0c instance

Content cleanse

• Look for content misfiring

• Duplicate content

• Rules tuned

Validate

• Validate that content migrated

• Ref/res check • Data feeds

• replicated

Migrate

• Contact your Sales Rep for a PS visit to migrate you

Merge into single instance

• Using packages, move all content to single 5.2 ESM

• VM/Test • Folder Structure

Clean ESMs

• Ref/res Check • Look for rule

errors in logs • Remove dated or

unused content


Supported migration paths

ESM 5.0 SP2 Patch 3 or higher

ESM 6.0c

Source Destination

ESM 5.5

ESM 5.2 Patch 1 or higher

ESM 6.5 SP1

This is a rolling (up to 3 year) roadmap and is subject to change without notice


Platform requirements for migration tool

• Supported on Linux OS • Installation in the same machine as destination ESM + CORR-E recommended • Machine must have at least 500 MB of free space • Manager and Web on both source and destination must be down during migration process • MySQL database must be up during the migration • Logger and Postgres must be up and running • Manager must be started only after upgrade is finished successfully • Apply all case customizations on target system, if any, prior to resource migration • When resource migration is completed, uninstall the tool


Clean ESMs

Goal: Move the smallest amount of content as possible Reason: Prevent content conflicts when merged • Use ref check or res check to help clean and identify extraneous resources • Clean out old rules • Clean out unused rules • No Real-Time content should sit in personal folders • Things to look for:

– Cull logs for error messages and identify culprits – Rules with high partial matches – Data Monitors hogging system resources


Merge into single 5.2 ESM

Goal: Bring all content together for migration Note: Can use a VM or Test/Dev box • First ESM can be moved with a table backup

– Remaining ESMs will need to use .arbs or archives to move content • Establish a folder structure to be used across all ESMs prior to merging

– Helps avoid content conflict and prevents content confusion • DO NOT move personal folders

– No technical reason, but an excellent time to establish a clean slate – If Analysts want certain content, they can re-create or move themselves later

• Users – All Users should sit in “Custom User Groups” NOT in “Default User Groups” – Link users to any subgroups to manage access control


Migrate to ESM 6.0c

Contact your sales rep Arrange a PS Consultation • PS will move your merged content to ESM 6.0c • Make sure to validate content post migration • Simple method

– Use two active channels and monitor rule fires – Both ESM 5.2 and ESM 6.0c should fire at the

same time • Ref check and res check to clean up any artifacts


Migration process steps

• Log in as user ‘arcsight’ on destination • Copy migration tool installer to destination • Run the installation file: GUI or Console

mode and follow screen prompts to completion

• Edit config/Config.databases.properties file to add Oracle and MySQL details

• Verify MySQL, Logger and Postgres on destination are up and running

• Stop the ArcSight Web and ArcSight Manager if they are running on both source and destination

• Run the Resource Migration tool using script located in bin folder ./arcsight_resourcemigration.sh

Target Machine/CORR-E

Source Machine/Oracle

Validation

Convert schema

Transfer data

Upgrade content

Process starts


Upgrade process steps

Version validation

Schema upgrade

Content upgrade

Resource validation

Target Machine/CORR-E

Source System Folder

• Schema/Content upgrade invoked automatically after successful migration to bring the schema and the standard content up to ESM 6.xc requirements

• Verify that upgrade was successful by checking • /opt/arcsight/manager/upgrade/out/

<datetime>/summary.html • Upgrade logs under

/opt/arcsight/manager/logs/upgrade • Check resource validation report

• Start Arcsight Web and Manager on the target machine


Configuration

• Config.database.properties – Oracle and MySQL database details

• SqlQueryPattern – Include/Exclude tables

• Version.properties – Version check using: • Major, Minor and Service Pack

• JRE – Embedded java

• Logs – Application log files


What gets migrated

• Resources can be migrated only to a freshly installed ESM + CORR-E instance • All resource tables that match the pattern query specified in the config file will be transferred • The following will not be migrated:

– Event data – Trend data – Case Event data – Domain fields • Fields will migrate but won’t work because domains aren’t supported on the CORR-Engine • Rule actions won’t fire on domain fields • Channels with domain fields will not have values


Upgrade

Includes upgrade features available in ESM Next Beta


Upgrade – quick look

• Supported upgrade paths: – ESM 6.0c (+ patches) to ESM Next Beta – ESM 6.5c sp1 to ESM Next Beta

• Supported platforms: RHEL 6.4, RHEL 6.5 • Older installation under /opt/arcsight • Run upgrade as user: arcsight • Upgraded components: Logger, Manager, Web • FIPS mode not supported in ESM Next Beta – upgrade of FIPS system disallowed • Rollback to earlier version not supported



Upgrade strategy

Very similar to migration strategy Reason: Reduce possibility of errors • Clean out old/unused rules • Use ref check or res check to help clean and identify extraneous resources • Things to look for:

– Cull logs for error messages and identify culprits – Rules with high partial matches – Data Monitors hogging system resources

• Familiarize yourself with the documentation


Upgrade process

Services start MySQL TZ update

Web upgrade Manager upgrade

Logger upgrade

Pre-flight checks

Stop older services (manual)



Pre-flight tasks and checks

Manual task • Run /opt/arcsight/manager/bin/remove_services.sh script as root

Pre-flight checks • OS version – RHEL 6.4 or 6.5 • User is ‘arcsight’ • Sufficient free /tmp directory space (> 3GB) • Sufficient free disk space under /opt/arcsight ( > 50GB) • Ports used by arcsight services are free • ulimit settings • /opt/arcsight contains valid supported 6.xc installation • Older 6.xc version not FIPS enabled • Postgres dump can be taken • Redundant name check test passes



Upgrade process

Logger upgrade - Backup earlier version under /opt/arcsight/logger/BLxxxx

• data under /opt/arcsight/logger/data NOT backed up

- Update postgres/mysql

- Logger configuration

- Start services

Manager upgrade - Pre-upgrade tasks

• Backup resources • Transfer configuration. - Resource schema upgrade

- System, core, foundation content upgrade

- Resource reference check

- Post-upgrade task

• Resource validation

Other upgrade tasks

- Web upgrade

• Transfer configuration files

- MySQL TZ update

- Start services

• Logger server restart • Manager start • Web start



Post-upgrade tasks

• Run setup_services.sh script as root • Verify all services are up



What has been achieved

One stop reporting • What originally took multiple

ESMs can now be achieved with only a single ESM

• All data is now available to report on

• Customers receive enriched reports

Better trending • Able to Trend over all your data • Trending over BASE Events • Be able to pull a more complete

and thorough data in your trends

• Gain access to more advanced features – Velocity Indicators

Increased correlation • With all data sources coming

into a single ESM you gain increased correlation ability

• Gain access to more advanced features – Identity View

• Asset vs Network centric monitoring

Increased capability


What else has been achieved

Access to advanced features • Velocity indicators

– How is your data changing over time

– Leverage trends to track expected behavior

• Identity view – Be able define WHO was the

actor behind an action without usernames

• ITOM integration

High availability • From the consolidators split

your event feeds to primary and secondary ESMs

• All connectors can still be managed via ConnApp

• Less destinations are less stressful for high volume connectors – Bluecoat – EPO – Juniper

Cost Discuss with your sales rep • Maintaining singular ESM • Capitalizing on open source

technology • Maximizing CORR-E DB

performance • Reduced network usage


Upgrade best practices

Prior to upgrade… • Ensure older 6.xc system is in good state • Run resource validation: /opt/arcsight/manager/bin/arcsight resvalidate -persist true

• Ensure X Server is setup correctly • Run /opt/arcsight/manager/bin/remove_services.sh script as root



Review your content

Content efficiency “What exactly am I trying to accomplish?” • Build content with single goal in mind • Catch-all content can be very inefficient • Build rule tiers;

– Identify what is bad – Identify who it is bad for

Connector efficiency The roots of ArcSight • Ensure your connectors have memory they need • Disable or tune down resource intensive settings

– Hostname resolution – Device status monitoring • Tune down

– Preserve ArcSight health events – Disable RAW event – Map connector name/customer to a field


Content development

Use all the tools available to you Work smarter not harder • Leverage your Network Model • Plan ahead

– IDview requires all uppercase usernames – Be the “lazy engineer”

• Trending vs ActiveLists – ActiveLists consume memory Trends are stored in the database

• Reporting – Parameterized Queries – Scheduled Jobs

• Global Variables to simplify content and can make future modification easier • Activate Framework


Upgrade failure


Failure detection

• Check message from installer and from high level log: – /opt/arcsight/upgradelogs/suite_upgrade.log

• Check services: – /opt/arcsight/services/init.d/arcsight_services status all

• Check individual component logs: – Logger: /opt/arcsight/logger/current/arcsight/logger/logs/ • logger_init_driver.log, initmysqluser.log, postgressql_upgrade.out

– Manager: /opt/arcsight/manager/upgrade/out/<time_of_upgrade>/ • logs/upgrade/server.upgrade.log, logs/upgrade/server.upgrade.std.log, summary.html

– Web: /opt/arcsight/web/logs/default/ • webserver.log, webserver.std.log



Recovery


Pre-upgrade failures

• Check /opt/arcsight/upgradelogs/suite_upgrade.log • Pre-flight check failures

– Modify OS configuration, settings • Pre-upgrade task failures

– Fix postgres backup failure – Fix redundant name check failure

Re-run installer/upgrader after correction.



Upgrade failure

• Check /opt/arcsight/upgradelogs/suite_upgrade.log • Contact Support. Get assistance to

– Investigate cause of failure – Identify failure stage – Receive failure recovery script with instructions

Do not attempt recovery without Support’s guidance!



Resources

• ESM Next Beta resource migration and upgrade guides are your best friends • Protect 724 website: https://protect724.arcsight.com/ • Customer support: http://support.openview.hp.com


https://protect724.arcsight.com/

http://support.openview.hp.com/


In review

What did we learn?

Build

• Leverage your new feature set to improve your capabilities

• Take advantage of new advanced features

• ITOM Integration

Maximize

• Reduce partial matches

• Question current methods

• Establish a Framework

• Active Framework

Baseline

• Make sure everything firing

• Clean again, remove duplicate content

• Make ESM just works

Merge and migrate

• File Structure to not lose content

• Use PS for your migration

• Resource Check to help clean artifacts

Plan and clean

• Understand your network and what is involved

• Clean out current ESMs

• Where will you be in 5 years


Please fill out a survey. Hand it to the door monitor on your way out.

Thank you for providing your feedback, which helps us enhance content for future events.

Session TB3059 Speakers Archana Bharathidasan, Brian Freedman

Please give me your feedback


Thank you

Leading change; planning for success – ESM migration and ... · High availability. What are you...

Documents

Transcript of Leading change; planning for success – ESM migration and ... · High availability. What are you...