Post on 11-Jan-2016
Managed by UT-Battellefor the Department of Energy
Best Ever Archive Utility, Yet
Creating a BEAST was easy.
BEAUtY seems harder.
kasemirk@ornl.gov
April 2009
2 Managed by UT-Battellefor the Department of Energy
CSS-based OPICSS-based OPI
ArchiveArchive EngineArchiveArchive Engineconfig.xmlconfig.xml
IOCIOC
Binary Data Files
Data ServerData Server
“ASCII” Config.“ASCII” Config.Channel AccessChannel Access
XML-RPCXML-RPC
Previous Setup
History~2000:
Started byBob Dalesio
~2003:Index Tools,Data Server
~2007:CSS Client
3 Managed by UT-Battellefor the Department of Energy
Details Older Java “archive viewer” also available.
Archive Engines running on “sampling” computer, data server on “serving” machine.
Scripts to restart engines, copy data, update indices.
Headaches with data maintenance:– Index time grows with data
– Stuck when index files reach 2GB
– SNS Users faced with ~80 sub-archives
– No clue what needs fixing after network/power problems
– No idea who contributes how many samples
– No way to remove selected channels or time ranges
– Improving on this means implementing an RDB
4 Managed by UT-Battellefor the Department of Energy
CSS-based OPICSS-based OPI
ArchiveEngineArchiveEngine
config.xmlconfig.xml
IOCIOC
RDB(Oracle/MySQL)
RDB(Oracle/MySQL)
Channel AccessChannel Access
New Setup
SamplesSamplesConfig.Config.
EngineConfig-Import
Other tools for config & samples
5 Managed by UT-Battellefor the Department of Energy
CSS Data Browser Handles Both New URL
Just one‘RDB’sub-archive
Old and newdata can becombined inone plot
6 Managed by UT-Battellefor the Department of Energy
Configure Sample Engines
Configuration is in RDB– Directly use SQL
– EngineConfigImport for legacy config files
– View/Edit via web
Hierarchical (as before)– Sampling engine (name, where to run, …)
Groups– Channels
No more duplicate channels!
7 Managed by UT-Battellefor the Department of Energy
Web Interface to Engine Config in RDB
Tomcat/JSP/Servlets to view and edit
8 Managed by UT-Battellefor the Department of Energy
Web Config View: Channel Stats
9 Managed by UT-Battellefor the Department of Energy
Web Config View: Sample Stats
OK
??
10 Managed by UT-Battellefor the Department of Energy
Stats 34 sample engines, 79400 channels
Host that runs sampling engine:– CPU load 45%, zero disk I/O wait, very responsive
Oracle performance– 8000 samples/sec peak in write tests
– Operationally maybe ¼ of that
Better configurationwould likely havefewer samples/sec
11 Managed by UT-Battellefor the Department of Energy
Report Example, SNS Beam Instr. Group
May be SNS-specific, but shows usefulness of data in RDB
12 Managed by UT-Battellefor the Department of Energy
Summary
Still testing, many Oracle setup issues– Ran since Oct 08, then..
– Oracle cluster setup since ~March
– Updated partitioning in April
Plan for about a year now:Make operational in maybe 2 months
Thanks go to– Oracle: Jeff Patton
– JSP: Katia Danilova
– Beam Instr. Reports: Mariano Padilla, Jeff, Douglas Edwardson
– IT: Greg Lawson
13 Managed by UT-Battellefor the Department of Energy
Stuff
14 Managed by UT-Battellefor the Department of Energy
Hurdles Months: Get new Oracle server configured
– Interface cards for storage array, fiber switches
– Firewall holes for office access, backup, admin
Changes in 10g– SELECT MIN(stamp), MAX(stamp) -> NULL, NULL
– No “range” partitioning on Index-Organized-Tables
Configuration issues– ORA-01654: unable to extend …
– ORA-00257: archiver error
What used to be impossible is now “easy”, but still expensive– DELETE FROM SAMPLES WHERE …
15 Managed by UT-Battellefor the Department of Energy
Other Ideas
Jlab’s MyA– Operational, but
Wrapper code around MySQL to create ‘cluster’ Handles less meta info (units, limits, …) Viewer is one-of C++/TCL/Tk
Gabriele Carcassi mentioned RDDTool– Toolset for logging data with data-aging
– Command-line RPN tools, web viewer
– May be faster than general-purpose RDB but lacks advantage of gpp RDB
16 Managed by UT-Battellefor the Department of Energy
SNS Setup for EngineConfigImport Tool
– Imports existing XML config files into RDB
– On srv02 in …/imported_config *.xml files copied from archive1, includes
some editing to remove duplicate channels Review *.log files
– 'chmk/chmkhebt/PPS_HEBT:PLC_B:Chmk100AlarmContactA’already in 'pps/pps_hebt’
Maybe edit “your” *.xml file, refer to import.sh to re-import
17 Managed by UT-Battellefor the Department of Energy
Getting Sources, Building binaries
Get sources from DESY CVS repository– Maybe binaries “real soon” from ChannelArchiver web page
Build binaries– Find somebody who understands “Eclipse RCP”, how to use
org.csstudio.archive.rdb/EngineConfigImport.product org.csstudio.archive.engine/ArchiveEngine.product org.csstudio.sns.product/SNS_CSS.product (or DESY CSS.product)
– Settings Compile-time defaults usually in “preferences.ini” and
“plugin_customization.ini” files. Either change, or prepare file for
-pluginCustomization command-line option
18 Managed by UT-Battellefor the Department of Energy
RDB Setup
Currently supported: MySQL or Oracle– Main table difference: Oracle TIMESTAMP has
nanosecs, MySQL has separate nanosecs col.
– All tools auto-configure based on JDBC URL
Table Schema– ERD, MySQL DBD in org.csstudio.archive.rdb/dbd
– Setup for Oracle (table types, partitioning) not final
19 Managed by UT-Battellefor the Department of Energy
Run Archive Engine
Command-line tool similar to previous engine– More settings (RDB, Logging, JCA/CAJ, …)
Compiled-in defaults from “preferences.ini” and “plugin_customization.ini”
Can specify overrides via-pluginCustomization my_plugin_customization.ini
Mandatory command-line options for instance– engine_name to locate config in RDB– port for consistency, must match settings in RDB
Built-in web server for status, restart, stop: http://host:port/: Empty page (used to be ‘main’) …./main: Main page …./reset: Reset statistics …./restart: Stop, re-read configuration, start again …./stop: Stop