RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS...

of 30 /30
RFC Operations in a RFC Operations in a Linux Environment Linux Environment DOH Science Steering Team DOH Science Steering Team Conference Call Conference Call 9 November 2004 9 November 2004

Transcript of RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS...

Page 1: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

RFC Operations in a RFC Operations in a Linux EnvironmentLinux EnvironmentDOH Science Steering Team DOH Science Steering Team

Conference CallConference Call9 November 20049 November 2004

Page 2: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

AWIPS NAS

Moving Hydro Files to New NASMoving Hydro Files to New NAS

DS

PX

/awips/hydro

/awips/hydroapps /awips/hydroapps

/awips/hydroapps/px_hydro(temporary)

• Done via a script as part of the DX/NAS installation

Page 3: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Transition to Transition to PostgreSQLPostgreSQLDOH TeleconferenceDOH Teleconference

Nov 9, 2004Nov 9, 2004

Prepared by Paul Prepared by Paul TillesTilles

Page 4: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Transition to Transition to PostgreSQLPostgreSQL/Schedule/Schedule

• AWIPS will officially move to PostgreSQLfor OB6

- Delivery to field in Fall 2005 - Scripts to convert from OB5/Informix to

OB6/PostgreSQL to be provided with install

- New OB6 executables and scripts installed

Page 5: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Transition to Transition to PostgreSQLPostgreSQL/Schedule/Schedule

• What can be done “early” at OB4 sites?

Page 6: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

• Sites should download copy of PostgreSQL7.4.6

• Sites should install on a Linux box via RPM

• Convert Informix IHFS OB4 db to PostgreSQL– Sites export Informix db on UNIX machine (OB4

snapshot)– Sites run OHD provided scripts on Linux machines to

create PostgreSQL db, create tables, primary keys, etc., edit .unl files, load data into tables

– OHD offers limited support to POCs

Transition to PostgreSQL/OB4Transition to PostgreSQL/OB4

Page 7: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Transition to PostgreSQL/OB4Transition to PostgreSQL/OB4

• Sites begin converting code and testing local apps– ecpg replaces esqlc– psql replaces dbaccess/sqlcmd

Page 8: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Transition to PostgreSQL/OB5Transition to PostgreSQL/OB5

• AWIPS installs OB5 (Spring, 2005)– Informix db converted to OB5– PostgreSQL version 7.4.5 delivered

• Convert Informix IHFS OB5 db to PostgreSQL– Sites export Informix db on UNIX machine (OB5

snapshot)– Sites run OHD provided scripts on Linux machine to

create PostgreSQL db, create tables, primary keys, etc,edit .unl files, load into tables

– OHD offers limited support to POCs

Page 9: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Transition to PostgreSQL/OB5Transition to PostgreSQL/OB5

• Sites continue work converting code and testing local apps using AWIPS installed PostgreSQL environment

Page 10: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Transition to Transition to PostgreSQLPostgreSQL/Lessons /Lessons LearnedLearned

• See HSEB website

www.nws.noaa.gov/oh/hrl/hseb/index.htm

Page 11: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Transition to Transition to PostgreSQLPostgreSQL/Current /Current Status at OHDStatus at OHD

• Currently testing OB6 versions of

– shefdecode– db_purge– DPA decoder– HydroBase

• Other apps to follow in next few months

Page 12: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Transition to Transition to PostgreSQLPostgreSQLTrainingTraining

• Most requested RFC workshop for FY05• Will be patterned after the one held in

Silver Spring for AWIPS development organizations

• Probably held at the NWS Training Center• 2nd quarter FY05 (Jan. – Mar. 2005) • Jeff Zimmerman working out details

Page 13: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Postgresql PerformancePostgresql Performance

Compared to Informix (on Linux)Compared to Informix (on Linux)At CBRFCAt CBRFC

Steve Shumate, DSST conf. call 9Nov2004

Page 14: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

MethodsMethods

• I made a reasonable effort to level the playing field, but in the end it was not perfectly level and differences are described in the next two slides about configuration and conditions

Page 15: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

ConfigurationsConfigurations

• Informix– Version 7.31(Linux)– Machine is a dual Xeon 2.2Ghz w/1.7G ram– Modified configuration

• Postgresql– Version 7.4.6 built from source code (not RPM)– Machine is a dual Xeon 2.8Ghz w/6G ram– Modified configuration

Page 16: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Conditions of the testConditions of the test

• Informix machine was operating with normal operational load

• Postgresql machine was otherwise idle• So the postgresql machine has the

advantage here, being faster with more ram and less load

Page 17: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Performance Performance –– mass loadingmass loading

• 1 million rows into one table– Informix, dbload, 1000 rows per transaction– Postgresql, psql, \copy

34

24

0 5 10 15 20 25 30 35 40

minutes to load 1million rows

PostgresqlInformix

Page 18: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Performance Performance -- queriesqueries

• Time series query from a 5 million row table with no index – ID and data-type specified without date information

0.1

11

0 2 4 6 8 10 12

seconds to returntime series query

PostgresqlInformix

Page 19: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Performance Performance –– storage efficiencystorage efficiency

• Informix – cooked file space – 37G ascii data took 69G of disk space

• Postgresql – 4.8G ascii data took 18G of disk space

54

27

0 20 40 60 80 100

storage efficiency

percent

PostgresqlInformix

Page 20: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

ConclusionsConclusions

• Mass loading performance is on par with Informix

• Storage efficiency loss can be solved with more disk space (may need 2x the disk space to store the same amount of data)

• Query performance may be similar to Informix (was ~equal when limit 1 was applied to select statement)

Page 21: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Conclusions, cont.Conclusions, cont.

• Better Performance? That depends on where you are coming from– Moving from HP-Informix it will be an

improvement– From Linux-Informix it may be a step back

• Better overall?– Open source, free– Arrays!

Page 22: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Conclusions, cont.Conclusions, cont.

• Did I mention Arrays? !!!– A better, less awkward way (than CBRFC-fastetc

method) to store time series data– May increase performance

• Maintenance & Fragmentation (extents)– Minimal!– No inter-table fragmentation with pg

• Not more than one table per file-system file– Vacuum commands will have to be ran routinely.

This is essential.

Page 23: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

RFC Ensemble RFC Ensemble ProcessorsProcessors

Page 24: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Where we want to goWhere we want to go• Purpose of REP: support Ensemble

Forecasting• Multi-Threaded applications

– Ensembles, Grids, Data assimilation, iterative model solutions

• Services oriented architecture centered on the REP

• Additional CPU as we need them• Hydro only DB engine on REP

– Multiple DB engines

Page 25: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

Current ConsiderationsCurrent Considerations• Two Network Attached Storage (NAS) devices

– AWIPS NAS best choice for daily operational data• 4 hour support

– REP NAS best choice for GIS, Calibration data• 24 hour support

• Network should be fast enough to support using AWIPS NAS for data and REP for computations– Each device on 1 gigabit LAN connected by 1

gigabit switch– REP cpu should see /awips/hydroapps

• RFC solutions/suggestions

Page 26: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

GIS for RFC Operations GIS for RFC Operations

Page 27: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

GIS for RFC OperationsGIS for RFC Operations

• April 2004 – OST agreed to consider ESRI on MS Windows as part of AWIPS baseline

• May 2004 – presentation by NGIT– Offered GRASS as alternative to ESRI

• May-June 2004 - Richards collected GIS requirements– Delay because of other priorities (Richards)– Revisit in Jan. 2005– Have third party evaluate GRASS against

requirements?

Page 28: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

GIS for RFC OperationsGIS for RFC Operations

• Is there sufficient consensus to establish requirements for a single baseline solution?

• Are GRASS and ESRI the only options?• Given budget constraints, importance of a

strong, unified approach

Page 29: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

GIS for RFC OperationsGIS for RFC Operations

• FOP– National, but support not centralized– Use requirements process to:

(1) Convert to Linux(2) Establish formal support

• CAP– Not nationally supported– Consider submitting as a requirement

Page 30: RFC Operations in a Linux Environment · 2013-12-23 · Transition to PostgreSQL/OB5 • AWIPS installs OB5 (Spring, 2005) – Informix db converted to OB5 – PostgreSQL version

GIS IssuesGIS Issues• Enterprise License Agreement with ESRI

– Although not ‘dead,’ no reason to expect agreement in near future

• Postgress– PostGIS [postgis.refractions.net]

• ‘Middleware’ to store/access geospatial info• Connectivity with:

(1) Minnesota Map Server [mapserver.gis.umn.edu/ ]

(2) GRASS (PostGRASS)(3) GeoTools Java GIS Toolkit(4) ESRI products

[www.esri.com/software/arcgis/extensions/datainteroperability/about/features1.html]