HEASARC and Related Data Center Activities
description
Transcript of HEASARC and Related Data Center Activities
HEASARC and Related Data Center Activities
HEASARC and Related Data Center Activities
Frank Marshall
Acting HEASARC Director
NASA’s archive for X-ray and Gamma ray data
• Established in Nov 1990 – First wavelength specific “active” archive
• Partnership between GSFC and SAO (since 1999)• Contains data from 26 missions • All data in FITS format, along with associated software and calibrations• Provides the necessary scientific and technical expertise for the use and
interpretation of the data • Develop/maintain multi-mission analysis/support tools such as XSPEC,
PIMMS, and Proposal Submission• Online access to all data, catalogs of observations and sources and
browse data products• Defines and coordinates data, software, and media standards
HEASARC OverviewHEASARC Overview
Laboratory for High Energy Astrophysics OrganizationLaboratory for High Energy Astrophysics Organization
LHEAFrank Jones (acting)
Lois Workman660
X-ray Astrophysics Branch
Robert Petre662
Gamma Ray, Cosmic Ray & Gravitational Wave Astrophysics Branch
Neil Gehrels661
Instrument Development Office
Lois Workman (acting)660.2
Office of Guest Investigator Programs
& Data ManagementFrank Marshall
660.1
HEASARC Overall OrganizationHEASARC Overall Organization
HEASARC provides the multi-mission infrastructure that is used by the GOFs and science support centers:
Archive, database, web services, FITS standards, multi-mission software, & expertise
Science staff are co-located with LHEA science branches
HEASARC provides the multi-mission infrastructure that is used by dedicated mission data centers (RXTE, Chandra, XMM-Newton, INTEGRAL, Swift, Astro-E2, and GLAST)
Archive infrastructure, database support, web services, proposal software, FITS standards, multi-mission software, & science expertise
Prevents duplication of effort and promotes reuse of software which result in cost savings to missions
At the end of mission life the HEASARC maintains the archive, science expertise and software (e.g. CGRO, ASCA, ROSAT)
Astro-E2
INTEGRAL
Swift
GLAST
Rossi XTE
XMM-Newton
HEASARC Mission SupportHEASARC Mission Support
Chandra
HEASARC Budget & StaffingHEASARC Budget & Staffing
• Budget of $1.5M that mostly pays for people
• LHEA Science Staff (4 USRA, 1 UMD, 3 GSFC): – Angelini (Restoration, BeppoSAX, Swift, XIMAGE, XRONOS)– Arnaud (XSPEC, XSELECT)– Corcoran (ROSAT, Caldb, HETE-2)– Drake (User support, EUVE, Archive population)– Lochner (E/PO)– McGlynn (Archive Scientist, VO, SkyView, ClassX)– Pence (FTOOLS, FITSIO, FITS standards, HERA, FV)– Frank Marshall (Acting Director while N. White is at HQ)
• Two addition FTE at SAO (separately funded)
• Science staff have 30% time for research– Successfully compete for observing time e.g. XMM-Newton and Chandra
• Programming Staff (5 Software contractor, 1 GSFC)– Database, archive access, data ingest, data restoration, web services,
XANADU analysis software, education & outreach, system support, and user support.
• Additional funding competed for via e.g. AISRP
HEASARC Co-located Data CentersHEASARC Co-located Data Centers
• Provide dedicated support to specific missions utilizing HEASARC infrastructure– Develop mission specific software
– Interface between community, instrument teams and/or foreign data center
– Data processing and archive creation
– Proposal support and grants administration
– Science expertise in the specific mission
• Currently there are 7 active GOFs/SSCs:– XMM-Newton, RXTE, BeppoSAX (ASI), INTEGRAL, Swift, Astro-E2, GLAST
– 1 to 5 scientists per mission (typically 3), plus comparable number of programmers and support staff
– Co-located staff at foreign data centers (XMM-Newton, INTEGRAL, Astro-E2)
– Current total cost ~ $3M (depending on mission phase)
SAO ConnectionSAO Connection
• Coordinates SAO Chandra and HEASARC activities– Chandra calibration data & archive interface– Transparent access to Chandra archive from
HEASARC Browse– DS9 Software– Remote Proposal Submission Software– Two FTE staff at SAO– Steve Murray directs the activity
Actions from Previous HUGActions from Previous HUG
• Offsite archive backup. Backup in second building is planned.• Consistent statistics for outreach web-sites. Done.• Browse suggestions/concerns: plotting, cross-correlation radii, data
products selection, exposure times, results formatting. Some implemented, others in process.
• Need HETE-2 Data in FITS. Limited FITS support, see HETE-2 talk.
• nH issues, limits. Link to MAST-supported alternative
• Integral simulator and calibration concerns. See Integral Talk.• NVO impact on HEASARC. ADEC White Paper.
HighlightsHighlights
• INTEGRAL launch• New load-balancing Web and database servers• 2500 Chandra and 1100 XMM datasets in archive by end of
2002.• Browse full integration of remote (Chandra) data products.• Bibliography/data links to and from ADS for XMM, RXTE,
ASCA, ROSAT• XTE table and data product simplification.• GRBCAT and “Is it an X-ray source?” tools.• Outreach CD rated “outstanding” in NASA review• SkyView featured in Scientific American and Wired.• GRB follow-up VO prototype shown at AAS.
Highlights: HEADCC MeetingHighlights: HEADCC Meeting
• Coordinate activities of all high-energy astrophysics missions and centers.
• Held Nov. 13, 2002 at CfA• Agreements
– Work to ensure interoperability of Chandra and XMM data sets.– Coordinate development of POW, FV and DS9 viewing tools– Look at upgrading CALDB to meet requirements of Chandra and
other newer missions.– Continue to work towards a single parameter interface layer.– Explore building standard model for coded aperture mask
instruments.
The Physical ArchiveThe Physical Archive
Active Missions
RXTE (1995- )Chandra (1999- ) [data at CXC]HETE-2 (2000- ) Integral (2001-)XMM-Newton (1999- )
Past Missions
Ariel 5 EXOSATASCA GingaBBXRT BeppoSAX CGRO HEAO 1Copernicus HEAO 3COS B OSO 8DXS ROSAT Einstein SAS 2EUVE SAS 3 Vela 5B
• Data from 26 missions currently in the archive
• 344 astronomical catalogs & mission tables (up from 307 at last HUG meeting)
• The archive volume was2700 Gigabytes as of the end of 2002
Upcoming Missions
Swift (late 2003)Astro-E2 (2005)GLAST (2006)
Usage & Data StatisticsUsage & Data Statistics
Gigabytes retrieved per quarter
Gigabytes transferred per year1998 1999 2000 2001 2002
FTP 842 1391 1846 2251 2481WWW (http) 424 603 1083 1428 2931
Browse tar file directory
Data Transfer Rates (ftp & http)Data Transfer Rates (ftp & http)
The ftp transfer rate in 2002 averaged 6.8 GB per day, up from 6.2 GB per day in 2001. Data transferred was predominantly RXTE (56%) with other missions of note, ASCA and ROSAT (3.7% each), CGRO (3.2%) and XMM-Newton (2.7%). Significant data transfers were also made from non-mission directories such as Software (17%) and Retrieve (13%).
Data Attractiveness (DA, defined for each mission as data ftped in a year divided by the archive size):
RXTE DA=129% ROSAT DA=63%CGRO DA=37% XMM DA=29%
The http transfer rate of Web pages, images, etc., in 2002 was 8 GB per day (244 GB per month), compared to 3.9 GB per day (119 GB per month) in 2001. The http transfer rate in 2002 was 118% of the ftp rate - the first time the http transfers exceeded the ftp transfers.
Monthly transfers from the HEASARC’s EPO areas “Imagine the Universe” and “StarChild” averaged ~500,000 requests and ~1,100,000 requests during the school years, respectively. They get fewer hits during the summer.
HEASARC Data ArchiveHEASARC Data Archive
As of November 2002, the HEASARC’s public data holdings (the anonymous ftp area) were about 2688 GB, including some still-proprietary datasets. (Duplicate ROSAT and ASCA datasets totaling ~160 GB have been removed from these statistics.) The biggest mission archives are RXTE (40% of the total) and ASCA (20%), with XMM-Newton and CGRO each ~8%, ROSAT and BeppoSAX each ~5%, and EXOSAT and HETE-2 just under 4% each.
In the last 12 months, the HEASARC archive growth was about 1.4 GB per day (~500 GB over the year), to which the rapid growth of the XMM-Newton archive and the opening of the HETE-2 archive were major contributors. The growth rate in 2003 is expected to be similar.
Future Archive GrowthFuture Archive Growth
In the next 3 years, there are several new missions that plan to archive data at the HEASARC:
Swift (late 2003 launch)Astro-E2 (early 2005 launch)GLAST (fall 2006 launch)
These new missions are expected to each generate 400-1000 GB of data per year so the HEASARC archive doubling time is predicted to be about 2.5 years.
Very rough HEASARC archive size projections:Date Archive size2004.0 41202005.0 60552006.0 7990
Summary and ProspectsSummary and Prospects• The HEASARC is a mature system with diverse and growing
holdings.• Archive and software systems continue to evolve to meet the
needs of the community• Over the next several years, many of the newer missions will
operate in the hard X-ray and gamma-ray regimes.– Science goals, operations and other requirements somewhat different from
many earlier missions.
• The diversity of the HEASARC holdings makes it a smaller scale ‘Virtual Observatory’– Links to other NASA and ground data enhances value of HEASARC holdings– HEASARC experience can be valuable in guiding VO efforts– Conversely, more generic VO approaches may be useful within the
HEASARC.