DuraSpace DSpace Technology Overview Bradley McLean DuraSpace Open Repositories 2009 05/20/2009
Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open...
Transcript of Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open...
![Page 1: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/1.jpg)
DuraCloudFederated Repositories and Cyberinfrastructure
Open technologies and services for managing durable data in the cloud
Bradley McLean, CTO DuraSpace
Thursday, October 15, 2009
![Page 2: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/2.jpg)
Open Source Portfolio
DuraCloud
Thursday, October 15, 2009
![Page 3: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/3.jpg)
Goals of DuraSpace• Stewardship:
– Support and align open source development communities for DSpace and Fedora
• Innovation:– Think beyond existing platforms – New strategies for enabling access and
preservation of digital content• Sustainability:
– Develop business model to sustain the non-profit and open technologies we support
–Thursday, October 15, 2009
![Page 4: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/4.jpg)
DSpace and Fedora Installations
Largest share of open repositories worldwide… over 700 institutions tracked in our registries
UniversitiesResearch CentersLibrariesArchivesCultural HeritageGovernmentMore…
Thursday, October 15, 2009
![Page 5: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/5.jpg)
Challenges(From our communities)
Digital preservation and archiving is hard to achieve , even just basic replication
Making digital content more accessible and useable to researchers
Easy and elastic provisioning of shared infrastructure (also across institutions!)
Robust compute environments for data mining and analysis of large datasets
Thursday, October 15, 2009
![Page 6: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/6.jpg)
Implications for our future work
mor
e d
istri
bute
d
mor
e co
llabo
rativ
e
mor
e w
eb-o
rient
ed
mor
e op
en
mor
e in
tero
pera
ble
Thursday, October 15, 2009
![Page 7: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/7.jpg)
What About the Cloud?
A style of computing where massively scalable IT-related capabilities are provided “as a service” using Internet technologies to multiple external customers.
(Gartner, 6/08).
Thursday, October 15, 2009
![Page 8: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/8.jpg)
Cloud services
Thursday, October 15, 2009
![Page 9: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/9.jpg)
Public Cloud ServicesElastic web-based infrastructure for storage and compute
Thursday, October 15, 2009
![Page 10: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/10.jpg)
Economies of Scale and Cost
Public cloud providers drive cost down through scale, location and virtualization
technology
Large Datacenters (tens of thousands of computers) Medium Datacenters (thousands)
Source: Hamilton, Internet-Scale Service Efficiency,, LADIS Workshop (Sept 08)
Technology* Cost Medium Datacenter
Cost Large Datacenter
Network $95 per Mbit/sec/mo
$13 per Mbit/sec/mo
Storage $2.20 per Gbyte/mo $.40 per Gbyte/mo
Admin 140 servers/admin >1000 servers/admin
Thursday, October 15, 2009
![Page 11: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/11.jpg)
Study of 605 government IT
Yet, only 13% utilizing cloud compute today
http://www.meritalk.com/2009-cloud-consensus.phpThursday, October 15, 2009
![Page 12: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/12.jpg)
Barriers
http://www.meritalk.com/2009-cloud-consensus.phpThursday, October 15, 2009
![Page 13: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/13.jpg)
Here to stay
http://www.meritalk.com/2009-cloud-consensus.php
Thursday, October 15, 2009
![Page 14: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/14.jpg)
DuraCloud PropositionTrust and durability in the cloud
DuraCloud is a platform aimed at supporting libraries, universities, and other cultural heritage organizations that wish to provide perpetual access to their digital content. The service replicates and distributes content across multiple cloud providers and enables the deployment of services to support:
* access * preservation * re-use
Thursday, October 15, 2009
![Page 15: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/15.jpg)
DuraCloudA web based service enabling management of
Data in the cloud
DuraCloudmediating
webService
Sun
EMCRackspace
Microsoft
Thursday, October 15, 2009
![Page 16: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/16.jpg)
Vision: Preservation Support
DuraCloud: content replication, auditing, and repair
Thursday, October 15, 2009
![Page 17: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/17.jpg)
Vision: Shared infrastructure
DuraCloud: collaboration and data linking of stored objects
Thursday, October 15, 2009
![Page 18: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/18.jpg)
Vision: Data Analysis and Mining
DuraCloud: running large compute jobs on stored content
Thursday, October 15, 2009
![Page 19: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/19.jpg)
Thursday, October 15, 2009
![Page 20: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/20.jpg)
Thursday, October 15, 2009
![Page 21: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/21.jpg)
DuraCloud Underlying software
• Open coreCore components available for others to
build on and runOpen source
• Architecture to create cloud networksPublic cloudsPrivate cloudsUniversity consortia
• Also useful in research partnerships
Thursday, October 15, 2009
![Page 22: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/22.jpg)
Preservation Services
-ability to replicate content to multiple providers and locations
-ability to synchronize backup with primary store or repository system
-management ,monitoring, audit and repair through web based interface
Thursday, October 15, 2009
![Page 23: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/23.jpg)
software services• Other DuraSpace-provided services on
top of content stored in the cloud – Data mining– Video Streaming– Format transformation– Repository hosting– discovery
Thursday, October 15, 2009
![Page 24: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/24.jpg)
DuraCloud: run your application as a service on content
Enable others to build and deploy services and apps in DuraCloud environment
Thursday, October 15, 2009
![Page 25: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/25.jpg)
Partners and Pilots• Selected initial cloud providers
• Selected 2 initial pilot partners
Thursday, October 15, 2009
![Page 26: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/26.jpg)
NYPL pilot
• -back up copy 700k images (50 TB data)
• -transformation from Tiff to JPEG 2000
• -run image server in cloud
• -Push JPEG 2000 back into Fedora Repository
Digital Gallery Collection
Thursday, October 15, 2009
![Page 27: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/27.jpg)
BHL pilot
• -back up copy entire corpus (40 TB data)
• -have multiple copies including Europe
• -Do compute intensive data mining over corpus
BioDiversity Heritage Library
Thursday, October 15, 2009
![Page 28: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/28.jpg)
Pilot use cases• NYPL• Replication and preservation support• Format conversion• Instant provisioning of image server• Synchronization with repository• BHL• Replication and preservation support• International collaborative infrastructure• Researcher platform for data mining
Thursday, October 15, 2009
![Page 29: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/29.jpg)
Timeline• Begin pilots(MOU’s in place) – September
2009• DuraCloud Alpha Pilot release- Oct 2009• Pilot data loading and testing – Fall 2009• Beta for repository community - Q1 2010• Pilot testing with software services Q1 2010• Cloud partner evaluations complete-Q2 2010• Strategic cloud partnerships in place- Q2
2010• Pricing Model determined-Q2 2010• Report pilot results – Q2 2010• Launch production service Q3 2010
Thursday, October 15, 2009
![Page 30: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/30.jpg)
Critical success factors
• Ease of use- simplicity• Trusted partner for end user• Cost effective• Scalable/Flexible• Can establish key partnerships with
service providers• Can build community of developers and
users
Thursday, October 15, 2009
![Page 31: Title Here for Preso€¦ · DuraCloud Federated Repositories and Cyberinfrastructure Open technologies and services for managing durable data in the cloud Bradley McLean, CTO DuraSpace](https://reader035.fdocuments.in/reader035/viewer/2022071217/604af9b0691611408a7a4ce8/html5/thumbnails/31.jpg)
Thank YouFor more information:DuraSpace Organization: http://duraspace.orgWiki: http://www.fedora-commons.org/confluence/display/duracloudpilot/[email protected] [email protected] Thursday, October 15, 2009