A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service...
Transcript of A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service...
![Page 1: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/1.jpg)
A Technical Overview
Bill Branan DuraCloud Technical Lead
![Page 2: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/2.jpg)
Agenda
• Introduction
• Storage
• Services
• Reporting
• UI
• Security
• Tools
• Roadmap
![Page 3: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/3.jpg)
Introduction
• DuraCloud is: – Hosted service
• Runs on cloud-based compute systems
• Connects to cloud-based storage systems
• Provides a service platform
– Open source software suite
• Webinar Focus: – Components of software platform
– Interacting with hosted DuraCloud
![Page 4: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/4.jpg)
File System Data
Repository
User Data Center
DuraCloud Management Console • Account Management • Instance Management
DuraCloud in Context
![Page 5: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/5.jpg)
Administrative User Interface (DurAdmin)
Storage Management (DuraStore)
Service Management (DuraService)
Service Container
Report Management (DuraReport)
DuraCloud Instance
REST API
REST API
REST API
DuraCloud Storage
![Page 6: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/6.jpg)
Storage Provider Interface
SDSC Cloud Storage
Microsoft Azure Storage
Rackspace Cloud Files
Amazon S3
User Data Center
Storage Management (DuraStore)
Amazon Storage Adapter
REST API
Rackspace Storage Adapter
Azure Storage Adapter
SDSC Storage Adapter
Storage Mediation
DuraCloud Storage
![Page 7: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/7.jpg)
Storage REST Interface
Space Actions
• Add Space
• Get/Set Space Properties
• Get Spaces List
• Get Space Content List
• Get/Set Space Access
• Delete Space
Content Actions
• Add Content
• Get/Set Content Properties
• Get Content
• Copy Content
• Delete Content
Other Actions • Get Stores • Get Tasks List • Perform Task
Storage Provider Interface
![Page 8: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/8.jpg)
Storage Adapters
Transform calls from DuraCloud interface into calls understood by a specific storage provider
• Determine how DuraCloud “spaces” translate into top level provider containers (buckets)
• Define how space and content properties are stored and retrieved
• Manage eventual consistency concerns
![Page 9: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/9.jpg)
Administrative User Interface (DurAdmin)
Storage Management (DuraStore)
Service Management (DuraService)
Service Container
Report Management (DuraReport)
DuraCloud Instance
REST API
REST API
REST API
DuraCloud Services
![Page 10: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/10.jpg)
DuraCloud Services
Bit Integrity Checker
Bit Integrity Checker
Tools
Bit Integrity Checker
Bulk
Duplicate on
Demand
Duplicate on
Change
Media Streamer
Image Transformer
Bulk
Image Transformer
Image Server
… so how are they deployed?
![Page 11: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/11.jpg)
Service Manager
REST API
Service Management (DuraService)
Service Registry
Service Bundle
Service Config
Service Plan
Services Available?
Services In Plan?
Service List
1
2
3
4
Service List
Part 1: Ask which services are available
DuraCloud Service Deployment
![Page 12: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/12.jpg)
Service Manager
REST API
Service Management (DuraService)
Service Registry
Service Bundle
Service Config
Service Plan
DuraCloud Service Deployment
Deploy Service
Retrieve Service Bundle
Service Bundle
1
2
3
Part 2: Deploy Service
Service Config
Service Configurator
Deployed Service Config
Deployed Service
Service Container
4 Deploy Service Service Config
Service Bundle
Service Bundle
![Page 13: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/13.jpg)
Service REST Interface
• Get Services
• Deploy Service
• Get (Deployed) Service
• Get Deployed Service Properties
• Update Service Configuration
• UnDeploy Service
![Page 14: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/14.jpg)
DuraCloud Instance Services
Bit Integrity Checker
Bit Integrity Checker
Tools
Bit Integrity Checker
Bulk
Duplicate on
Demand
Duplicate on
Change
Media Streamer
Image Transformer
Bulk
Image Transformer
Image Server
DuraCloud Distributed
Services
• Runs on DuraCloud Instance • Connects to DuraStore • Can be direct Java service • Can be deployed web app
• Runs primarily outside of DuraCloud Instance
• Connects to DuraStore • Makes use of cloud network
or computation features
![Page 15: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/15.jpg)
Service Manager
REST API
Service Management (DuraService)
Service Registry
Bundle
Config
Service Plan
DuraCloud Bulk Service Deployment
1
2
3
Config
Deployed Service
Service Container
4
Task Management (DuraStore)
REST API
Hadoop
5
Run Service
Task 6
Run Hadoop
Job
![Page 16: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/16.jpg)
Administrative User Interface (DurAdmin)
Storage Management (DuraStore)
Service Management (DuraService)
Service Container
Report Management (DuraReport)
DuraCloud Instance
REST API
REST API
REST API
DuraCloud Reporting
![Page 17: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/17.jpg)
Report Management (DuraReport)
Storage Management (DuraStore)
Service Management (DuraService)
REST API
DuraCloud Report Generation
Message Broker
Storage Report Builder
Service Report Builder
Deposit Reports
Service Event Messages
REST API
Storage Data
![Page 18: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/18.jpg)
Report REST Interface
Storage Report
• Start Storage Report
• Get Latest Storage Report
• Get Storage Report List
• Get Storage Report
• Get Storage Report Info
• Cancel Storage Report
• Schedule Storage Report
• Cancel Storage Report Schedule
Service Report
• Get Deployed Services Report
• Get Latest Completed Services Report
• Get Completed Services Report List
• Get Services Report
![Page 19: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/19.jpg)
Administrative User Interface (DurAdmin)
Storage Management (DuraStore)
Service Management (DuraService)
Service Container
Report Management (DuraReport)
DuraCloud Instance
REST API
REST API
REST API
DuraCloud UI
![Page 20: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/20.jpg)
Storage Management (DuraStore)
Service Management (DuraService)
REST API REST API
Report Management (DuraReport)
REST API
Administrative User Interface (DurAdmin)
Store Client Service Client Report Client
DuraCloud UI Interaction
![Page 21: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/21.jpg)
DuraCloud Security
Storage Provider
DurAdmin
DuraStore
DuraService
DuraReport
Application Security
Storage Provider Access Control
User Data Center
Transport Security
Instance Firewall
![Page 22: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/22.jpg)
DuraCloud Tools
Transfer to DuraCloud • Upload Tool
– GUI to select files and folders
– Visual transfer progress
• Sync Tool – Monitors for changes – Run one time or
continually
• Chunk Tool – Transfer large files – Acts on a single file
Transfer from DuraCloud • Retrieval Tool
– Retrieve files from one, selected, or all spaces
• Stitch Tool – Retrieve large files – Acts on a single file
![Page 23: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/23.jpg)
Storage Management (DuraStore)
REST API
DuraCloud Tools
User Data Center
Store Client
Chunk Tool
Sync Tool
Store Client
Stitch Tool
Retrieval Tool
Upload Tool
![Page 24: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/24.jpg)
Near Term Road Map New features • Access control at the space level • Improved format identification • Automated bit integrity checking • Email notification on service completion • Multi-Account DuraCloud instances • Content indexing and Search
– Improved discovery – Improved content browsing
• Folder-based navigation • Improved handling of very large files • Multi-Item updates/deletes Integrations • SDSC storage provider (production version) • Fedora-in-the-cloud prototype • DSpace-in-the-cloud prototype
![Page 25: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/25.jpg)
Long Term Road Map New Features
• Video transcoding
• Document format transformation
• Improved image support services
• Media streaming access controls
• Metadata extraction services
Integrations
• Azure storage provider (production version)
• Eucalyptus Walrus storage provider
• Rackspace compute instances
• Managed Fedora-in-the-cloud
• Managed DSpace-in-the-cloud
![Page 26: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/26.jpg)
Questions?
![Page 27: A Technical Overviewdocs.duraspace.org/documents/DuraCloud-Technical... · Hadoop 5 Run Service Task 6 Run Hadoop Job . Administrative User Interface (DurAdmin) Storage Management](https://reader034.fdocuments.in/reader034/viewer/2022050510/5f9ac769ba355806ce47b273/html5/thumbnails/27.jpg)
Upcoming DuraCloud Webinars
Fedora and DuraCloud January 11, 2012 at 1pm ET
DSpace and DuraCloud November 30, 2011 at 1pm ET