Digital Kniznica 09 Ben Osteen Oxford
-
Upload
guest045ab43 -
Category
Technology
-
view
242 -
download
1
Transcript of Digital Kniznica 09 Ben Osteen Oxford
![Page 1: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/1.jpg)
Oxford's DAMS Architecture
Ben O'SteenSoftware Engineer,
Oxford University Library Services
Neil JefferiesR&D Project Manager
Oxford University Library Services
![Page 2: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/2.jpg)
'An institutional repository needs to be a service with continuity
behind it........Institutions need to recognize that they are making
commitments for the long term.'
Clifford Lynch, 2004
![Page 3: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/3.jpg)
Continuity and Services
We anticipate that the content our systems will hold or even
hold now will outlive any software, hardware or even
people involved in its creation.
![Page 4: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/4.jpg)
Continuity and Services
What content are we anticipating to store for the long-term?
![Page 5: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/5.jpg)
Continuity and Services
● “Bookshelves for the 21st Century”● Legal Deposit● Voluntary Deposit● Digitisation● Extended Remit - Research Data● Administrative data – researchers, projects, places,
dates● And anything else deemed worthy.
![Page 6: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/6.jpg)
Continuity and Services
What core principles do you design a system with when you
know the content will outlive you?
![Page 7: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/7.jpg)
Continuity and Services
● Oxford DAMS – Digital Asset Management System
–Not a single piece of software, more a set of guiding principles and aims.
![Page 8: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/8.jpg)
Continuity and Services
#1 Keep it simple
– simple is maintainable, easy to understand.
![Page 9: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/9.jpg)
Continuity and Services
● #1 Keep it simple – simple is maintainable, easy to understand. – Think about how much we have achieved with the
simple concept of using a single command on a simple protocol (HTTP GET) to get an HTML document. (HTML being a loose, ill-constrained set of elements.)
“GET / HTTP1.1”
![Page 10: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/10.jpg)
Continuity and Services
● #1 Keep it simple – simple is maintainable, easy to understand. – simple protocols such as HTTP, simple formats
such as JSON and API patterns such as REST are heavily favoured
– Remember, at some point, administration and development of the system will need to be handed over.
![Page 11: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/11.jpg)
Continuity and Services
#2 Everything but the content is replaceable
![Page 12: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/12.jpg)
Continuity and Services
● #2 Everything but the content is replaceable– The content that is being held by the system at any
one time is what is important, not the surrounding infrastructure
– The content should survive and be reusable in the event that services like databases, search indexes, etc crash or corrupt.
![Page 13: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/13.jpg)
Continuity and Services
1st logical outcome:
Do not use complex packages to hold your content!
Minimise the amount of software and work that you need to maintain to understand how
your content is stored.
![Page 14: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/14.jpg)
Continuity and Services
● 1st logical outcome:
–Do not use complex packages to hold your content!
– The basic, low-level digital object in our system is a “bag”
● Each bag has a manifest, currently in either FOXML or RDF, which serves to identify the component parts and their inter-relationships.
![Page 15: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/15.jpg)
Continuity and Services
#3 The services in the system are replaceable and can be rebuilt from
the content.
![Page 16: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/16.jpg)
Continuity and Services
● #3 The services in the system are replaceable and can be rebuilt from the content.– Services such as:
● Search indexes (eg via Lucene/Solr)● Databases and RDF triplestores (MySQL, Mulgara, etc)● XML databases (eXist db, etc)● Object management middleware (Fedora)● Object registries
![Page 17: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/17.jpg)
Continuity and Services
● 2nd logical outcome:
–A content store must be able to tell services when items have changed, been created, or deleted, and what items an individual store holds.
– We need slightly smarter storage to do this over a network. Local filesystems have had this paradigm for a long time now, but we need to migrate it to the network level.
![Page 18: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/18.jpg)
Continuity and Services
● 2nd logical outcome:
– A content store must be able to tell services when items have changed, been created, or deleted, and what items an individual store holds.
– Each store and set of services needs a way to pass messages in a transactional way, a messaging system.
– [Tech note: we are moving from the JMS to use the new AMQP via Rabbit MQ because on balance it is much more practical, flexible and useful.]
![Page 19: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/19.jpg)
Continuity and Services
● 3rd logical outcome:
–The storage must store the information and documentation for the standards and conventions its content follows.
–The storage must be able to describe itself.
![Page 20: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/20.jpg)
Continuity and Services
#4 We must lower our dependence on any one node in the system – be it
hardware, software or even a person.
![Page 21: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/21.jpg)
Continuity and Services
● #4 We must lower our dependence on any one node in the system – be it hardware, software or even a person.– This is another reason for #1 – keeping things
simple. ● Lowers the cost of hardware replacement and upgrade –
simple hardware tends to be cheaper and more readily available; lowering the number of crucial features broadens what can be used.
● Software – Oxford has been around for 1000 years. Vendors will not be.
● People – a simpler system is more readily understood by new/replacement engineers.
![Page 22: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/22.jpg)
Challenges to maintaining Continuity
● Flexibility● Scalability● Longevity● Availability● Sustainability● Interoperability
![Page 23: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/23.jpg)
Challenges to maintaining Continuity
● Flexibility● Open standards and open APIs allow us to provide
the tools people need to create the experience with the content they want.
● 'Text' based metadata – XML, RDF, RDFa, JSON● Componentised web services providing open APIs –
flexibility through dynamically selecting what services run at anytime.
● Open Source
![Page 24: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/24.jpg)
Challenges to maintaining Continuity
● Scalability● We need to anticipate exponential growth in demand,
with the knowledge that storage will be longterm● A rough estimate of all the data produced in the world
every second is 7km of CDROMs – broadcast video, big science, CCTVs, cameras, research, etc. This figure is only ever going to go up.
● To scale like the web, we have to be like the web; not one single black box and workflow, but a distributed net of storage and services, simply interlinked.
● The “Billion file [inode] problem” must be avoided
![Page 25: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/25.jpg)
Challenges to maintaining Continuity
● Longevity● Live Replicas (Backup Problems)
● MAID (Power)● Self-healing Systems (Resilience)● Simplicity of Interfaces● Avoid 3rd Party dependence● Support Heterogeneity
● Resolvers/Abstraction Layers
![Page 26: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/26.jpg)
Challenges to maintaining Continuity
● Availability● Basic IT availability● Enhanced long-term availability● Archival recoverability● Digital Preservation
● Conversion● Emulation● Archive Preservation
![Page 27: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/27.jpg)
Challenges to maintaining Continuity
● Sustainability● Budget and cost as a conventional library● Factor archival costs into projects● Leverage content to generate income● Migrate skills
![Page 28: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/28.jpg)
Challenges to maintaining Continuity
● Interoperability
● Interoperability is an ongoing process● Support for emerging and established standards
● Persistent, stable, well-defined interfaces● Ideally implement interfaces bidirectionally● Avoid low-level interfaces – abstract as much as
feasible● Embrace the web – if you do it right, people will
reuse your content in ways you never thought of and never thought possible.
![Page 29: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/29.jpg)
VMWare ESX
Public
SA
TA
VM ImageStore (2TB)
HC 1 HC 2
MD
ICS
Phase 1 – current hardware
![Page 30: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/30.jpg)
VMWare ESX
Public
SA
TA
VM ImageStore (2TB)
HC 1 HC 2
MD
ICS
Phase 1 – current hardware
Content layer
![Page 31: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/31.jpg)
VMWare ESX
Public
SA
TA
VM ImageStore (2TB)
HC 1 HC 2
MD
ICS
Phase 1 – current hardware
Service storage layer
![Page 32: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/32.jpg)
VMWare ESX
Public
SA
TA
VM ImageStore (2TB)
HC 1 HC 2
MD
ICS
Phase 1 – current hardware
Service execution layer
![Page 33: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/33.jpg)
VMWare ESX VMWare ESX
Public
Archive
Storage
VM ImageStore (2TB)
NF
S
VM ImageStore (2TB)
NF
S
ZF
S
ZFS Replication
ZF
S
HC 1 HC 2
Private
MD
ICS
Phase 2 – (expecting hardware delivery this month)
![Page 34: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/34.jpg)
VMWare ESX VMWare ESX
Public
Archive
Storage
VM ImageStore (2TB)
NF
S
VM ImageStore (2TB)
NF
S
ZF
S
ZFS Replication
ZF
S
HC 1Thumper
1Thumper
2HC 2
Split Writes &Crosschecks
Private
Sun
Ray
sF
utur
eArc
h
MD
ICS
RSL Osney?
Aim – multisite (add sites as required to scale up)
![Page 35: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/35.jpg)
Backend services (blue)
fit over the content storage in 'stacks' and
provide capability to
other services (green)
![Page 36: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/36.jpg)
Storage layer
...
Columns of services over common content/storage layer
ORA
FMO
Databank
![Page 37: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/37.jpg)
End of Presentation
Any Questions?
![Page 38: Digital Kniznica 09 Ben Osteen Oxford](https://reader036.fdocuments.in/reader036/viewer/2022081506/55855cbad8b42a47518b5374/html5/thumbnails/38.jpg)
Ben O'[email protected]
Neil [email protected]
www.sers.ox.ac.uk
Oxford Research Archivehttp://ora.ouls.ox.ac.uk
Developer's Bloghttp://oxfordrepo.blogspot.com