High Level Requirements Specification Commonwealth Big...
Transcript of High Level Requirements Specification Commonwealth Big...
3
1. Background _______________________________________________________________________ 4
1.1. Commonwealth Secretariat _______________________________________________________ 4
1.2. IT Services ______________________________________________________________________ 4
2. Overview _________________________________________________________________________ 5
3. Key Business Requirements__________________________________________________________ 7
4. Methodology ______________________________________________________________________ 7
5. Functionality ____________________________________________________________________ 13
5.1. Key Functional Requirements _____________________________________________________ 13
5.2. Functional Component Requirements ______________________________________________ 14
6. Services to which the cloud environment can facilitate access _____________________________ 16
6.1. Internal authentication __________________________________________________________ 16
6.2. Open Data interface ____________________________________________________________ 16
6.3. Enterprise Search Functionality ___________________________________________________ 16
6.4. Taxonomies ____________________________________________________________________ 16
7. Service Quality and Delivery _______________________________________________________ 17
7.1. Hosting and Availability__________________________________________________________ 17
7.2. Reliability _____________________________________________________________________ 17
7.3. Performance ___________________________________________________________________ 17
7.4. Maintainability _________________________________________________________________ 17
7.5. Scalability _____________________________________________________________________ 18
7.6. Backup & Recovery _____________________________________________________________ 18
7.7. Client Technology ______________________________________________________________ 18
7.8. Secure Architecture _____________________________________________________________ 18
7.9. Maintenance & Support __________________________________________________________ 18
8. Standards & Compliance ___________________________________________________________ 18
9. Contractor/Vendor Profile _________________________________________________________ 18
4
1. Background
1.1. COMMONWEALTH SECRETARIAT The Commonwealth Secretariat provides guidance on policy making, technical assistance and
advisory services to Commonwealth member countries. We support governments to help achieve
sustainable, inclusive and equitable development.
Our work promotes democracy, rule of law, human rights, good governance and social and economic
development. We are a voice for small states and a champion for youth empowerment.
Priority areas of work are agreed at Commonwealth Heads of Government Meetings, which occur
every two years. The next summit is in the United Kingdom in 2018.
Our vision is to help create and sustain a Commonwealth that is mutually respectful, resilient,
peaceful and prosperous and that cherishes quality, diversity and shared values.
1.2. IT SERVICES IT Services (ITS) works to support the Commonwealth Secretariat to achieve its strategic goals by
providing and maintaining high quality and reliable IT services and their enabling infrastructure. This
allows staff to securely access and manage their information and control its dissemination to other
interested parties and stakeholders.
The IT Services Section is currently implementing a Transformation program to:
Enable the delivery of IT Solutions as a Service allowing flexibility in the rapid
deployment, usage and enhancement of systems
Enhance technology solutions to provide the organisation with the tools for embarking on
industry leading initiatives
Deliver desired business agility and enhanced service levels through new IT service models
Deploy an environment supported by enterprise tools for the provision of organisation-
wide knowledge sharing and collaboration
Enable technology and solutions to contribute to business transformation
The Big Open Data initiative forms a part of this Transformation program.
5
2. Overview
The Commonwealth Secretariat has embarked upon an Enterprise wide transformation program
which seeks to leverage the power of current state of the art technologies to realise the goals
outlined in The Commonwealth Strategic Plan 2013/14 – 2016/17. A major enabler to realising these
goals is the establishment of an open data platform which would greatly simplify the collation,
analysis and publication of structured data that is produced across our organisational departments.
This would increase insight into the work done within the Secretariat and bring significant value to
the corpus of work produced by the Secretariat.
Ultimately such a platform should enable the Secretariat to:
Build trust and engage with its stakeholders globally
Become a Knowledge Centre and resource for all its stakeholders globally
Reduce the cost and inherent inefficiencies of multiple technical environments for similar
data-centric projects
Provide global access to data sets, combined and collected by various projects and provide
the tools and interface to allow individuals and institutions to access and use those data
sets to build meaningful reports and gain insight from the data.
This platform would enable the Secretariat to produce a number of engaging infographics,
visualisations, interactive maps, data stories and non-linear insights which should increase the
appeal of the Secretariat’s work to general public and, equally importantly, justify the value of
our work to our stakeholders. Further to this, it would enable the Secretariat to organise and
manage the vast corpus of data that has been generated by our internal Divisions over the past
five decades. This offers a tremendous opportunity and challenge to:
1) Leverage our work to not only add value internally but to also contribute to the work of
international organisations. Once achieved the Secretariat may possibly be par none in this
domain.
2) Become possibly the only instance of an international organisation of this stature creating an
open data platform that spans its entire business domain. This opens up a number of potential
for member states and bodies that each have their own unique data analysis needs.
To date, a significant number of Commonwealth data assets are stored and managed within
disparate and isolated systems. The general approach to building these assets involves the
employment of external resources which are assigned to a specific departmental context and
which usually has little or no knowledge of the wider information ecosystem. Naturally this has
proven to be expensive, inefficient and confusing. Furthermore, any changes to the data
underpinning these assets necessitates the re-engagement of these external resources to
perform the asset modifications needed in order to accommodate them. The Secretariat,
6
therefore does not have ownership of its data nor the management process. This approach is
unsustainable and impractical with ever increasing levels of data collated.
The Big Open Data platform is the Secretariat’s solution to this problem. By focusing on
centralising data and leveraging data management tools, Divisional partners and project
managers alike would be able to create, update and publish material within a controlled
environment with the assurance that it is secure. Partners and unrelated actors on the platform
would be able to consume this data and recast it from their view point – gaining previously
unforeseen insight into pre-existing data. This single managed platform would serve as the base
upon which the Secretariat can generate meaningful information and value to a specific project
domain.
7
3. Key Business Requirements
Our goal is to build a robust, accessible, scalable, Open Data platform for use by projects throughout
the Commonwealth Secretariat.
This platform will provide a centralised data resource for all data driven projects. A successful
implementation of this project would:
allow ‘open’ – data that has been approved for publishing to be accessed by anyone for use and
re-use
benefit the Commonwealth as an Organisation through:
o Increased transparency. Better availability and accessibility of data about performance of
the Secretariat, e.g., budgetary data or public contracts data, specialised project data
sets, health, governance, education, etc.
o Improved public relations and attitudes towards the Secretariat and to better inform
Commonwealth Stakeholders about its actions. This can help to build trust, understanding
and general attitude of all associated citizens and organisations.
o Increased reputation of the civil society initiatives and publications of the Commonwealth
allowing greater openness and transparency of our institution.
o Better understanding and management of data within the Secretariat.
o The cataloguing, collection and analysis and publication of maintained data sets to
promote better understanding of data assets for related and funded projects.
o Support re-use from which new value can be drawn
o Produce outputs and results - an open data platform will promote re-use of knowledge
assets, Increasing value of the data
This technology will be used by progressive applications, fostering innovation in this sector.
Therefore, it is necessary that we agree upon, and document, the existing requirements on
this platform and this is the purpose of this document.
4. Methodology
The aim is to develop a platform for the management of information flow with respect to the
authenticity of the content and the participating stakeholders. The goal is to achieve a new level
of transparency, information dissemination and collaboration in the Secretariat.
This will be achieved by taking a Division-centric approach to data management and ownership.
Data uploaded by a specific Division will be owned and managed by that Division – this data will
only be shared and exposed as per the limits set by said Division. The extent of these parameters
8
will be encoded into the design of the platform. This guarantees the integrity of data as well as
enshrining the principles of data governance and custodial responsibility.
Whilst there is no prescription to a specific methodology we have nonetheless distinguished
between five (5) classes data Fig 1.
They are:
1) Source Data (Data Source)
2) Raw data
3) Data asset
4) Information set
5) Data Consumer (Data Sink)
Raw data is fed into the Open Data Platform from a data source. The raw data is stored
in a predefined structure within the platform, yielding a data asset, which is maintained
by the platform.
Specific interpretations of a data asset can be represented as information sets. An
information set is provided to a data sink using appropriate formats – e.g. presenting a
tabular dataset to a map based platform to depict regional hotspots. Data sinks use the
information over the natural course of their particular business processes – it is important
to note that they might also contribute the information as data to the platform by way of
a feedback loop.
9
During the overall data processing steps, there are a number of interpretation activities
that introduce subjective input into the processes (e.g., deciding on the structuring of
the data assets, the choice of linked data that yields the information sets, the assignment
of semantics to the data assets). It is therefore necessary to provision additional analysis
processes that uses the interpreted information sets to build up knowledge to assess the
Figure 1 Data Classification and Information Flow Concept
10
authenticity of a data source. Necessary to this approach would be interoperability vis-à-
vis web based APIs and secure communication channels. For example, it would be
necessary to closely manage meta-data to leverage data-analytics. Referring to one of
the key aims of the project, which is to provide non-linear insight into basic data, we
would need to provide value chain/consumption flow mapping of data. This would enable
end users to not only identify the source of the data but to also invoke data at any point
in the value chain to gain greater insight as per their requirements.
Such functionality requires end-to-end data tracking, high scalability and a robust API
framework. Equally important is to be able to guarantee the integrity of data held on the
platform.
Although trust management is considered a part of security management, there are notable
differences when compared to conventional security approaches. Conventional security
systems usually employ mechanisms that aim at preventing users from executing certain
operations or accessing certain information. Trust-based mechanisms do not prohibit users
from performing access or execution functions, but continuously assess the behaviour of
both parties, as well as interpretation processes, to decide on the authenticity of each.
This information can then be used in conjunction with conventional security mechanisms
(e.g., refined access controls via the use of policies or the use of encryption for message
confidentiality) to achieve higher levels of system security.
A trust-based mechanisms will be especially useful in this situation where the user group
may be too large or too diverse to be sufficiently controlled using a closed mechanism.
The platform should follow a Service-Oriented Architecture (SOA). Platform components
(data classes) may belong to separate Divisions, may be globally distributed (member
states), may fail at any time, or may need to be replaced. To support this architectural
style, the platform components should be loosely coupled. For example, this may require
components to rely on messages in order to communicate (or exchange) information. We
envision the platform will be based on web technologies and as such, services will be
accessed using HTTP based mechanisms (such as RESTful interfaces, Web Service
endpoints, or SOAP messaging).
11
Secretariat Data Management
We identified nine processes employed for data management (DM), as depicted in Figure 2.
Control
The control process ensures that the overall
operation of the DM system is working correctly.
It regulates the system behaviour in accordance
with the operational policies as set forth by a
system operator and the responsible legal
authorities.
Definition
The definition process encompasses the
identification of data sources and definition of
data types, structure and technologies used to
store the data assets. The outputs of this process
are clearly defined structures for data assets and
the data sources used for acquiring the raw data.
Figure 2 Data Management Concept
Figure 3 Data Management Process
Acquisition
During the acquisition process, raw data is entered into the structure created by the
definition process, resulting in data assets. Based on type and source of the raw data this
process uses conversion operations to transform the data from the raw format into a
suitable storage format. The outputs of the process are data assets, stored in pre-defined
data structures.
Organisation
The organisation process constructs information sets from the data assets by constructing
meaningful links between individual data values or sets. The outputs of this process are
information sets.
Provision
The provision process enables a distribution of the linked information created by the
organisation process. The output of this process is an information set, in a format suitable
12
for consumption by clients of the platform. The output (i.e., the information) may also
serve as input data to the acquisition process.
Archival
The archival process is used at the end of the lifetime of an information set to guarantee
a required retention period, where the information is not provided through the platform
anymore, but still available to a system operator. This process has no output.
Maintenance
The maintenance process encompasses all of the activities necessary to guarantee the
proper operation of the platform. This includes activities like backup, restore and data
validation.
Interpretation
The interpretation process is a vertical process that encompasses activities where
subjective input can influence five other DM processes, namely, definition, acquisition,
organisation, provision, and archival. This does not only refer to the processes themselves,
but also to the artifacts shared between processes. Typical interpretation activities are
visualisation, transformation, or computation using the prescribed artefacts. The output
of the interpretation process can be used to assess the system’s operation.
Analysis
The analysis process evaluates the operation of the system and assesses the
interpretation of the stored data assets and information sets. The results of the analysis
process can be used to influence the interpretation process and therefore, the platform’s
operation
13
5. Functionality
5.1. KEY FUNCTIONAL REQUIREMENTS
The Open Data Platform would be delivered as a highly scalable service-based application. The
following are key functional requirements of this system:
Scalable and flexible: The Secretariat seeks an Open Data Platform delivered as a service
which can scale to a large number of datasets and users without a significant overhead in
maintenance or performance. The preference is for it to be hosted in a cloud based platform,
however internally hosted solutions can be considered provided that maintainability and
performance are not negatively impacted.
Simple interface for managing data from the front-end: The platform should be simple and
intuitive to use and manage. One that allows users with low technical skills to analyse data
and interpret data without the need for ‘heavy’ additional tools. The platform should include
a rich range of out-of-the-box (not requiring new development, OOTB) dataset management
capabilities.
Simple interface for analysing and visualising data from the front-end: these OOTB data
manipulation and visualisation tools should allow users to manipulate data in interactive
charts and maps. Additionally, they should be versatile enough to allow savvy users to
combine otherwise unrelated datasets to create higher level data visualisations.
Easy Integration with 3rd Party Tools: the solution should be simple enough to enable non-
technical users to embed visualisations on other website and blogs. It should also provide
more experienced users with a richly defined API that provides the capability to combine
data, charts, maps, text and other 3rd party content. This API must conform to industry and
market standards.
Advanced management capabilities: the platform should provide a flexible and simple to
use workflow with different levels of authorisation and access to data and data manipulation
features.
Open standards and developer friendly environment: The Secretariat is keen to encourage
developers to use its published data in their own applications. In order to achieve this, the
Open Data Platform needs to be able to publish our data using open standards and allow
developers to query the data using the most popular development tools.
Data APIs on datasets: The Secretariat seeks to empower third parties (members, citizens) to
use the Secretariat data in ways that matter to them. Automatic APIs on every tabular
dataset OOTB, including automatic-generation of context-sensitive API documentation with
code snippets on each dataset API, are essential.
14
Advanced analytics: The Secretariat wants to understand how users are interacting with the
data presented, not limited to how many times it is viewed. We seek to leverage the power
of Open Data to gain insight into how our data is interpreted and modified and the ways in
which in contributes to a given topic or field of interest. To this end, the platform and
solution must provide capabilities to track usage, referrers, traffic, viewpoints and
contributors on all pages and sites where the data is referenced. This information should to
fed-back to the Secretariat in a meaningful way. One that will provide us with insight into
the value added by the provision of the original dataset.
Ambitious development roadmap: The Secretariat is aware that the Open Data field is fast
paced and technology is constantly changing. It is very important that the platform keeps
pace with new developments in this field and is capable of implementing changes quickly
without compromising on availability, performance or introducing significant additional costs.
Understanding the supplier's roadmap for adding new capabilities, and the process for
prioritising and delivering them will be an important criteria.
5.2. FUNCTIONAL COMPONENT REQUIREMENTS
The above mentioned functionality (5.1) are expected to encapsulate the following central tenets of
the system:
1. An open and published API
2. The capability to Publish, Discover, Contextualise, and Distribute signficant amounts of data
3. Specifically designed for the Open Data objectives
4. The system must be available 24/7
5. The system must support data formats for CSV, Excel, TSV, Shapefiles, KML, KMZ and XML files
6. The system must use Open Data architecture - Open Data Protocol
7. The system must support CRUD - Create, Read, Update and Delete
8. The system must support KML - Keyhole Mark-up Language, XML notation representing geographic
annotations used for presenting data on maps in browsers.
9. The system must support REST - Representational State Transfer and use the HTTP methods GET,
PUT, POST and DELETE for searching and reading data in XML, JSON or RDF.
10. The system must include an out-of-the-box interface with a variety of data sources, so as to
ensure seamless import of updated data from databases.
11. The system must have an API driven service to export data into XML and/or JSON formats.
12. The system must support the monitoring of global indicators and defined development
effectiveness indicators. The Open Data platform should ensure that the indicators for
monitoring can be embedded.
13. The system must have ability to create and publish data assets alongside web page assets
14. The system should be able to spatially represent data on an interactive map.
15. The system should be capable of interfacing with existing enterprise level systems and
applications e.g. Project Management Systems.
16. The system should provide functionality that allows internal and external partners the ability to
monitor and track progress of relevant data in a work flow like fashion. Such a functionality for
15
example, should allow for the consolidation of key statistical data in order to facilitate analysis
and empower decision-makers.
17. The system must have a user-friendly interface.
18. The System must be publicly accessible via the internet on all modern standard compliant
browsers as well as mobile devices such as tablets and smart phones.
19. The system should allow for the design, viewing and printing of reports in flexible and modifiable
formats this includes the ability to generate various types of reports based on the data stored and
meta-information.
20. The reporting feature should be intuitive and user-friendly. It should include but not be limited to
the following formats CSV, XLS, TXT, JSON, XML, XBRL, GeoJSON, MS Word, MS Excel and PDF.
21. The system should provide optimised keyword search on data .
22. The system should support a data publishing API that supports append, replace and upsert
(update and insert) operations.
23. The system should allow publishers the ability to create multiple derived ‘views’ on the same
dataset thus enabling them to derive insights into data that were otherwise not obvious. Access
to these views should be controllable in such a way that they can be pointed to a specific
audience if necessary.
24. The system should allow for multiple user experiences for both internal and external web users
on a single digital platform.
25. The data platform must allow for real time modification and update of data.
26. The data platform must require no coding skills or staff to implement, maintain, or operate.
27. The data platform must provide mobile-optimised API’s.
28. Ideally, the system should provide a turnkey, secure, cloud-based managed-service that scales on
demand with no legacy constraints.
29. The system should include an interface that allows for seamless export of activities.
30. The system must have a web interface, including the possibility of decentralised data entry by
remote users.
31. The system must provide SLA’s with 99.95% uptime.
32. The system should provide real-time usage analytics allowing data publishers and site
administrators to measure data consumption, distribution and user engagement.
33. The system should easily integrate with Google Web Analytics.
34. The system should track and report usage and performance of key indications and metrics across
the platform to measure success.
35. The system must have connectors for, SDKs, and libraries for Google Android, DataSync SDK,
Apple iOS, Java, PHP, Ruby, Scala, Swift, PhpSoda, .NET, Julia, and Python. Additionally, all
hardware and network requirements and costs associated with these interfaces must be clearly
defined and documented for the Secretariat.
36. In the likelihood of coding it must adhere to industry standard software development principles
and quality assurance methodologies. Furthermore, full documentation of source code and
internal functions of the system must be handed over to the Secretariat.
37. The system must be fully documented for end users. This documentation must be handed over to
the Secretariat at the end of the development cycle. The Contractor will be required to explicitly
authorise staff in the Secretariat and provide training/certification for them to manipulate the
system in whatever manner they see fit, in order to meet evolving requirements of the system.
16
38. There must be at least six (6) months of warranty period after production implementation and
sign off.
6. Services to which the cloud environment can facilitate access
6.1. INTERNAL AUTHENTICATION
A method of accessing, replicating or linking to the Secretariat’s internal Active Directory
must be part of the Open Data Platform.
6.2. OPEN DATA INTERFACE
Access via API to an Open Data platform allowing presentation, visualisation and analysis
of relevant datasets.
It must be possible to connect directly to external data sets and retrieve data.
6.3. ENTERPRISE SEARCH FUNCTIONALITY
Integrated, configurable search facility which can access both internal and external data
sources.
6.4. TAXONOMIES
Specialised taxonomies defining and correctly applying relevant terms to structured
searches.
17
7. Service Quality and Delivery
7.1. HOSTING AND AVAILABILITY
The system can be hosted either on a platform provided by the Secretariat or by the
service provider. In the case of being hosted on the Secretariat’s platform the system
should be optimised for performance with a Windows 2012 R2 environment running on
Azure.
In the case of being hosted by the provider, it is important that the implementation
comply with the following:
1) The hosting environment must provide an acceptable, industry standard level of
availability of at least 99.95% as a monthly average
2) The hosting environment must have no single point of failure in its own underlying
systems
3) The hosting environment connectivity should allow for a connection of any reasonable
bandwidth and latency that may be required to support application and management
needs in line with current Industry standards
4) All services should be recoverable within a reasonable period in line with current
Industry standards
7.2. RELIABILITY
The system must be stable both functionally and operationally.
The bandwidth and latency of any connectivity within the cloud and via the direct
perimeter connection to Internet services should be stable and as specified.
7.3. PERFORMANCE
The services provided by, and hosted, in the cloud environment should be should be of an
acceptable, Industry Standard, level of performance and fully meet any specification
provided. Any additional requirements or modifications to the current infrastructure must
be clearly specified indicating the likely costs. The system must not suffer from adverse
performance speeds or response times from concurrent access.
The system must cater to accessing via low bandwidth connections and DSL
communications without experiencing performance degradation. Any foreseen issues must
be highlighted in the proposed solution.
7.4. MAINTAINABILITY
Any essential maintenance work must be performed in predictably scheduled and agreed
maintenance windows wherever possible.
18
Any emergency maintenance required should be approved beforehand wherever possible,
except in cases of recovery from unexpected service loss.
7.5. SCALABILITY
All services must be highly scalable to support possible future expansion or contraction of
required workload and any supplier imposed constraints should be explained.
7.6. BACKUP & RECOVERY
The proposal must indicate measures taken to ensure a safe backup and recovery process.
7.7. CLIENT TECHNOLOGY
The devices used will include desktop, laptops, tablets and also smart mobile devices so
we require adaptive and responsive design and interface.
7.8. SECURE ARCHITECTURE
As a leading international affairs organisation, the Commonwealth Secretariat has a strong
reputation to protect. Security must adhere to Commonwealth Secretariat standards
which are in line with ISO 27001: Information security management systems, and ISO
27002: Information security – code of practice for information security practice.
7.9. MAINTENANCE & SUPPORT
The Supplier shall provide maintenance and support throughout the contract period.
8. Standards & Compliance
The cloud environment should be regularly assessed for standards compliance and should have all usual
industry standards accreditation including ISO27001.
9. Contractor/Vendor Profile
1. Prior experience in designing, developing and supporting implementation of web-based data
collection, data storage and reporting systems.
2. Experience of implementing such systems in one or all of the following organisations:
a. Developing countries
b. Multinational Governmental Organisations
c. Non-Governmental Organisations
3. Extensive development and implementation knowledge and experience in the following fields:
a. UX Design
b. Web-based Application Development
c. Data Analytics
19
4. Proven track record in the following:
a. International Relations
b. Development Cooperation/Economics
c. Aid Effectiveness
d. Aid Transparency
5. Proven experience in using CMMI or equivalent framework for all product and service
development.
6. The solution provider will provide a team of experts suitable and sufficient for the timely
implementation of the agreed solutions.
7. Provide evidence of multiple Open Data Platform implementations