International School of Law and Business Vilnius, Lithuania.
FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS Software tools what we built and building...
-
Upload
jeremy-white -
Category
Documents
-
view
213 -
download
0
Transcript of FOOD AND AGRICULTURE ORGANIZATION OF THE UNITED NATIONS Software tools what we built and building...
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
Software toolswhat we built
andbuilding community.
Vilnius, LithuaniaPC-Axis Conference, September 2009
Yves Jaques ESSS, FAO of the UN
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
2 September 2009PC-Axis Conference, Vilnius Lithuania
Outline
• what we built– PX-Web customisation– PX-Publisher– CountrySTAT installer– CountrySTAT virtual image– SDMX
• building community– Best practices– Tools– Administration
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
3 September 2009PC-Axis Conference, Vilnius Lithuania
Outline
• what we built– PX-Web customisation– PX-Publisher– CountrySTAT installer– CountrySTAT virtual image– SDMX
• building community– Best practices– Tools– Administration
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
4 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Web customization - intro
• Over the last few years PX-Web code has been modified to fit CountrySTAT needs.
• The core query engine and exporting / graphing / mapping components have remained as is while,
• Many other features are not used.
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
5 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Web customization - Rationale• CountrySTAT represents a unique use-case of PX-
Web– Almost twenty countries running under one server in a
shared environment– A flexible, configurable CMS was needed.
Solution• Build a site around PX-Web query engine,
– borrow what was useful from PX-Web,– make use of other available JavaScript and ASP libraries,– write bespoke code.
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
6 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Web customization - Differences• Houses as many sites as desired in one instance of
PX-Web• An easy to use admin area• Rich text editing of pages• Menu tree a separate process not using PX-Web
with some new visual features• A flexible RSS feature (whole site, or any folder)• Custom icons in the query area• Support for confidentiality (public, internal private)• Support for a validity period
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
7 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Web customization - Issues
• Perhaps not compatible with newer versions of PX-Web?
• Not a community project, so who will maintain over the long-term?
• How to handle and prioritize new feature requests?
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
8 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Web customization - Conclusions
• Should investigate if an off-the-shelf CMS could be used to house PX-Web.– Think of PX-Web more as a plug-in, e.g. within Umbraco.– Benefit from a huge community of users.– Other countries seem to be doing so (e.g. Switzerland)
• Find it at: http://www.countrystat.org
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
9 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Publisher
• What is it?– A web-based .px file uploader.– A data tree manager.– A .px file publisher.
• Implementation?– VB.NET to conform to the upcoming PX-Web.NET.
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
10 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Publisher - Rationale
• Again, CountrySTAT represents a unique use-case of PX-Web– Almost twenty countries running under one server in a
shared environment– Within countries, data uploaded by numerous sibling
institutions
• PX-Web not designed to upload PX files– FTP– Logging– Roles– Versioning & Synchronization
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
11 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Publisher - Solution
• A new component: PX-Publisher– define users and their roles– manage data tree– role-based upload and management of .px files – role-based publication of .px files and folders– definition of data validity date range– define minimum required metadata to permit file
upload– multi-lingual– maintain previous versions of PX files
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
PX Publisher – New keywords
• VALIDFROM– taking the format DD/MM/YYYY, this field references the date from which the data is
valid.– Used by CountrySTAT CMS to disable table listing of a PX until the data passes the
criteria.
• VALIDTO– taking the format DD/MM/YYYY, this field references the date until which the data is
valid.– Used by CountrySTAT CMS to disable table listing of a PX until the data passes the
criteria.
• OWNER– A field containing the ID of the person responsible for the PX file.– Used by PX-Publisher to assign user rights to PX files.
12 September 2009PC-Axis Conference, Vilnius Lithuania
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
13 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Publisher - Status
• Version 1.0 deployed with basic needed functionality:– user management– file upload– folder association– Publishing
• http://devel.si-vs506.com/upload
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
14 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Publisher - Next steps• Training
– CountrySTAT users at FAO advanced training next week
• Next iteration(s)– data tree manipulation
– enforce user roles on data tree
– minimum metadata requirements
– group actions zip archives publish whole branches
– restore files
– metadata editing
– PX-Edit integration?
– Reconfigure with PAXIOM class?
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
15 September 2009PC-Axis Conference, Vilnius Lithuania
PX-Publisher - Want to try?
• You can always download the latest build here:– http://si-vs506.com:81/svn/countryNet/trunk/uploader/
(login: pc-axis, pw: reader)– Ask us for a JIRA account if you want to report issues
• Interested in developing or extending?– Just ask and we can set you up in the source control
repository on a branch for later integration.
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
16 September 2009PC-Axis Conference, Vilnius Lithuania
CountrySTAT Installer
• Simplify local installations for training purposes.– Written using open-source INNO
• Contains:– PX-Edit 2.4– PC-Axis 2008– PX-Web 2007 w/ CountrySTAT data– PX-Map 1– FileZilla FTP
• Checks for:– OS type– presence/absence IIS (performs silent install)– problematic Anti-virus products (Norton, MacAfee)– Skype (often blocks port 80)– previous data and backs up if necessary
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
17 September 2009PC-Axis Conference, Vilnius Lithuania
CountrySTAT Virtual Image• What is it?
– A stock Windows XP/IIS5.1/IE6 with– PC-Axis– PX-Edit– Excel– CountrySTAT website with PX-Web2007, PX-Graph and PX-Map1.
• How does it work?– User installs Virtual Box, a free virtual machine from SUN.– User imports the CountrySTAT snapshot and starts it.
• Advantages– Runs on Windows XP/Vista, Mac OSX, Linux and OpenSolaris– Resolves many installation problems by providing a common running
environment for training sessions.
• Disadvantages– Can be confusing to users since they are running two operating systems.
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
18 September 2009PC-Axis Conference, Vilnius Lithuania
SDMX - CountrySTAT vision
• Promote it as a way to harmonize national data at regional and international level.
• Offer a standard with which to disseminate national/regional data
• Disseminate FAO classification systems metadata in a standard format.
• Streamline data input to FAO statistical working system
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
19 September 2009PC-Axis Conference, Vilnius Lithuania
SDMX - Timeline (June-November 2009)
• Create Data Structure (DSD) and Metadata Structure (MSD) Definitions for several FAOSTAT domains– e.g.. production, commodities.
• Install a metadata registry on CountrySTAT server and train FAO personnel
• Define a mapping between PX and SDMX
• Develop prototype deployable within CountrySTAT capable of– Using the registry, perform round-trip PX–SDMX conversion for a
limited set of structures
– Aggregate from national to regional level.
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
20 September 2009PC-Axis Conference, Vilnius Lithuania
SDMX - Issues
• Not possible to automatically map between PX and SDMX.
• Requires a lot of groundwork and training to really benefit.
• Very sophisticated, how to hide complexity from most users?
• Not clear how to simplify the interface between PX and SDMX– depends a lot on the data structure and
– we have MANY different data structures coming from our national partners.
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
21 September 2009PC-Axis Conference, Vilnius Lithuania
SDMX - sharing
• Registry is open for you to use– just ask and we can create you as an organization with
user rights for your schema.
what we built
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
22 September 2009PC-Axis Conference, Vilnius Lithuania
Outline
• what we built– PX-Web customisation– PX-Publisher– CountrySTAT installer– CountrySTAT virtual image– SDMX
• building community– Best practices– Tools– Administration
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
23 September 2009PC-Axis Conference, Vilnius Lithuania
Outline• what we built
– PX-Web customisation– PX-Publisher– CountrySTAT installer– CountrySTAT virtual image– SDMX
• building community– Best practices– Tools– Administration
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
24 September 2009PC-Axis Conference, Vilnius Lithuania
Building community - Successes• PC-Axis is a vibrant community of some 50 countries
• Dissemination website contains much useful information
• Forum is helpful in resolving issues
• Member countries provide valuable plug-ins
• Components function quite well
but,
Could it be time to take things one step further?
building community
Highlights
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
25 September 2009PC-Axis Conference, Vilnius Lithuania
Building community - Status
• At a crossroads?– The community’s success has caused it to grow
beyond anyone’s best guess.– It will become increasingly difficult to maintain
coherency as time passes.
• The community’s present size suggests a more formal approach to dealing with development.
building community
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
26 September 2009PC-Axis Conference, Vilnius Lithuania
Building community – Components
building community
Process
Methods
Tools
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
27 September 2009PC-Axis Conference, Vilnius Lithuania
Building community – Best practices• Clarify development process(es)
– Waterfall, RAD, Agile, etc.?– Essential for collaboration
• Use accepted development methods– How are requirements developed and negotiated?– Make spec and design docs available– Publish timelines and indications of state of progress– Involve the community in V&V– Create standardized API documentation– Create standardized end-user documentation– Release source code to community
building community
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
28 September 2009PC-Axis Conference, Vilnius Lithuania
Building community - Tools
• Make use of development process tools– Add bug tracking to allow for systematic
documentation of problems and their resolution
– Add issue tracking to help organize what we built
– Create a community code repository
– Publish a WIKI for community to organize documentation
building community
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
29 September 2009PC-Axis Conference, Vilnius Lithuania
Building community - AdministrationTreat PC-Axis as a “closed” open-source community
• Create a simple charter with a clear set of roles & responsibilities– Community admin– Repository admin– WIKI admin
• Agree on a minimum level of software artifacts– Requirements specification– API documentation– etc.
• Alignment– existing work added to the community as possible– what we built conform to the charter
building community
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
30 September 2009PC-Axis Conference, Vilnius Lithuania
Building community – Next steps (a proposal)
• Adopt a cautious, iterative approach– As they teach all medical students,
Primum non nocere!
• Start with the two most essential tools– A documentation WIKI divided by component– A code repository
• Add existing artifacts– Fill in WIKI with requirements, design docs, technical docs, etc.– Baseline code in the shared repository
• Assess the situation and decide how best to continue
building community
FOOD AND AGRICULTURE ORGANIZATIONOF THE UNITED NATIONS
31 September 2009PC-Axis Conference, Vilnius Lithuania
Conclusion• PC-Axis continues to meet CountrySTAT requirements
– Easy installation– Powerful conversion tools– Simple human-readable file format– Excellent query interface– Flexible display– Great plugins
• CountrySTAT promotes the use of PC-Axis.– presentation of the tools to interested countries– development and delivery of training materials,– definition and tracking of unmet requirements and
• We plan to keep developing new and existing tools– CountrySTAT CMS,– PX-Publisher,– PC-Axis package installer and– training materials.
• We will continue embracing a closed open-source model– standard processes, methods and tools,– community access,– open code,
Thank you for your time.