Applied Semantic MediaWiki

SMWCon Spring 2011 Tutorial #42011.04.28

Who Are WeJesse Jiaxin Wang

Sr. Software Architect, Vulcan Inc.

http://semantic-mediawiki.org/wiki/User:Jesse

Ning HuSr. Software Developer, TeamMersion LLC

Wil SmithSoftware Engineer, Vulcan Inc.

3

About VulcanVulcan Inc. was established in 1986 by investor and philanthropist Paul G. Allen, co-founder of Microsoft, to manage his business and philanthropic efforts. Allen is chairman of Vulcan and his sister, Jody Allen, is president and CEO.

Agenda

Jumpstart Customize

Integrate

Jumpstart

Ways to quickly start a new wiki

Using a BundleSemantic Bundle

http://www.mediawiki.org/wiki/Semantic_Bundle

Features: (Pros and Cons)A set of author-selected extensions

A fixed set of extensions, few customization options

Simplified download and configuration processStill need manual configuration

Usually up-to-date and work well togetherCommunity tested and supported

Installation Package

SMW+ Package http://smwforum.ontoprise.com/smwforum/index.php/Help:SMW%2B

Features:A set of (fixed) company selected extensions

Simplifies installation and configurationWindows installer and VM images are great

And they just became free!

You may still be curious about what it does…

Professionally documented and testedUsually weeks after major release of latest SMW

Real-world Deployment

When you want a set of custom extensions, you want

Your OWN

deployment mechanism!

Deployment Practices

Your own codebase, with checkpointsSVN or Git to get the base, and then

Customize the wikiScript or Manual or Mixed

A third-party deployment framework and repository

Ontoprise: http://smwforum.ontoprise.com/smwforum/index.php/Help:Deployment_Framework

RPI: http://code.google.com/p/smwbp/wiki/setup_wiki

Referata: http://smw.referata.com/wiki/Category:Packages

Build SystemYou own it!

Your own selection of extensions

And versions of them

Own patches, scripts and templates

Vulcan: Internal GIT repositories of completed build

Versioning and branches between projects

Other examples:RPI Best Practice (Google code)

Hudson build (PNNL)

Wish ListA public wiki installation repository

with lots of packages

Including necessary data

Semantically marked up

Well documented (or even reviewed)

Easy to find and use

Federated wiki package store (Wiki AppStore)

Customize

Get your own functionality, look and feel

What to Customize

Skins

Extensions

Schema and Data

Templates and forms

Anything else you can code…

SkinsHow to customize a skin

Create your skin file in /skins folder within your wiki directory, like Ontoskin.php

Finish this skin files following Ontoskin.php skin filesThere are two classes in this skin files:

Inherit from SkinTemplate, set the CSS and template filter.

Inherit from QuickTemplate, set the UI style

Create skin folder with your skin name in /skinsCopy image and CSS files to your skin folder

Use your own skinIn LocalSettings.php set

$wgDefaultSkin = 'wiking_skin'

Seahawk Video Skin

Skin Customization

ExampleVulcan Development Wiki and Seahawk wiki.

We have customized these two skins for our projects and the key is the method of QuickTemplate. In this method, you can decide what to show and how it shows on your wiki

You can customize wiki site like menu links, page titles, CSS and JavaScript files in the data properties of QuickTemplate

For example, show wiki side bar in your skin<?php foreach ($this->data['sidebar'] as $bar => $cont) { ?> <li> <a href="#"><h5><?php $out = wfMsg( $bar );></h5></a> <ul> <?php foreach($cont as $key => $val) { ?> <li id="<?php echo $val['id'] ?>" > <a href="<?php echo $val['href'] ?>"><?php echo $val['text'] ?></a> </li> <?php } ?> </ul> </li><?php } ?>

Personalized SkinSkin supports customization via __USERNAME__

User can have personalized styles within a skin

http://www.mediawiki.org/wiki/Manual:Skin_configuration#User_CSS

Tip: A way to turn off Wikipedia banner ad:

http://en.wikipedia.org/wiki/User:Jesseone/vector.css

Schema and Data“Just do it!”

Easy to start

Flat, large ontology, at beginning

Many properties only valid in certain cases

But it’ll get better!

“Designer” OntologyProfessionally generated (Protégé)

More data (restriction) than necessary

Data importer isn’t powerful to fully support all OWL features

“The middle road” practical schema design: think, adapt, evolve

N-ary RelationsType:Record

From 1.5 + supports Semantic Search

Still no “Allowed values”, no unit, and no other visual customizations

Semantic Internal Object extension

Adding an object in the middle

Using parameterized (patterned) properties

Using a triple store (external)

ExtensionsWhere to find extensions

http://www.mediawiki.org/wiki/Extension_Matrix

By supported version (1.16, 1.17, …)

By status (beta, stable, …)

By type (API, AJAX, hook, user rights…)

By new-ness (sort by most recently created time)

By talk page new-ness…

By most updated versions…

Get the Extensions!

Some essential ones!http://semantic-mediawiki.org/wiki/Help:MediaWiki_extensions

Semantic MediaWiki extensions are quite a few:http://semantic-mediawiki.org/wiki/Help:SMW_extensions

Find what you want Not exactly? Then patch it!

Nothing close? Then build it!Or find people to build it

Our ExperiencesWe built several wikis (2008-2011)

The most useful extension (SF) often needed patches

Auto-completionFixed set of allowed values

All values in a category

All values having a specific property

All values from a query

More parameters support (default values, UID, etc.)

New functionality in other extensions or SMW CoreAggregation (average, max, sum, etc.)

e.g. > and < comparison

Security Extensions

Know thyself! – What you want, really.Often customers don’t know what they really want

Many choices = nothing is perfect

From basic to advanced:MediaWiki built-in security

Simple Security (Page Security)

HaloACL

Featu

res

Restrictio

ns

A Short StoryProposal Management System (Wiki)

Users (including privileged users) submit proposals

Reviewers (privileged users) rate and review them

Reviewers discuss and make decision and assign

What security extension should it employ?

HaloACL?

SimpleSecurity?

MediaWiki built-in groups?

How to Select | Test

Inclusion / transclusion

Special:Export

Aton/RSS feed, Watch page

Listing and SearchingSpecial:Search

RecentChanges, AllPages

Diff and Revision URL

Action links (raw, render)

API (???)Action=query?

3rd party APIs

Author backdoor?

Caching?

Files and Images?

Redirects?

Other extensions?

Visualization: Our Path

2008: SRF used to only have few options

2009: Exhibit seemed coolPorted webit

Fixed some bugs and enhanced features Runway, TimePlot, multiple rows in Map view etc.

2010: Open Flash ChartMore bars, pies, lines, scatter plot…

Interaction

Richer information display

Comparing SRF Options

Google Charts (pie and bar)Google does it! (G gets your data, no offline choice…)

Static bar and pie, no color options…

ExhibitMore choices (timeline, map, facet, runway…)

Browser compatibility issues

OFC (Open Flash Chart)Many bars, charts, plots, even scattered plots

Flash required

Sidebar Customization

Basic: MediaWiki:Sidebar

Enhanced: semantic queries, and tree views

Advanced: use variables together with queries

Custom: Using an extension: DynamicSidebar Customize your own sidebar at User:<username>/Sidebar

http://www.mediawiki.org/wiki/Extension:DynamicSidebar

Example: http://wiking.vulcan.com/dev/

Current Sprint in Sidebar

*Current sprint{{#ask: [[Category:Project sprints]][[Sprint start date::<{{CURRENTYEAR}}/{{CURRENTMONTH}}/{{CURRENTDAY}}]][[Sprint end date::>{{CURRENTYEAR}}/{{CURRENTMONTH}}/{{CURRENTDAY}}]]|format=template|template=Sidebar query template|link=none|}}

My Active Sprint Tasks in Sidebar

*My tasks{{#ask: [[Category:Project tasks]][[Project task owner::<q>[[User alias::{{CURRENTUSER}}]]</q>]][[Project story::<q>[[Project sprint::<q>[[Category:Project sprints]][[Sprint start date::<{{CURRENTYEAR}}/{{CURRENTMONTH}}/{{CURRENTDAY}}]][[Sprint end date::>{{CURRENTYEAR}}/{{CURRENTMONTH}}/{{CURRENTDAY}}]]</q>]]</q>]][[Project task status::In progress||Not started]]|format=template|template=Sidebar query template|link=none|}}

More Extension Examples

Twitter Feedhttp://www.mediawikiwidgets.org/Twitter_Search (example)

Flickr Feedhttp://www.mediawikiwidgets.org/Flickr

Amazon Carouselhttps://widgets.amazon.com/Amazon-Carousel-Widget/

Google Analyticshttp://www.mediawiki.org/wiki/Extension:Google_Analytics

Optify Analyticshttp://www.optify.net/

Templates and Forms

MediaWiki template is essential for its structured data handling

Editing, especially Template editing, is not trivial

Semantic Forms extension greatly lowers the bar

Customizing templates and forms can provide very visually appealing pages

Form with a Style

http://www.thethirdturn.com/w/index.php?title=Form:Driver&action=edit

Forms with autocompletion

Basic Auto-Completion on Category Values

Advanced Auto-Completion on Customized Query Results

Story TemplateTo have a field (form link) for users to quickly create tasks that belong to the story (parameter preset)

{{#forminput:Project Task|45||Create a new task|Project Task[story]={{PAGENAME}}&Project Task[owner]=__USERNAME__&Project Task[status]=Not started}}

http://wiking.vulcan.com/dev/index.php/Prepare_for_SMWCon_Spring_2011

Example: Daily Report

We want to achieve “burn-down” chart in our sprint overview

We need:Daily new tasks (created, but not started yet)

Daily in-progress tasks (started by not done yet)

Daily finished tasks (completed)

Task ownership and completion timeline

And similar things for bugs (other analysis)

Quiz #1:

How do you specify Today, Tomorrow in query?

Date Magic and Templates

Magic words: CURRENTYEAR/MONTH/DAY

Some MediaWiki date-related templates to use

DATEDIFF, DATECOMP, …

NEXTDAY, NEXTDAYDATE, …

http://en.wikipedia.org/wiki/Category:Date-computing_templates

http://en.wikipedia.org/wiki/Category:ISO_date_templates

Quiz #2:

How do you do strictly “less than” or “greater than”?

(Before 2010.03)

Strict ComparisonDate query : how to do “In progress” with a finish date “>” today?

Extra markup would reduce usability

Semantic MediaWiki did NOT support it until 2010.11

We patched Semantic MediaWiki Core

Now (as of SMW 1.5.3 + ) we have < and > implemented

Use ≤ and ≥ (on ASCII keyboard 242 and 243)

And a configuration to turn it on

Quiz #3:

What if “Today” changes to “Tomorrow” or later dates in

your query template?{{CURRENTYEAR}}/{{CURRENTMONTH}}/{{CURRENTDAY}}

Freeze the TimeSubclusion is your friend!

Subclusion = Substitute + inclusion

Today's newly started # tasks: [[Report start task count::{{subst:#ask: [[Category:Project tasks]][[Project task start date::{{{date|}}}]]| format=count|}}]]

More Customization

Out there in the Wild Wiki World

Integration

Guidelines and case studies on building wikis

StepsKnow what to build

Customers, customers, customers!

Get your collection of extensions…

Customize your wiki (template, forms)

Handle the data (schema, query, API, …)

Case 1. Proper Wiki

Focus on Skin, Form and Template

Steps:1. Pick a (default) skin

2. Design some forms1. And templates (queries) behind the forms

3. Build starter pages and Go!

Biology Ontology Wiki

People need discuss about the details related to construct a biology ontology

Classes (categories)

Properties

Relations

Textbook analysis

Discussion

Forms necessary to keep content tidy

Ex: Property Form

Forms on Namespace

http://www.mediawiki.org/wiki/Extension:Semantic_Forms#Based_on_namespace

“Easy-peazy, lemons squeezy” ?

Case 2: Simple Workflow App

Idea: go from one form to another form

Need form links in the template

May need user rights management

Proposal Review Workflow Example1. User submits a proposal via a form

1. Fill in basic information about the proposal

2. Set a field in the form to be, say, “New”

3. User doesn’t see further information about “Reviewer” or “Owner”, etc.

2. Semantic notification (email and RSS feed) facilitates communication

3. Reviewer finds the form in a canned query1. Reviewer changes the status to, say, “Approved”

2. Reviewer field (say “Reviewed by”) is (automatically) set

3. Then reviewer assigns the proposal to a owner, say, “Alice”

4. The owner (“Alice”) now sees it and can start work on it1. Changes the status to “Active”

2. Adds a start date or maybe an estimated end date too

Ways to Handle Workflow

Add a link to open another formForce edit “review” using another form (e.g. reviewer form)

When saving the new form, the category is changed

Now we have a different category a different default form

Change a value to include or exclude to another template

May need #if (or #switch) statement in template to change (or include/exclude) template values

Use “Page has default form” property (Semantic Forms extension)

Case 3: Wiking Dev Wiki

Our project management wiki

We use it for *all* things it can do:Proposals, ideas, email messages, features, bugs

Agile development: milestones, sprints, stories, tasks

Code/Feature association: SVN mapped to tasks/bugs

Progress analysis: burn-down charts, work calendar

Collaboration, Documentation, Demonstration

Examples of Data I/O

Wiking Development Wiki

An Imported emailhttp://wiking.vulcan.com/dev/index.php/Demo_scenarios

An email imported as a Project Bughttp://wiking.vulcan.com/dev/index.php/Issue_11886_Insert_a_single_property

A task uploaded via Outlook with multiple commits

http://wiking.vulcan.com/dev/index.php/WikiTags_Release_Structure

Microsoft Office Connector

Leverage Microsoft Office applications and technology

Bring SMW info to Office applications on-demand

API for data I/O: add and modify wiki data within Microsoft Office

Utilize semantics to improve relevance

Smart actions for semantic properties

Connections

API

API

SmartJesse Wang | SemTech 2010

Our Agile Project Management (Scrum)

57

Sprint Backlog

Sprint Output

Sprint2-3 week

BurnupBurndownVelocity

Daily ScrumEvery 24 hours

Bac

klog

Mee

ting

mem

o

Etc

...

Code commit

Spr

int o

utp

ut p

ack

ageCreate stories and tasks

Mail upload

Task /

bug

upda

te

Generate daily report

User specified notificiations

Product documentation

Wiking

Jesse Wang | SMWCon Amsterdam 2010

Wiki Data I/OWiking development wiki has two external applications

Subversion integration

Microsoft Office integration

Extensions to support itSemantic Wikitags

Semantic Connector

Data I/O Extensions

MediaWiki API – very basic read/write

Page Object ModelFirst Data I/O extension to allow access some wiki structure data (links, title, basic template field)

SMWWriterBased on POM, support annotations (semantic properties)

Data API extension in SMW+

Semantic WikiTags and Semantic Connector

Wiki Object ModelA powerful and comprehensive object model (Data API)

Read/write at finer object level (sentences, template parameters)

Wiki Object ModelPrevious wiki data API efforts are not good enough to build applications

There are strong needs for access the structured data and unstructured data in the wiki as a data store

From internal and external applications’ point of view

A DOM-like approach is reasonable choiceXpath is a great tool and standard

Granularity offers flexibility and power

People like to operate on known objects rather than parsing and handling wiki texts

APIs provide access to both internal and external apps

Wiki Object Model Objects

Category

Link

Property

Text

Magic word

HTML tag

Sentence

Word*

Image

61

Page

Section

Parameter

Parameter value

Template

Template field

Parser function

List item

Table

Table cell

* Not yet implemented as of April 2011

Demo of WOM Demo at http://wiking.vulcan.com/dev_sandbox/ Demo 1: APIs

The API is part of MediaWiki API, with our additions to allow get and set (read/write) the content of a wiki page

http://wiking.vulcan.com/dev/index.php/Extension:Wiki_Object_Model/Apis

Demo 2: Inline Editor A simple inline editor (AJAX styled) to let user modify

a specific portion of the wiki page Based on

http://wiking.vulcan.com/dev/index.php/Extension:Wiki_Object_Model/Functions

62

4. Final Case Study

Stanford University CS 227 Car WikiDeveloped for Masters level CS class for assignment in “Knowledge Representation Within a Social Context”

Over 40 students created pages populating an ontology, created a personal page documenting their automobile, and compiled site statistics into group pages

Extra credit was offered to the entire class as a group upon initial wiki population assignment

“Dream Car” Wiki Ontology

The ontology was left intentionally incomplete and students were encouraged to update the ontology

Car manufacturer pages

Car make & model pages

Personal “Licensed Car” Page

Custom properties added to ontology and personal pages

Extensions for Car Wiki

Standard SMW extension suite downloaded from SMW+ code repository

Custom Semantic Results Formatter used with Open Flash Charts

Widgets extension included for creating custom JavaScript APIs to other social frameworks

Forms and Templates

Semantic Forms for creating company, car, and personal vehicle

Templates for presenting your car and linking the make & model to Widget templates for social links

Demo of Car WikiOntology

Sample Car Manufacturer : Audi

Sample Car Make & Model : Audi A6/A7

Sample Licensed Cars : William_A6, William_A7

Final Chart of Completed Assignment

A Wiki in a WeekOntology, Code Installation and Sample Page Population Completed within a Day

SMW+ with core extensions

Minimal Updating and Bug Fixing of Components Taken from Other Wiki Projects

Custom widgets and result formatters

Final Population of Wiki Articles Completed Over the Course of a Week Long Assignment

SummaryJumpstart

Deployment of your bundle/suite/package

Code, pseudo-code/meta-data, and data

CustomizeSkins, Styles

Extension choices

Integrate Know the requirements and data

Put everything together and evolve

Thank youQuestions and/or comments?