Laying Pipe with Transmogrifier

25
Clayton Parker, Senior Web Developer Laying Pipe with Transmogrifier PLONE CONFERENCE 2010 Thursday, October 28, 2010

description

Transmogrifier is a tool that gives you the ability to create a configurable pipeline to import and export from various systems. Each section of the pipe can be turned into a re-usable blueprint that can be given back to the community. This gives us the ability to quickly create migration paths for existing systems and maintain them as Python packages.

Transcript of Laying Pipe with Transmogrifier

Page 1: Laying Pipe with Transmogrifier

Clayton Parker, Senior Web Developer

Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010

Thursday, October 28, 2010

Page 2: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Who Am I?

• claytron

• Python dev since 2003

• Plone Core Committer

• Foundation Member

Thursday, October 28, 2010

Page 3: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010What Will We Learn?• What is Transmogrifier

• How to use it in Plone

• How to package a migration

Thursday, October 28, 2010

Page 4: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Migrations

• One off scripts

• In multiple places

• No re-use

Thursday, October 28, 2010

Page 5: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Transmogrifier

• A framework for migrations

• Re-usable parts

Thursday, October 28, 2010

Page 6: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Basics

• Pipeline

• Blueprints

• Sources

Thursday, October 28, 2010

Page 7: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Pipeline[transmogrifier]pipeline = blog title text text-mimetype date type transitions urlnormalizer path constructor schemaupdate comment-insert comments workflow reindexobject

[blog]blueprint = transmogrify.bloggerfilename = acost.policy.import:export.xml

[title]blueprint = collective.transmogrifier.sections.manipulatorkeys = _transmogrify.blogger.titledestination = string:titledelete = ${title:keys}

[text]blueprint = collective.transmogrifier.sections.manipulatorkeys = _transmogrify.blogger.contentdestination = string:textdelete = ${text:keys}

[text-mimetype]blueprint = collective.transmogrifier.sections.inserterkey = string:text.mimetypevalue = string:text/html

[date]blueprint = collective.transmogrifier.sections.manipulatorkeys = _transmogrify.blogger.published.rfc822destination = string:effectiveDatedelete = ${date:keys}

[type]blueprint = collective.transmogrifier.sections.inserterkey = string:_typevalue = string:BlogEntry

[transitions]blueprint = collective.transmogrifier.sections.inserterkey = string:_transitionsvalue = string:publishcondtion = python:item["_transmogrify.blogger.state"] == "published"

[urlnormalizer]blueprint = plone.app.transmogrifier.urlnormalizersource-key = titledestination-key = string:_id

[path]blueprint = collective.transmogrifier.sections.inserterkey = string:_pathvalue = string:${settings:blog-root}/${item/_id}

[constructor]blueprint = collective.transmogrifier.sections.constructor

[schemaupdate]blueprint = plone.app.transmogrifier.atschemaupdater

[comment-insert]blueprint = collective.transmogrifier.sections.manipulatorkeys = _transmogrify.blogger.commentsdestination = string:_commentsdelete = ${comment-insert:keys}

[comments]blueprint = transmogrify.commentscomment-type = plone.app.discussion

[workflow]blueprint = plone.app.transmogrifier.workflowupdater

[reindexobject]blueprint = plone.app.transmogrifier.reindexobject

[debug]blueprint = collective.transmogrifier.sections.inserterkey = string:debugvalue = python:modules['pdb'].set_trace()

Thursday, October 28, 2010

Page 8: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Blueprints

• Python code

• Packages

Thursday, October 28, 2010

Page 9: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Sources

• A blueprint

• First item in your pipeline

Thursday, October 28, 2010

Page 10: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010configure.zcml

<configure xmlns="http://namespaces.zope.org/zope">

<include package="collective.transmogrifier" />

<utility component=".blueprint.MySection" name="my.section" />

</configure>

Thursday, October 28, 2010

Page 11: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010blueprint.pyfrom zope.interface import implementsfrom zope.interface import classProvidesfrom collective.transmogrifier.interfaces import ISectionfrom collective.transmogrifier.interfaces import ISectionBlueprint

class MySection(object): """A blueprint for importing something into plone """ classProvides(ISectionBlueprint) implements(ISection)

def __iter___(self): # add any other sources into the stream for item in self.previous: # Do something with each item # Pass it on to the next blueprint yield item

Thursday, October 28, 2010

Page 12: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Items

• Each item is a mapping

• Keys are fields

• Keys with a leading underscore are controllers

Thursday, October 28, 2010

Page 13: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Pipeline Settings

• Use annotations

• Globally imported settings

Thursday, October 28, 2010

Page 14: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010GenericSetup

• Make migration part of your release

• Ability to package migrations

Thursday, October 28, 2010

Page 15: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Package Layout

my.migration!"" __init__.py!"" configure.zcml!"" import#   !"" __init__.py#   !"" configure.zcml#   !"" my_items.csv#   %"" my_migration.cfg%"" profiles %"" default %"" transmogrifier.txt

Thursday, October 28, 2010

Page 16: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010import/configure.zcml

<configure xmlns="http://namespaces.zope.org/zope" xmlns:transmogrifier="http://namespaces.plone.org/transmogrifier">

<include package="collective.transmogrifier"/> <include package="collective.transmogrifier" file="meta.zcml"/>

<transmogrifier:registerConfig name="my_migration" title="Import items based on CSV file" description="" configuration="my_migration.cfg" />

</configure>

Thursday, October 28, 2010

Page 17: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010transmogrifier.txt

my_migration

Thursday, October 28, 2010

Page 18: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Example Pipeline[transmogrifier]pipeline = csv_file constructor schemaupdater

[csv_file]blueprint = collective.transmogrifier.sections.csvsourcefilename = my.migration.import:my_items.csv

[constructor]blueprint = collective.transmogrifier.sections.constructor

[schemaupdater]blueprint = plone.app.transmogrifier.atschemaupdater

Thursday, October 28, 2010

Page 19: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010my_items.csv

_path,_type,title,description/folder1,Folder,First Folder,This is folder One/folder2,Folder,Second Folder,This is folder Two/folder1/foo,Document,One Foo,A document named foo/folder2/foo,Document,Two Foo,Another doc named foo

Thursday, October 28, 2010

Page 20: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010The Result

Thursday, October 28, 2010

Page 21: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Stock Blueprints• Constructor section

• Folders section

• Codec section

• Inserter section

• Condition section

• Manipulator section

• Splitter section

• Savepoint section

• CSV source sectionThursday, October 28, 2010

Page 22: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Useful Sources and Blueprints• plone.app.transmogrifier

• transmogrify.filesystem

• transmogrify.sqlalchemy

• transmogrify.webcrawler

• wordpress / zine / blogger

Thursday, October 28, 2010

Page 23: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010plone.app.transmogrifier

• ATSchema updater section

• UID updater section

• Workflow updater section

• Browser default section

• Criterion adder section

• Portal Transforms section

• URL Normalizer section

• Mime encapsulator section

• Indexing sectionThursday, October 28, 2010

Page 24: Laying Pipe with Transmogrifier

PLONE CONFERENCE 2010Links

• collective.transmogrifier

http://pypi.python.org/pypi/collective.transmogrifier/

• plone.app.transmogrifier

http://pypi.python.org/pypi/plone.app.transmogrifier/

Thursday, October 28, 2010

Page 25: Laying Pipe with Transmogrifier

Check out

sixfeetup.com/demos

Thursday, October 28, 2010