Changing Responsibilities And Roles for Professional Biologists for Professional Biologists.
MOLGENIS For rapid prototyping of web applications biologists want to have
-
Upload
iliana-neal -
Category
Documents
-
view
23 -
download
0
description
Transcript of MOLGENIS For rapid prototyping of web applications biologists want to have
2009
2009
|
MOLGENIS.org
For rapid prototyping of web applications biologists want to have
Morris Swertz Joeri van der Velde
Joris Lops
NBIC/BioAssist programmers meetingUtrecht, 18 December 2009
1
EBIBiobanking platform
Outline
› Introduction Motivation Conceptual strategy MOLGENIS implementation Expected results Underlying architecture
› Practical Install a MOLGENIS virtual machine Work through examples in manual See what it can do for you
2
Biological challenges4
inbreedinbreed
100
10.000
1,000,000
100,000
10,000
10
10,000,00
QTL profiles
QTL profiles
networknetwork
correlatecorrelate
genomegenomestrainsstrains
individualsindividuals
markersmarkers
expressionsexpressions preprocesspreprocess
probesprobesmicroarraysmicroarrays
100
hybridizehybridize
100,000
genotypegenotype genotypesgenotypes
norm exprs.norm exprs.
10.000
mapmap
Main work flow
Data dependency
Biomaterial/result
Lab/analysis process
Scale of information
Associated data files
processprocess
materialmaterial
10,000
! Many materials! HTP processes! Complex, distr. workflows! Large data! Trace dependencies! Adapt to new protocols
|[email protected] | [email protected] / 5
For example
Dozens of samplesComplex relationships
Large datasetsProcessing
|[email protected] | [email protected] / 6
For example
Swertz et al (2004) Bioinformatics 20, 2075-83
|[email protected] | [email protected] / 7
For example
Swertz et al (2004) Bioinformatics 20, 2075-83
|[email protected] | [email protected] / 8
For example
Swertz et al (2004) Bioinformatics 20, 2075-83
Informatic challenges9
Input markers,traits, genotypes
AMap QTLs(qtlMapTwoPart)
QTL profiles{qtlProfiles}
QTL threshold{qtlThres}
BSimulation/FDR(qtlThreshold/qtlFDR)
QTL summary{qtlSumm}
Input mass/chargepeaks
CQTL summary(qtlSummary) Significant QTLs
DZero-ordercorrelation
Peak multiplicity{peakMultiplicity}
HPeak multiplicity(findPeakultiplicity)
Correlation matrix{corrZeroOrder}
FPermutation(qtlCorrThreshold)
E2nd-order correlation(qtlCorrSecondOrder )
Correlation matrix{corrSecondOrder}
Correlation threshold{corrThres}
GCreate network(createCytoFiles)
Network files[network.sif,
network.eda]
researchers
Communication + integrationinfrastructure
datainfrastructure
processinginfrastructure
user interactioninfrastructure
bioinformatician
Swertz & Jansen (2007) Nature Reviews Genetics
… try hitting a moving target10
biologistbiologist
bioinformatician softw engineers
arab 220903
100 200 300 400 500 600 700 800 900 1000m /z0
100
%
Koornneef0007 526 (11.117) AM (Top,4, Ar,10000.0,556.28,0.70,LS 10); Sm (Mn, 2x1.00); Sb (1,40.00 )1.40e3171.1702
1396
649.3804551
526.3066248172.1795
162
650.3882224
809.4496;80
inbreedinbreed
100
100.000
10,000,000
1000
10,000
10
1000genotypegenotypeindividualsindividuals
mass peaks
mass peaks
genotypesgenotypes QTL profiles
QTL profiles
strainsstrains
networknetwork
SNP arraysSNP arrays
correlatecorrelate
LC/MSLC/MS
genomegenome
mapmap
preprocesspreprocess aligned peaks
aligned peaks
€
biological challenges suitable infrastructure
… repeatedly
http://www.molgenis.orgSwertz & Jansen (2007) Nature Revies Genetics 8, 235-243
NextGenSeq
MassSpec/Metab.
Experiments
WorkBiologist challenges Software solutions
Metabolomics support platform
11
Standardization + Variation 13
http://www.molgenis.orgSwertz & Jansen (2007) Nature Reviews Genetics 8, 235-243
http://www.molgenis.org
Model of a variant<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"
Model of a variant<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"
Small model<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"
Small model<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"
Model variation, Reuse software
Code generationCode generation
+
inbreed
100
10.000
1,000,000
100,000
10,000
10,000,00
QTL profiles
network
correlate
genomestrains
individuals
markers
expressions
preprocess
probesmicroarrays
100
hybridize
100,000
genotype genotypes
norm exprs.
10.000
map
Step 1:Model variation Points(biology)
Step 2. Automate commonPatterns(informatics)
Step 3. Reuse in family of projects
Your model<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"
Your model<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"
Add new biology to model15
Reusable framework and generators
Reusable framework and generators
Model of a variant<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"
Model of a variant<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int" +
Model new protocols
15
Use new protocols
inbreed
100
10.000
1,000,000
100,000
10,000
10,000,00
QTL profiles
network
correlate
genomestrains
individuals
markers
expressions
preprocess
probesmicroarrays
100
hybridize
100,000
genotype genotypes
norm exprs.
10.000
map
Your model<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"
Your model<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"
Add new features to software16
Reusable framework and generators
Reusable framework and generators
Model of a variant<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int"
Model of a variant<!-- entity organization --><entity name="Experiment" label="Experiment"> <field name="ExperimentID" key="1“ readonly="true" label="ExperimentID(autonum)"/> <field name="Medium" type="xref" xref_field="Medium.name"/> /> <field name="Protocol" label="Experiment Protocol"/> <field name="Temperature" type="int" +
inbreed
100
10.000
1,000,000
100,000
10,000
10,000,00
QTL profiles
network
correlate
genomestrains
individuals
markers
expressions
preprocess
probesmicroarrays
100
hybridize
100,000
genotype genotypes
norm exprs.
10.000
map
Add features once
16
Added automatically
… repeatedly
http://www.molgenis.orgSwertz & Jansen (2007) Nature Revies Genetics 8, 235-243
NextGenSeq
MassSpec/Metab.
Experiments
WorkBiologist challenges Software solutions
Metabolomics support platform
17
|[email protected] | 19
Step 0: Brainstorm20
Observedvalue
Observedvalue*
Observationtarget
Observationtarget
time
Observablefeature
Observablefeature
*
PanelPanel IndividualIndividual* Observed Relation
Observed Relation
Inferred Value
Inferred Value
*
Step 1a: define the data model
1: molgenis_db.xml
506 lines of XML code16 entities, 67 fields
Step 1b: add ui model
1: molgenis_db.xml
50 lines of XML code(forms, menu’s and plugins)
Language elements<?xml version="1.0" encoding="UTF-8"?>
<!ELEMENT molgenis (description? , (module | entity |form | menu | plugin)*)>
<!ATTLIST molgenis name CDATA #REQUIRED>
<!ATTLIST molgenis label CDATA #IMPLIED>
<!ATTLIST molgenis version CDATA #IMPLIED>
<!ELEMENT description ANY>
<!ELEMENT module (description?, entity+)>
<!ATTLIST module name CDATA #REQUIRED>
<!ELEMENT entity (description?,field*,unique*)>
<!ATTLIST entity name CDATA #REQUIRED>
<!ATTLIST entity abstract (true|false) #IMPLIED>
<!ATTLIST entity implements CDATA #IMPLIED>
<!ATTLIST entity extends CDATA #IMPLIED>
<!ATTLIST entity decorator CDATA #IMPLIED>
<!ATTLIST entity description CDATA #IMPLIED>
<!ELEMENT field EMPTY>
<!ATTLIST field name CDATA #REQUIRED>
<!ATTLIST field type CDATA #IMPLIED>
<!ATTLIST field label CDATA #IMPLIED>
<!ATTLIST field length CDATA #IMPLIED>
<!ATTLIST field xref_field CDATA #IMPLIED>
<!ATTLIST field xref_label CDATA #IMPLIED>
<!ATTLIST field enum_options CDATA #IMPLIED>
<!ATTLIST field default CDATA #IMPLIED>
<!ATTLIST field auto (true|false) #IMPLIED>
<!ATTLIST field nillable (true|false) #IMPLIED>
<!ATTLIST field unique (true|false) #IMPLIED>
<!ATTLIST field readonly (true|false) #IMPLIED>
<!ATTLIST field hidden (true|false) #IMPLIED>
<!ATTLIST field description CDATA #IMPLIED>
<!ELEMENT unique EMPTY>
<!ATTLIST unique fields CDATA #REQUIRED>
<!ATTLIST unique subclass (true|false) #IMPLIED>
<!ATTLIST unique description CDATA #IMPLIED>
<!ELEMENT form (form*,menu*,plugin*)>
<!ATTLIST form name CDATA #REQUIRED>
<!ATTLIST form entity CDATA #REQUIRED>
<!ATTLIST form label CDATA #IMPLIED>
<!ATTLIST form view (list|record) #IMPLIED>
<!ATTLIST form readonly (yes|no) #IMPLIED>
<!ELEMENT menu (form*,menu*,plugin*)>
<!ATTLIST menu name CDATA #REQUIRED>
<!ATTLIST menu label CDATA #IMPLIED>
<!ELEMENT plugin (form*,menu*,plugin*)>
<!ATTLIST plugin name CDATA #REQUIRED>
<!ATTLIST plugin type CDATA #REQUIRED>
<!ATTLIST plugin label CDATA #IMPLIED>
See practical.
23
Generates 150 files, 30k lines of Java, SQL and R code + docs(tomcat/mysql; hsqldb, psql, jpa/hibernate, jetty in alpha )
0 INFO [myFactory] working dir: D:\Development\Molgenis33Workspace\molgenis4phenotype78 INFO [myFactory] MOLGENIS version 3.3.0-testing94 INFO [myFactory] Using options:model_database = [pheno_db.xml] #File with data structure specification (in MOLGENIS DSL).model_userinterface = pheno_ui.xml #File with user interface specification (in MOLGENIS DSL). Can be same file as model_databaseoutput_src = generated/java #Output-directory for the generated project.output_hand = handwritten/java #Output-directory for the generated project.output_sql = generated/sql #Output-directory for the generated sql files.output_doc = WebContent/doc #Output-directory for the generated documentation.output_type = #Output type of the project, either war (for use in tomcat) or jar (standalone).output_web = WebContent #Output-directory for any generated web resourcesdb_driver = com.mysql.jdbc.Driver #Driver of database. Any JDBC compatible driver should work.db_user = molgenis #Username for database. db_password = xxxxxx #Password for database. db_uri = jdbc:mysql://localhost/pheno #Uri of the database. Default: localhostdb_filepath = attachedfiles #Path where the database should store file attachements. Default: nulldb_jndiname = jdbc/molgenisdb #Used to create a JDBC database resource for the applicationobject_relational_mapping = subclass_per_table #Expert option: Choosing OR strategy. Either 'class_per_table', 'subclass_per_table', 'hierarchy_per_table'. Default: class_per_tablemapper_implementation = multiquery #Expert option: Choosing wether multiquery is used instead of prepared statements. Default: falseexclude_system = true #Expert option: Whether system tables should be excluded from generation. Default: trueforce_molgenis_package = false #Expert option. Whether the generated package should be 'molgenis' or the name specified in the model. Default: falseauth_loginclass = org.molgenis.framework.security.SimpleSecurity #Expert option.verbose = true #This switch turns the verbose-mode on.compile = false #This switch makes the factory also compile (usefull outside IDE).mail_smtp_protocol = #Sets the email protocol, either smtp, smtps or null. Default: null meaning email disabledmail_smtp_hostname = localhost #SMTP host server. Default: localhostmail_smtp_port = 25 #SMTP host server port. Default: 25mail_smtp_user = #SMTP user for authenticated emailing. Default: null.mail_smtp_password = #SMTP user for authenticated emailing. Default: null.
110 INFO [MolgenisLanguage] parsing db-schema from [pheno_db.xml]780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable797 WARN [Entity] [WARNING]: missing key 0 for entity Nameable797 WARN [Entity] [WARNING]: missing key 0 for entity Nameable844 INFO [MolgenisLanguage] parsing ui-schema937 INFO [main] generating ....1717 INFO [TableDocGen] generated WebContent\doc\tabledoc.html2076 INFO [EntityDocGen] generated WebContent\doc\objectmodel.html2436 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram.dot2545 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.system.dot2748 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.observation.dot2842 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.target.dot2998 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.variable.dot3138 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.protocol.dot3997 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-minimal-diagram.dot4184 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.system.dot4388 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.observation.dot4606 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.target.dot4731 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.variable.dot4887 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.protocol.dot5184 INFO [ClassDocGen] generated WebContent\doc\classmodel.html5293 INFO [InMemoryDatabaseGen] generated generated\java\ui\data\InMemoryDatabase.java5609 INFO [MySqlCreateSubclassPerTableGen] generated generated\sql\create_tables.sql5671 INFO [JDBCDatabaseGen] generated generated\java\ui\JDBCDatabase.java5921 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Identifiable.java5921 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Nameable.java5968 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\OntologySource.java6014 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\OntologyTerm.java6030 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Investigation.java6061 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservableFeature.java6124 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservedValue.java6170 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservedRelationship.java6217 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\InferredValue.java6233 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservationTarget.java6280 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Individual.java6311 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Panel.java6326 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\CodeList.java6327 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Code.java6374 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Protocol.java6390 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ProtocolApplication.java6405 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ProtocolParameter.java6437 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ParameterValue.java6452 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\InferredValue_derivedFrom.java6468 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Panel_individuals.java6483 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Protocol_observableFeatures.java6499 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Protocol_protocolComponents.java6624 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\OntologySourceMapper.java6655 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\OntologyTermMapper.java6671 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\InvestigationMapper.java6702 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservableFeatureMapper.java6733 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservedValueMapper.java6780 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservedRelationshipMapper.java6827 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\InferredValueMapper.java6842 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservationTargetMapper.java6873 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\IndividualMapper.java6889 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\PanelMapper.java6905 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\CodeListMapper.java6936 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\CodeMapper.java6951 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ProtocolMapper.java6983 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ProtocolApplicationMapper.java6998 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ProtocolParameterMapper.java7029 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ParameterValueMapper.java7045 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\InferredValue_derivedFromMapper.java7061 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\Panel_individualsMapper.java7076 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\Protocol_observableFeaturesMapper.java7092 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\Protocol_protocolComponentsMapper.java7217 INFO [JDBCMetaDatabaseGen] generated generated\java\ui\JDBCMetaDatabase.java7263 INFO [CountPerEntityGen] generated generated\sql\count_per_entity.sql7310 INFO [CountPerTableGen] generated generated\sql\count_per_table.sql7341 INFO [FillMetadataTablesGen] generated generated\sql\insert_metadata.sql7405 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\OntologySourceCsvReader.java7420 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\OntologyTermCsvReader.java7420 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\InvestigationCsvReader.java7436 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservableFeatureCsvReader.java7452 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservedValueCsvReader.java7467 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservedRelationshipCsvReader.java7483 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\InferredValueCsvReader.java7498 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservationTargetCsvReader.java7514 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\IndividualCsvReader.java7514 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\PanelCsvReader.java7530 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\CodeListCsvReader.java7545 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\CodeCsvReader.java7545 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ProtocolCsvReader.java7561 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ProtocolApplicationCsvReader.java7561 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ProtocolParameterCsvReader.java7576 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ParameterValueCsvReader.java7576 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\InferredValue_derivedFromCsvReader.java7592 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\Panel_individualsCsvReader.java7608 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\Protocol_observableFeaturesCsvReader.java7608 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\Protocol_protocolComponentsCsvReader.java7748 INFO [REntityGen] generated generated\java\pheno\core\R\OntologySource.R7748 INFO [REntityGen] generated generated\java\pheno\core\R\OntologyTerm.R7764 INFO [REntityGen] generated generated\java\pheno\core\R\Investigation.R7779 INFO [REntityGen] generated generated\java\pheno\core\R\ObservableFeature.R7779 INFO [REntityGen] generated generated\java\pheno\core\R\ObservedValue.R7795 INFO [REntityGen] generated generated\java\pheno\core\R\ObservedRelationship.R7795 INFO [REntityGen] generated generated\java\pheno\core\R\InferredValue.R7810 INFO [REntityGen] generated generated\java\pheno\core\R\ObservationTarget.R7810 INFO [REntityGen] generated generated\java\pheno\core\R\Individual.R7826 INFO [REntityGen] generated generated\java\pheno\core\R\Panel.R7826 INFO [REntityGen] generated generated\java\pheno\core\R\CodeList.R7842 INFO [REntityGen] generated generated\java\pheno\core\R\Code.R7857 INFO [REntityGen] generated generated\java\pheno\core\R\Protocol.R7857 INFO [REntityGen] generated generated\java\pheno\core\R\ProtocolApplication.R7873 INFO [REntityGen] generated generated\java\pheno\core\R\ProtocolParameter.R7873 INFO [REntityGen] generated generated\java\pheno\core\R\ParameterValue.R7888 INFO [REntityGen] generated generated\java\pheno\core\R\InferredValue_derivedFrom.R7888 INFO [REntityGen] generated generated\java\pheno\core\R\Panel_individuals.R7888 INFO [REntityGen] generated generated\java\pheno\core\R\Protocol_observableFeatures.R7904 INFO [REntityGen] generated generated\java\pheno\core\R\Protocol_protocolComponents.R7998 INFO [RApi] generated generated\java\source.R8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\IdentifiableHtmlForm.java8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\NameableHtmlForm.java8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\OntologySourceHtmlForm.java8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\OntologyTermHtmlForm.java8060 INFO [HtmlFormGen] generated generated\java\pheno\core\html\InvestigationHtmlForm.java8060 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservableFeatureHtmlForm.java8076 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservedValueHtmlForm.java8076 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservedRelationshipHtmlForm.java8076 INFO [HtmlFormGen] generated generated\java\pheno\core\html\InferredValueHtmlForm.java8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservationTargetHtmlForm.java8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\IndividualHtmlForm.java8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\PanelHtmlForm.java8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\CodeListHtmlForm.java8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\CodeHtmlForm.java8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ProtocolHtmlForm.java8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ProtocolApplicationHtmlForm.java8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ProtocolParameterHtmlForm.java8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ParameterValueHtmlForm.java8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\InferredValue_derivedFromHtmlForm.java8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\Panel_individualsHtmlForm.java8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\Protocol_observableFeaturesHtmlForm.java8138 INFO [HtmlFormGen] generated generated\java\pheno\core\html\Protocol_protocolComponentsHtmlForm.java8138 INFO [MolgenisServletContextGen] generated WebContent\META-INF\context.xml8169 INFO [MolgenisContextListenerGen] generated generated\java\servlet\ContextListener.java8232 INFO [MolgenisServletGen] generated generated\java\MolgenisServlet.java8403 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\InvestigationsForm.java8560 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ObservableFeaturesForm.java8591 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\PanelsForm.java8654 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\Panels\IndividualsForm.java8701 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ObservedValuesForm.java8732 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplicationsForm.java8825 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenu\ParameterValuesForm.java8857 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenu\ObservedValuesForm.java8888 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenu\InferredValuesForm.java9013 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\InferredValuesForm.java9044 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservableFeaturesForm.java9137 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservationTargets\IndividualsForm.java9169 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservationTargets\PanelsForm.java9200 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ProtocolsForm.java9293 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Protocols\ProtocolMenu\ParametersForm.java9325 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Protocols\ProtocolMenu\ProtocolComponentsForm.java9496 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\OntologyTermsForm.java9528 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\OntologySourcesForm.java9606 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\OntologySources\OntologyTermsForm.java9638 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\CodeListsForm.java9700 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\CodeLists\CodesForm.java9965 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenuMenu.java10012 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\MainMenu.java10059 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenuMenu.java10152 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenuMenu.java10230 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservationTargetsMenu.java10293 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\Protocols\ProtocolMenuMenu.java10324 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\OntologiesMenu.java11354 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\Main\ReportPlugin.java11557 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\Main\Ontologies\OntologyManagerPlugin.java11604 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\Model_documentationPlugin.java11604 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\RprojectApiPlugin.java11620 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\HttpApiPlugin.java11635 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\WebServicesApiPlugin.java11651 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\report\InvestigationOverview.ftl11807 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\OntologyBrowser\OntologyBrowserPlugin.ftl11807 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\DocumentationScreen.ftl11807 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\RprojectApiScreen.ftl11823 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\HttpAPiScreen.ftl11823 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\SoapApiScreen.ftl11854 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\report\InvestigationOverview.java12057 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\OntologyBrowser\OntologyBrowserPlugin.java12072 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\DocumentationScreen.java12088 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\RprojectApiScreen.java12088 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\HttpAPiScreen.java12088 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\SoapApiScreen.java12103 INFO [MolgenisServletContextGen] generated WebContent\META-INF\context.xml12259 INFO [SoapApiGen] generated generated\java\ui\SoapApi.java12353 INFO [CsvExportGen] generated generated\java\tools\CsvExport.java12431 INFO [CsvImportByNameGen] generated generated\java\tools\CsvImportByName.java12636 INFO [CopyMemoryToDatabaseGen] generated generated\java\ui\tools\CopyMemoryToDatabase.java
Step 2: Generate (MOLGENIS + Eclipse)24
Step 2: Generate25
customize... Model file XML
Generate
APIs in Java, R, Web services and HTTP
MyScriptPlugins
FormGen
MenuGenTreeGen
PluginGenMatrixGen
JTypeGenJDBCMapGen
JListGenJReadCsvGen
HSQLGen
JDatabaseGen
MySQLGen
RMatrixGen
WSGen
RListGen
datainfrastructure
user interactioninfrastructure
Communicationinfrastructure
Step 3: Use and evaluate
1: molgenis_db.xml
2: generate
3: use
Demo: http://wwwdev.ebi.ac.uk/microarray-srv/pheno/ Source: https://svn.gene.le.ac.uk/gen2phen/pheno-model
Step 4: optionally create plugins27
Alternative: existing database28
Existing db
Xml model
Generate
Connnects to
MolgenisExtractModel(automatic)
Result: Web GUI
1: molgenis_db.xml
2: brainstorm doc*
3: exchange format*
4: back and frontend*
Demo: http://wwwdev.ebi.ac.uk/microarray-srv/pheno/ Source: https://svn.gene.le.ac.uk/gen2phen/pheno-model
*autogenerated
Result: Documentation (generated-docs/objectmodel.html)
1: molgenis_db.xml
2: documentation*
*autogenerated
2009
2009
|
Result: Exchange format
my GaP
researcherresearcher
my Data
my Data
Investigation, ProtocolMaterial, Data annotations
Raw and processed data
Between labs
Central access
32
Result: Exchange format
1: molgenis_db.xml
2: documentation*
3: exchange format*
*autogenerated
Tools•Db•Import•Export
Result: Exchange format1: molgenis_db.xml
2: documentation*
3: exchange format*
*autogenerated
REST interface (/api/find and /api/add)
› http://localhost:8080/molgenis_distro/api/find/example.Data?name=mydata
35
/ 36
Result: R interfaces(/api/R)
Swertz et al, CASIMIR consortium, GEN2PHEN consortium (submitted)
source(“http://localhost:8080/xgap/api/R”)MOLGENIS is connected
#download data traits <- get.metabolitedata(name=“mytraits”)25 metabolite downloaded in 30ms
genotypes <- get.markerdata(name=“mygenotypes")744 marker downloaded in 30ms
#calculate ...
#upload results for others to useadd.data(qtls, name=“myqtls”) 18.600 data items added in 2sec
36
|[email protected] | / 37
Smedley, Swertz, Wolstencroft et al (2008) Brief. in Bioinf.
Result: SOAP interface(/api/soap?wsdl)
|[email protected] |38
Connect to R statistics
Connect to R statistics
Workflow ready web-services
Workflow ready web-services
UML documentation of your model
UML documentation of your model
Edit & trace your data
Edit & trace your data
Import/export to Excel
Import/export to Excel
plugin your own scripts (OntBrowse)
plugin your own scripts (OntBrowse)
Tech keywords: object oriented data models, multi-platform java, tomcat/glassfish web server, mysql/postgresql database, Eclipse/Netbeans IDE, Java API, WSDL/SOAP API, R-project API, MVC, freemarker templates and css for custom layout, open source.
find.investigation()102 downloaded
obs<-find.observedvalue(43,920 downloaded
#some calculationadd.inferredvalue(res)36 added
Architecture40
customize... Model file XML
Generate
APIs in Java, R, Web services and HTTP
MyScriptPlugins
FormGen
MenuGenTreeGen
PluginGenMatrixGen
JTypeGenJDBCMapGen
JListGenJReadCsvGen
HSQLGen
JDatabaseGen
MySQLGen
RMatrixGen
WSGen
RListGen
datainfrastructure
user interactioninfrastructure
Communicationinfrastructure
Generators?
<#list entities as entity>
CREATE TABLE ${SqlName(entity)} (
<#list dbFields(entity) as f>
<#if f_index != 0>, </#if>
${SqlName(f)} ${mysql_type(model,f)}
<#if !f.nillable> NOT </#if>NULL
<#if f.getDefaultValue()?exists>${f.getDefaultValue()}</#if>
<#if f.auto && f.type == "int"> AUTO_INCREMENT</#if>
<#if f.type == "xref">
, FOREIGN KEY (${SqlName(f)}) REFERENCES ${SqlName(model.getEntity(f.XRefEntity).getRootAncestor() )} (${SqlName(f.XRefField)}) ON DELETE RESTRICT
</#if>
</#list>
<#list entity.getAllKeys() as key>
<#if key_index == 0>
, PRIMARY KEY(${csv(key.fields)})
<#else>
, UNIQUE(${csv(key.fields)})
</#if>
</#list>
<#list entity.getAllIndices() as i>
, INDEX ${SqlName(i)} (${csv(i.fields)})
</#list>
) ENGINE=InnoDB;
41
Generator results
CREATE TABLE Investigation (
`
id INTEGER NOT NULL AUTO_INCREMENT
, name VARCHAR(255) NOT NULL
, description TEXT NULL
, PRIMARY KEY(id)
, UNIQUE(name)
, UNIQUE(id)
) ENGINE=InnoDB;
42
Database API:
› db.Database- count(filter)- find(filter)- add(entity)- add(list)- update(entity)- update(list)- remove(entity)- remove(list)- beginTx(),commitTx()- add(CsvReader)- find(CsvWriter)
› db.Query- equals(field,value),- greaterThan(f,v)- lessThan(f,v)- limit(f,v)- offset(f,v)- or()- etc.
› util.Entity- find(db,filter)- get(db,id)- query(db)
43
|[email protected] | 44
//findDatabase db = new app.JDBCDatabase("molgenis.properties");for(Contact c: db.find(Contact.class)) System.out.println(c);
//queryQuery<Address> q = db.query(Address.class);q.like(“lastname",“Lops");for(Address t: q.find()) System.out.println(t);
//updateContact one = new Contact();one.setFirstName(“Joeri");db.add(one);
//update in batchList<Contact> many = new ArrayList<Contact>();for(int i = 0; i < 10; i++){
Contact c = new Contact();c.setLastName(“friend_"+i);many.add(c);
}db.add(many);
form
MVC user interface framework
Tree MVC triple
45
FormModel.java
(state)
FormView.ftl
(layout)
FormController.java
(logic)
request(target,action)
handleRequest()reload()
response
layout
Database API
main
Menu
Form
menu
form
Plugin: your own MV(C)
public class FileServingPlugin extends PluginModel(AndController)
{
public String getViewTemplate()
{
return "plugin/examples/file/FileServingPlugin.ftl";
}
public String getViewName()
{
return "plugin_examples_file_FileServingPlugin";
}
public void handleRequest(Database db, Tuple request)
{
//if( request.getAction("do_add") )
}
public void reload(Database db)
{
//Query q = db.query(Experiment.class);
//q.like("name", "test");
//List<Experiment> recentExperiments = q.find();
}
}
46
File with view
Name of view (next slide)
What do on request?
What do on page refresh/after
request?
Plugin: view in freemarker<#macro plugin_examples_file_FileServingPlugin screen>
<form method="post" enctype="multipart/form-data" name="${screen.name}">
<input type="hidden" name="__target" value="${screen.name}"" />
<input type="hidden" name="__action" />
<!-- this shows a title and border -->
<div class="formscreen">
<div class="form_header" id="${screen.getName()}">
${screen.label}
</div>
<#--optional: mechanism to show messages-->
<#list screen.getMessages() as message>
<#if message.success>
<p class="successmessage">${message.text}</p>
<#else>
<p class="errormessage">${message.text}</p>
</#if>
</#list>
<div class="screenbody"><div class="screenpadding">
<#--begin your plugin-->
Create your plugin layout code here.
<#--end of your plugin-->
</div></div>
</div>
</form>
</#macro>
47
Future
› ‘Cooler’ UI framework?- GWT, ServerFaces, YUI, Flex?
› Semantic integration?- /api/rdf, semantically enhanced model
› Large files- Hadoop, BinaryMatrix, …
› Computation/services/workflows- Galaxy integration, GridGain, …
› Documentation!
› Your wish list here
48
Practical
› Copy and install VirtualBox (dvd/usb)› Copy and run molgenis-vm.vdi image› Pick up a manual› Start generating
50
2009
2009
|
Acknowledgements
Morris SwertzJoris LopsJoeri van der VeldeRichard ScheltemaMartijn DijkstraRudi AlbertsDanny ArendsBruno M. TessonGonzalo Vera RodriguezTomasz AdamuziakJuha MuiluDamian SmedleyKaty WolstencroftAndrew R. Jones …and others
Ritsert C. JansenCisca WijmengaHelen E. Parkinson Carole GoblePaul Schofield John M. HancockKlaus Schughartthe NBIC/BioAssist consortiumThe LifeLines consortiumthe GEN2PHEN consortium (Mm)the CASIMIR consortium (Hs)the Brassica Nutr. consortium (At)The PANACEA consortium (Ce)
51
NLNLEBI
|[email protected] | 52
|[email protected] | [email protected] / 53
Add WSDL to Taverna
|[email protected] | [email protected] / 54
Add WSDL to Taverna
|[email protected] | [email protected] / 55
Add WSDL to Taverna
|[email protected] | [email protected] / 56
Find in MOLGENIS from Taverna