Polymorphism & Restriction Fragment Length Polymorphism (RFLP)
Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis •...
Transcript of Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis •...
![Page 1: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/1.jpg)
Database and data-analysis infrastructure for atmospheric
studies
Takeshi HorinouchiRISH, Kyoto Univ.
![Page 2: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/2.jpg)
New IT Infrastructure for theInformation Explosion Era
![Page 3: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/3.jpg)
Info-plosion in geophysical fluid sciencesT Horinouchi et al
Info-plosion in astronomyVitual Observatory
M. Ohishi et al
Research Groups:
![Page 4: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/4.jpg)
Topics
• GPhys – a Ruby library for data analysis and visualization
• Gfdnavi – a tool to create Web-based database for data search, analysis, and visualization (demonstrated earlier by SeiyaNishizawa)
![Page 5: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/5.jpg)
Object-oriented languages are suitable to data analysis
• Information hiding can hide data structure
• Polymorphism can unify access
One can create a programming library to support consolidated handling of a variety of atmospheric data irrespective of dimensionality, sampling, and formatsCan be create software infrastructure for scientific data analysis
![Page 6: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/6.jpg)
Why Ruby?• Scripting language (interpreter) Rapid
development• Can be used interactively Good for trial and
error• Pure object-oriented and easy to use
Easy to develop reusable flexible softwareCommunity software sharing
• Easy to extend Reuse C and Fortran libraries• Many and increasing libraries (Networking / GUI
toolkit / database / distributed objects / etc.etc.) Easy to implement high-level services
• Good string-handling capability (like perl) You need it even when you deal with scientific data
![Page 7: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/7.jpg)
GPhys : a Ruby class library for Gridded Physical quantities
Developed by T Horinouchi, R Mizuata, S Takehiro, S Nishizawa, etc
• Supports unified access to NetCDF, GrADS, GRIB, and NuSDAS files (HDF5-EOS will also be supported)
• Easy to use as GrADS, but more powerful, flexible, and programmable
Available from http://ruby.gfd-dennou.org/products/gphys/
![Page 8: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/8.jpg)
Sample scripts (1)• Contour plot
1: require "numru/ggraph"2: include NumRu 3: gphys = GPhys::IO.open('T.jan.nc', 'T') 4: DCL.gropn(1)5: GGraph.contour(gphys) 6: DCL.grcls
One can put these lines in a start up file for interactive session
This file is a NeCDF.But the program will be the same for a GrADS file:
gphys = GPhys::IO.open(’T.jan.ctl’, ’T’)
![Page 9: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/9.jpg)
Sample scripts (2)• Line plot1: require "numru/ggraph"2: include NumRu3: gphys = GPhys::IO.open('T.jan.nc', 'T') 4: DCL.gropn(1)5: GGraph.set_fig( 'itr'=> 2 )6: GGraph.line( gphys.cut(135,35,false),
true, 'exchange'=>true )7: DCL.grcls
The only difference form the previous script
One can incrementally add options for publication-level graphics
![Page 10: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/10.jpg)
Sample scripts (3)• Deviation from zonal mean1: require "numru/ghys"2: include NumRu 3: gphys = GPhys::IO.open('T.jan.nc', 'T') 4: eddy = gphys – gphys.mean(’lon’)
5: ofile = NetCDF.create(’zm.nc’) 6: GPhys::IO.write( ofile, eddy )7: ofile.close
If you want to save thedata in a new NetCDF file (along with attributes and coordinate variables)
![Page 11: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/11.jpg)
Sample scripts (3’)• Deviation from zonal mean (by defining a
“method”)1: require "numru/ghys“2: module NumRu3: class GPhys4: def eddy(dim)5: self – self.mean(dim)6: end7: end8: end9:10: include NumRu 11: gphys = GPhys::IO.open('T.jan.nc', 'T') 12: eddy = gphys.eddy(’lon’)
![Page 12: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/12.jpg)
Also supported are
• Virtual concatenation over multiple files (tiling etc.)
• Big data handling by automatic process subdivision
• Mathematical / statistical oprations (e.g. FFT);
• Numerical/scientific libraries such as GSL can be combined. You can further extend GPhys by C / Fotran libraries
• etc etc
![Page 13: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/13.jpg)
Gfdnavi
• Developed under• by T. Horinouchi, S. Nishizawa, C.
Watanabe, etc. etc.• Most part was written by S. Nishizawa
Available from http://www.gfd-dennou.org/arch/davis/gfdnavi/
![Page 14: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/14.jpg)
Technologies
• GPhys for consolidated data access
• Ruby on Rails: A development framework for web-application with relational database – very comprehensive and productive
![Page 15: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/15.jpg)
• Register local data (and/or remote OPeNDAPdata) automatically with a command
• Serve the data for search, analysis, and visualization
• Supported data: gridded numerical data (in NetCDF, grib, etc.)
• UI: Web browsers and Web service
![Page 16: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/16.jpg)
Metadata DB
Directory tree
group
attributes
data files variables (numeric data)
supplementarytext files
description = “……..”param1 = value1param2 = [val21,val22]
• Metadata– name-value attributes– coordinate info (spatial, time, or any coordinate e.g. wavelength)
• Generated by directory scan– All directories, numerical data files, image files, etc are registered in
relational database
start end
temporal_attributes
1 1
1
n1
n
spatial_attributes
keyword_attributes
1n
variables
directories1
n
name type value
Treated asdirectories
![Page 17: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/17.jpg)
Screen shots
![Page 18: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/18.jpg)
Data finder userinterface
![Page 19: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/19.jpg)
Visualizaion / analysis window
![Page 20: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/20.jpg)
Search interface with Google Map support
![Page 21: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/21.jpg)
Network of GfdnaviUnder development by C Watanabe (Ochanomizu
Univ; database science)To create peer-to-peer network for cross search and cross useThen one can access local data and remote data together
RDB
RDB RDB
RDBRDBdataA dataB
dataC
![Page 22: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/22.jpg)
Knowledge/finding DB creation (new feature)
Visualization window Input form
![Page 23: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/23.jpg)
View the document
Re-drawing⇒ changesetting andredraw etc…
本文
![Page 24: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/24.jpg)
Can be used to create a page like thisTRMM Web pagehttp://www.eorc.nasda.go.jp/TRMM/index_j.htm
![Page 25: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/25.jpg)
Generated by:dcmodel-thumby山田@北大
![Page 26: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/26.jpg)
New search interface (Ocha U)
![Page 27: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/27.jpg)
3D visulaization (to be implemented at Nara W univ)
![Page 28: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/28.jpg)
Concluding remarks
• Ruby, a modern object-oriented scripting language, can help and accelerate atmospheric data analysis– The GPhys library can be a community tool
for atmospheric sciences• Gfdnavi can be a community tool for
atmospheric data sharing/publication– Active development is ongoing
![Page 29: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/29.jpg)
![Page 30: Database and data-analysis infrastructure for atmospheric ...€¦ · to data analysis • Information hiding Æcan hide data structure • Polymorphism Æcan unify access ÆOne can](https://reader033.fdocuments.in/reader033/viewer/2022050504/5f95d4c1dd854e125b1de9f1/html5/thumbnails/30.jpg)
What’s Ruby?• An object-oriented scripting (interpreted)
language• Open software (http://www.ruby-lang.org/)• Simple and sophisticated syntax
• Online version of a English bookwww.rubycentral.com/book by Thomas and Hunt