Building a platform from open source
-
Upload
dustin-whittle -
Category
Technology
-
view
1.728 -
download
2
Transcript of Building a platform from open source
Building a platform from open source
SYMFONY-PROJECT.COM | DEVELOPER.YAHOO.COM
Overview
Why symfony?
symfony vs. ysymfony
Social Search: Delicious and Answers
International: ShopGenie.co.uk, FoxyTunes, Digu
Yahoo! Open Strategy Developer Tools
YUI, Design Patterns, Tutorials
Data & Social APIs YQL: Yahoo! Query Language
Who am I?
Working with symfony since open source symfony Core Team Member
Responsible for the development and support of ysymfony at Yahoo! Y! Answers, Delicious, Y! Widgets, Y! Application Platform
Developer advocacy for Yahoo! Developer Network
Contact [email protected]
@dustinwhittle
DEVELOPER.YAHOO.COM EXAMPLES | TUTORIALS | CODE SAMPLES
YAHOO! IS POWERED BY OPEN SOURCE TECHNOLOGIES
FREEBSD | LINUX | APACHE | PHP | MYSQL | BUGZILLA | HADOOP | SYMFONY
YAHOO! EMBRACES OPEN STANDARDS
W3C | MICROFORMATS | OAUTH | OPENID | OPENSOCIAL
YAHOO! GIVES BACK TO OPEN SOURCE
YUI | BROWSER PLUS | DESIGN PATTERNS | R3 | YSLOW + PERFORMANCE RULES
YAHOO! SHARES ITS DATA THROUGH OPEN APIS AND WEB SERVICES
YQL | PIPES | BOSS | CONTACTS | UPDATES | MAIL | DELICIOUS | FLICKR | UPCOMING | HOTJOBS | MAPS | FIREEAGLE | GEOLOCATION | LOCAL | TRAFFIC | WEATHER |
MUSIC | ANSWERS | SHOPPING | FINANCE | TRAVEL
YAHOO! ENGAGES COMMUNITIES WITH OPEN HACK EVENTS AROUND THE WORLD
Conferences | Hack Days | HackU | Tech Talks | YDN Theater
Users
Load Balancers
Frontend
PHP APC, PEAR, PECL, Custom Extensions
Linux
ysymfony / YUI Apache Custom Modules
Backend
MySQL/Memcache YQL / Web Services Ad API User API
Why a frontend platform?
Rasmus says “frameworks are not well suited for Yahoo!” Build applications to requirements
Do exactly what you need: no more, no less
Understand that frameworks add a lot of overhead
Choosing functional components is a better t
Despite choosing open source or building your own Everyone uses a framework
If you use open source, use only the pieces you need
Yahoo! needs from a frontend platform Fit existing environment (RHEL/PHP5/Apache) Development Cycle – How easy to develop, test, and
deploy? Clean separation between data, logic, and display (MVC) Independent model layer to t service oriented
architecture Extensible and pluggable Internationalization and localization support Detailed documentation and active community of
support Open source and ability to contribute back
Why a framework at all?
Another software layer (ysymfony, yphp, yapache) Factors out common patterns
Code Layout Conguration URL Routing Authentication / Security Form Validation / Repopulation Internationalization / Localization
Encourages good design Abstraction > Consistency > Maintainability
The choice to adopt symfony?
Philosophy Full-stack framework for building complex web
applications Adopt best ideas from anywhere, using existing code if
available (Mojavi, Prado, Rails, Django)
Design Clean separation between Model, View, and Controller Controller using modules and actions Views using templates in straight PHP with helpers Easy to reuse view modules to compose a page
Layouts, Components, Partials, Slots
The choice to use symfony
Congurability / Flexibility Features we do not want are easily disabled
Use of factories for easy customization
Documentation / Support Community The Denitive Guide to symfony
Askeet, Jobeet, Cookbooks, Advents
Active community with wiki, mailing lists, forums, irc channel
Why ysymfony for Yahoo! teams?
Eliminate common patterns by adding a layer on PHP Code layout/structure (MVC)
Conguration
Internationalization
ysymfony is just a toolkit Learn one set of tools
Shift between multiple projects
Consistency
Long term maintainability through platform
How Yahoo! uses symfony Full Stack
Yahoo! Bookmarks Yahoo! Answers Delicious Yahoo! Widgets Yahoo! Application Platform ShopGenie.co.uk, Foxytunes, Digu Internal Dashboards + Reporting Tools
Components Dependency Injection Forms Caching (APC/Memcache) Routing
A look at Yahoo! Answers
http://answers.yahoo.com
Yahoo! Answers is the largest collection of human knowledge on the Web with more than 135 million users and 515 million answers worldwide (Yahoo! Internal Data, March 2008).
Yahoo! Answers is the 2nd ranked education & reference site on the web (comScore)
Available in 26 markets and 12 languages
Yahoo! Answers at the beginning
Started as a small development team on PHP4 from a fork of Yahoo! Taiwan Knowledge+
Launched December 2005 by December 2006 there were 60 million users and 65 million answers
The code base eventually became difcult to maintain and iterate new features
Large distributed development teams (US / UK)
The big picture
A complete platform for building web applications from frameworks PHP Framework
JavaScript Framework
CSS Framework
UI Design Patterns + Best Practices
Development Tools (logger, proler, debugger, docs)
Unit + Functional Testing Frameworks (LIME / YUI Test)
Deployment Tools (rsync deployment system)
What does Yahoo! change? Minor changes to t our environment
Most changes are easily implemented via factories/plugins
Dropped the ORM and pushed down the stack (SOA) Added a parallel API Dispatcher (ysfAPIClientPlugin)
Added dimensions to congurations (ysfDimensionsPlugin) Integrated R3 translation/template management (ysfR3Plugin)
R3 - http://developer.yahoo.com/r3/
Created a build and deployment solution (ysfBuildPlugin) Uses internal tools for packaging/deployment
Integrate support for Y! User Interface libraries (ysfYUIPlugin)
Propel or Doctrine or ???
No ORM for large projects
Doctrine for medium sized projects
Service Oriented Architecture Platforms/Backends as services (reusable to all)
Thin Controller/Fat Model (where model == services)
Use PHP as the frontend glue No heavy lifting in PHP = Push down the stack
Java/C++/Erlang + JSON/XML
Localizing with dimensions
Cascading Conguration based on YAML Framework -> Project -> Application -> Module
Extending the cascade to be based on dimensions Dimensions can be anything (and can be chained together)
Data Center + Environment for customizing congurations
Culture for localizing user interface + data
Theme for customizing look and feel
User info (is user on corporate intranet?)
Caching
A build and deployment system
Aggregate and minify stylesheets and javascripts Rewrite templates, css, js for CDN (Akamai, S3, …) Generate translations for congurations + templates Generate conguration cache Aggregate core classes + remove debug statements Run lint, unit, functional tests Package applications as .tgz Deployment via packages Integrated with Yahoo! internal tools
Debug Toolbars
Hello World Performance
Hello world benchmarks are generally not useful You don’t use a framework to write hello world
die(‘hello world’);
Performance is relative to the features you use ORM
I18N
Output Escaping
Forms/Validation
Ability to scale != performance
What does it mean to scale?
A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system.
High Availability + Scalability + Performance Bigger dataset, more trafc, maintainable Not about performance
PHP is slow, but it is very rarely your bottleneck
Languages do not scale, architectures do Planning to grow and planning to fail
Capacity Planning Business Continuity Planning
Scaling – The basics in PHP PHP is rarely the bottleneck (even though it can be slow)
“Most performance comes not from the language, but from application design” – Rasmus
Share Nothing Architecture Independent, self-sufcient, no single point of contention No local storage = No PHP Sessions
Use a database (works for distributed) Use a small signed cookie (ideal)
Important data in database Individual expiration on session objects Small data items
Use a distributed cache Memcache
Forget about small efciencies Premature optimization is the root of all evil.
“ vs. ‘, echo vs. print
Scaling Databases – The basics
Drop the ORM It’s a choice of convenience over performance
Master/Slave Replication First steps Helps with reads, writes are still bottleneck
Partitioning Segmenting data
Sharding (horizontal partitioning) Segmenting data onto different physical machines Make problems smaller, easier to grow
Improving latency with Caching
Always use PHP opcode cache (APC, Xcache, etc) Use for routing and i18n cache
Memcache (distributed cache) Use for view cache
Distributed invalidation can be a pain sfViewCacheManager makes this easy!
Be intelligent about cache_keys (uri, user, state)
There is a ne line to caching At what point do you spend more time managing the
cache, than reading from it?
Tweaking Performance
Do not use .htaccess (move to real apache cong) Set a minimal include path Increase realpath_cache_size + realpath_cache_ttl Use apc.stat=0 Don’t use features you do not need
Disable in settings.yml / factories.yml
Use core_compile to aggregate classes to reduce le i/o Remove debug statements + optimize le lookups
sfOptimizerPlugin / project:optimize
Use @routeName and use caching for factories
Symfony
Symfony 2 is a set of cohesive yet decoupled components Easier to use a component to solve a single problem
Easier to build micro frameworks that solve specic problems
Yahoo! Teams generally prefer solutions that are specic to their exact problem
Selling the full stack can be difcult when a team only wants a few components
Symfony 2 is the right direction, even if it breaks backwards compatability
symfony-reloaded.org
Do it yourself for cheap
Open source software = Free Apache PHP MySQL Memcache / Perlbal / MogileFS / Squid / Gearman symfony / Doctrine / Propel / Swift Nagios / Cacti
Amazon Shared Infrastructure = Cheap EC2 Cloud S3 Storage + Cloudfront SimpleDB
What is Yahoo! Developer Network? The Yahoo! Developer Network offers open source tools
and open data APIs to make it easy for developers to build applications and mashups.
50+ APIs / Web Services Developer Dashboard to create/manage Oauth applications Tutorials + Code Samples on using our apis Complete API Documentation
Yahoo! User Interface libraries ASTRA Flash Components Design Patterns Library Evangelism: Conferences / Theater / Blogs / Events
JavaScript Framework Utilities - YAHOO, Dom, Event, Animation, Browser History
Manager, Connection Manager, Cookie, DataSource, Drag and Drop, Element, Get, ImageLoader, JSON, Resize, Selector, Loader
Controls / Widgets - AutoComplete, Button, Calendar, Charts, Color Picker, DataTable, ImageCropper, Rich Text Editor, Slider, Uploader
Container (Module, Overlay, Panel, Tooltip, Dialog), Layout Manager, Menu, TabView, TreeView
Debug – Logger, Proler, Test
Y! Developer Network – YUI Javascript
CSS Foundation Reset - Neutralizes browser CSS styles
Base - Applies consistent style foundation
Fonts - Foundation for typography and font-sizing
Grids - Thousands of wireframe layouts
User Interface Design Patterns Library Proven solutions to common interfaces
http://developer.yahoo.com/ypatterns/
Grade Browser Support / Progressive Enhancement
Y! Developer Network – YUI CSS
More than 275 functional examples http://developer.yahoo.com/yui/examples/
YSlow + Performance Rules http://developer.yahoo.com/performance
YUI Blog http://yuiblog.com/
Mailing List @ Yahoo! Groups http://tech.groups.yahoo.com/group/ydn-javascript/
Y! Developer Network – Documentation
A Query Language for the Web
SELECT * FROM INTERNET
Thousands of web services that provide valuable data
Require developers to read documentation and form URLs/queries.
Data is isolated and can not be combined
Needs combining, tweaking, shaping even after it gets to the developer.
Before YQL
SQL-Like Language Synonymous with Data access
Familiar to developers
Expressive enough to get the right data
Self Describing - show, desc table
Allows you to query, lter and join data across Web Services.
Y! Open Stack – YQL
developer.yahoo.com/yql/console/
YQL – Open Tables
Twitter Google Facebook Friendfeed Wesabe Whitepages Zillow Search Weather
Flickr Upcoming Delicious Dopplr Github New York Times Shopping …any web service can be
as easy as SQL
Available on github - http://github.com/spullara/yql-tables/
YQL – Open Tables
YQL – Open Tables
YQL – Open Tables
select * from social.connections
select * from delicious.feeds.popular
select * from ickr.photos.interestingness
select * from friendfeed.status
select * from github.checkins
YQL - Examples
OAuth Endpoint
http://query.yahooapis.com/v1/yql?q=...
Public Endpoint
http://query.yahooapis.com/v1/public/yql?q=
YQL Console
http://developer.yahoo.com/yql/console
YQL – API End Points
Open Source Communities
http://www.symfony-project.org/jobeet/1_2/
Documentation
http://www.symfony-project.org/doc
Practical symfony - Jobeet
The Denitive Guide to symfony
The symfony Reference Guide
symfony Forms in Action
Doctrine ORM for PHP
Cookbooks + Cheat sheets + Wiki
API Documentation
Books worth reading
QUESTIONS? (I will be around, so please say hello and ask questions)
DEVELOPER.YAHOO.COM EXAMPLES | TUTORIALS | CODE SAMPLES
WANT TO JOIN YAHOO? WE ARE HIRING!