Post on 26-Dec-2015
• Background
• Implementation
• Pitfalls
• Factors to Consider
• Questions and Answers
Presentation Overview
Background / History
• Web/Portal History– Static – Dynamic – Application based - Portal
• iAssessment Evolution and Products– K-12 ASP Compliance and Reporting– Assessments and Survey– Teacher Resource Cataloging– Professional Development Grant/Policy
Management– Communication Tools
• uPortal – iAssessment Gateway Presentation System (GPS)– User management– Entity management– Individual client branding
• Controlled User Experience– User profile based– Role based applications– Client controlled not user controlled
• Integrated User Experience– Other disparate systems
Background / Solution GPS K12 Smart Portal
Implementation / Case Study
• GPS Case Study / CTAP2 California Department of Education– >300,000 teachers & administrators– 50,000 initial students for beta student system (with
one district having ~2 million students)• Use Model
– Administrators– Teachers– User created accounts– Gradual adoption– Peaky use times (grant deadlines, etc.)– Account reuse
Implementation / Case Study
• Current CTAP2 Statistics– ~200,000 accounts– Peaks of 1,000 concurrent logins– Peak “season” average 5,000 logins per day– Tables with more than 20 million rows
• Architecture Requirements– Expandable at top level
• Load balancing• Scaling• Easy machine addition
– Scalable software model• Fixed session at login
Implementation / Case Study
• CTAP2 Hardware– Web/Application Servers
• Sun 220R (2CPU/2GB) NFS Server• Four X1’s (1CPU/1GB)
– Database Server• SunFire v880 (6CPU/12GB)• T3 Disk Array redundant pair
– Cisco content switch– Cisco Pix Firewall
Implementation / Case Study
• Current Hardware Capabilities– X1 can handle ~150 users each– 220R ~400 users– Application front end current max load ~1,000
concurrent users– Database max load 3,000 users (keeping in
mind the different load factor of iAssessment’s applications running in uPortal)
• Still could use optimization as well.
Implementation / Case Study
• Software– Solaris 8 & Linux (web/application servers)– Sybase 12.5 / Solaris 8 (database servers)– Tomcat 4.0– Apache 1.3
• mod_webapp
– JDK 1.4.0– uPortal 2.0
Non-performance-based uPortal changes
• Created generic servlet wrapper channel– Run with and without uPortal
• Layout management modifications– Layout per user type, school type, etc.
• Authentication modifications– First name, last name, id, password
• Database connection pool sharing
Things That Worked
• Planned uPortal extensibility generally worked well
• Relative URL space worked well– Load balancing (
http://ctap2w{1,2,3,4,5}.iassessment.org/)– Integration into existing URL namespace– https might be a problem
Problems & Fixes
• HTML production: XML “well-formed”-ness and XLST processing speed– jTidy didn’t work well– Fearful of XSLT overhead– FIX: avoided XSLT processing
Problems & Fixes
• High memory use (400MB-1GB/process)– Intrinsic needs of applications and ???– Bugs (SmartCache, database connection
pool)• FIX: code changes
– Caused java.lang.Runtime.exec failure (no vfork equivalent in Java)
• FIX: created “exec server”
Problems & Fixes
• Large database hit with large groups– Debug Sybase JDBC driver and ~20,000 UP_USER rows
– uPortal 2.0 queries database for each member in set; long delay during account creation
– FIX: used production JDBC driver; uPortal 2.0 code should be changed
Problems & Fixes
• Proxy servers and “User-Agent” HTTP header removal– FIX: wrapped original HttpServletRequest &
protected BrowserInfo– Better FIX: all in one wrapper
Problems & Fixes
• Cisco content switch problems– Used IP mapping because cookies didn’t work– Load balancing from NAT/proxy didn’t work
well with IP mapping– Single point of failure– No method of determining which machine
user was on– FIX: using content switch for login redirection
only
Problems & Fixes
• Caching of layouts– Performance gain, but required inter-JVM
communication• FIX: created small inter-JVM communication
system
– Aggregated Layouts implication (with multiple JVMs)?
• User-caused problems– Multiple submits of same request
• FIX: used a ‘please wait’ screen where needed
Performance Modifications
• Did layout caching– Per JVM– Reloaded on demand from inter-JVM
message
• Bypassed channel-level XSLT transformations
• Didn’t use debug drivers in production
• Limited duplicate burdensome requests
Scalability Modifications
• Support for multiple web/application server machines
• Support for large JVM size
Lessons Learned
• Memory is limiting factor for GPS
• Well-formed HTML hard to achieve even when we control all the code
• Simple load balancing via publicly exposed web/application servers easiest and most effective
Lessons Learned
• Environment is hostile– Unfriendly proxy servers– “Mega proxies” (AOL)– Impatient users– Impatient firewalls
Things to Consider
• Architecture to support multiple web/application servers– Inter-JVM issues possibly– Per-server information (# of users, etc.)?
• Existing servlet code– To wrap or not?– Bypass XSLT?
Things to Consider
• Environment– Non-technical users– Unfriendly proxies, or firewalls?
• Application types– Large JVM size?– CPU/database intensive?