Introduction Introduction to Web to Web

EngineeringEngineeringTsinghua University, Tsinghua University,

BeijingBeijing

April 2008April 2008

Bebo WhiteBebo White

bebo@slac.stanford.edubebo@slac.stanford.edu

My Basic PremisesMy Basic Premises

Developing a Web-based system is no Developing a Web-based system is no longer an event, it is a processlonger an event, it is a process

Web-based systems are growing systemsWeb-based systems are growing systems

Web-based applications are adding a whole Web-based applications are adding a whole new dimension to software developmentnew dimension to software development

Terms I UseTerms I Use

Web System – an infrastructure or system Web System – an infrastructure or system enabling the operation of a Web applicationenabling the operation of a Web application

Web Application – a distributed application Web Application – a distributed application that accomplishes a certain business need that accomplishes a certain business need based on the technologies of WWW and that based on the technologies of WWW and that consists of a set of Web-specific resourcesconsists of a set of Web-specific resources

The Web Started The Web Started SimpleSimple

““Even simple rules lead to complex Even simple rules lead to complex behavior” – Wolfram, behavior” – Wolfram, New ScienceNew Science, 2002, 2002

““It is now technically possible and feasible It is now technically possible and feasible to put the entire creative works of man to put the entire creative works of man online” – Kahle, The Internet Archivesonline” – Kahle, The Internet Archives

E-commerce (Web/Internet) accounted for E-commerce (Web/Internet) accounted for 2% of US retail sales in 4Q, 2004 - $18.4 2% of US retail sales in 4Q, 2004 - $18.4 billion – US Census Bureaubillion – US Census Bureau

Project Failure Rates Project Failure Rates (1/2)(1/2)

IEEE SoftwareMarch/April 2005www.computer.org/software

Project Failure Rates Project Failure Rates (2/2)(2/2)

IEEE SoftwareMarch/April 2005www.computer.org/software

Problems with Large Problems with Large Web-based ProjectsWeb-based Projects

(Source: Epner, M., Cutter Consortium)

What About Complex Web Systems?

Range of ComplexityRange of Complexity

StaticSite

DataCollector Site

Dynamic DataAccess

Web Application

Service-BasedWeb Applicationand Federations

Dynamically Created

Site

Service

Application CenteredDocument Centered

Com

plex

Sim

ple

Following “Web Site Engineering: Beyond Web Page Design”, by Th. Powell et al.

CharacteristicsCharacteristicsToday’s focus on large-scale and ubiquitously useable Web ApplicationsToday’s focus on large-scale and ubiquitously useable Web Applications

Many Users – many languages – many culturesMany Users – many languages – many culturesDifferent access mechanisms Different access mechanisms Many User AgentsMany User Agents

Presents large volume of interrelated information (including different Presents large volume of interrelated information (including different media) and processesmedia) and processes

Appropriate presentationAppropriate presentationProgression through activities – finish one thing before starting Progression through activities – finish one thing before starting anotheranotherOften guides userOften guides user

Growing and increasing complexity Growing and increasing complexity Many product iterations/versions/refinements (calls for Reuse)Many product iterations/versions/refinements (calls for Reuse)Many developers and operators, complex handling of temporal media Many developers and operators, complex handling of temporal media (e.g. publishing of company news)(e.g. publishing of company news)Customization, Personalization, Security issues Customization, Personalization, Security issues

and a lot more...and a lot more...E.g. “Up-to-date” by following trendsE.g. “Up-to-date” by following trends

44thth Generation of Web Generation of Web Systems?Systems?

SOA Functionality Composition Engine Federation, Security Transaction, etc.

SOA Functionality Composition Engine Federation, Security Transaction, etc.

Configuration/Context Components, End Points Semantic Web Policy, Permissions, etc.

Configuration/Context Components, End Points Semantic Web Policy, Permissions, etc.

Model-drivenModel-drivensupport systemssupport systems

Model-drivenModel-drivensupport systemssupport systems

Communication

Identification Provider / Security Token ServiceIdentification Provider / Security Token Service

IP/STSIP/STSIP/STSIP/STS

SecurityContext

SecurityContext

Presence /Location Server

Presence /Location Server

P/LSP/LSP/LSP/LS

Presence/LocationContext

Presence/LocationContext

Cluster:Cluster:

ClassificationClassificationClassificationClassificationClassificationClassificationClassificationClassification

Web Services Universe

ClassificationClassificationClassificationClassification

DiscoveryDiscoveryDiscoveryDiscovery

HTTP/SOAPHTTP/SOAP

Web Application Web Application DevelopmentDevelopment

Still Ad-Hoc instead of a disciplined procedureStill Ad-Hoc instead of a disciplined procedure Copy-and-Paste ParadigmCopy-and-Paste Paradigm

Lack between Design Model and Implementation Lack between Design Model and Implementation ModelModel

Design concepts get lost in the underlying modelDesign concepts get lost in the underlying model

Short lifecycle of a Web Application -> Maintenance Short lifecycle of a Web Application -> Maintenance and Evolution issues -> Reuse issuesand Evolution issues -> Reuse issues

-> Web Crisis

Web Application Web Application Development Current Development Current

PracticesPractices Lacks rigor, systematic approachLacks rigor, systematic approach

The completed system is not what the user The completed system is not what the user wantswants

System not developed on time, cost overrunsSystem not developed on time, cost overruns Lacks scaleability and maintainability, hence Lacks scaleability and maintainability, hence

limited useful lifelimited useful life Does not meet performance requirementsDoes not meet performance requirements Resources are wastedResources are wasted

Web Systems: Web Systems: ProblemsProblems

ProblemsProblems Inability to maintainInability to maintain Unable to meet evolving needs and grow at the Unable to meet evolving needs and grow at the

rate needed – scaleabilityrate needed – scaleability Unreliable – crashesUnreliable – crashes

Web-dependent organizations cannot afford to Web-dependent organizations cannot afford to havehave Faulty systems – reliability, security issuesFaulty systems – reliability, security issues Frequent downtime – dependabilityFrequent downtime – dependability Wrong, inconsistent, or stale content/informationWrong, inconsistent, or stale content/information

Web systems problems are not easy to hideWeb systems problems are not easy to hide

Web Development Web Development IssuesIssues

Many developers think that Web application Many developers think that Web application development is just accomplished using “off the shelf” development is just accomplished using “off the shelf” toolstools

They have been taught to think this wayThey have been taught to think this way

Certain classes of applications do fit this simple Certain classes of applications do fit this simple generalizationgeneralization

Many other Web applications do notMany other Web applications do not

““There is more to Web application development than There is more to Web application development than visual design and user interface”visual design and user interface”

Planning, system design, testing, continual Planning, system design, testing, continual maintenance, quality assurance, performance maintenance, quality assurance, performance evaluation, scalability,….evaluation, scalability,….

Consideration to Consideration to Quality?Quality?

Lack of consideration Lack of consideration to:to: NavigationNavigation AccessibilityAccessibility ScalabilityScalability ReliabilityReliability MaintainabilityMaintainability UsabilityUsability

Compatibility and Compatibility and interoperabilityinteroperability

SecuritySecurity ReadabilityReadability

Web Development – Web Development – Political ChallengePolitical Challenge

Anyone can be a Web content creator and Anyone can be a Web content creator and maintainer!maintainer!

Web System Web System Development Summary Development Summary

(1/2)(1/2) Less attention is given to development Less attention is given to development

methodologies, testing and evaluation, quality methodologies, testing and evaluation, quality assessment and controlassessment and control

Largely relies on individual’s own development Largely relies on individual’s own development practicespractices

Lack of realization of its lifecycleLack of realization of its lifecycle Analysis of needs, redesign, development (including Analysis of needs, redesign, development (including

coding), management, metrics, maintenancecoding), management, metrics, maintenance Calls for significant system-level and design decisionsCalls for significant system-level and design decisions

It is an exercise – not an eventIt is an exercise – not an event

Legitimate concern about the manner in which they Legitimate concern about the manner in which they are created and their long-term quality and integrityare created and their long-term quality and integrity

Web System Web System Development Summary Development Summary

(2/2)(2/2) In many cases, Web systems development isIn many cases, Web systems development is

ChaoticChaotic Failure-proneFailure-prone UnsatisfactoryUnsatisfactory

Why Do You Think That Web Systems are the #1 Target

of Attack?

Why Do You Think That Why Do You Think That Web Systems Are the #1 Web Systems Are the #1

Target of Attack?Target of Attack?

Risks to Web-Based Risks to Web-Based BusinessBusiness

Long system delivery timeLong system delivery time

Low responsiveness to business changesLow responsiveness to business changes

High project development and ongoing High project development and ongoing support costssupport costs

Questionable system qualityQuestionable system quality

Desires of Web-Based Desires of Web-Based BusinessesBusinesses

Project delivery time – Project delivery time – shortershorter

System quality – System quality – improvedimproved

Technology investment - Technology investment - optimizedoptimized

Need for ProcessNeed for Process

Domination of the different requirements calls for Domination of the different requirements calls for a systematic approacha systematic approach

Producing high-quality applications in a cost-Producing high-quality applications in a cost-effective wayeffective way

Goal – application should beGoal – application should be Maintainable and evolvableMaintainable and evolvable ReliableReliable EfficientEfficient Appropriate for UIAppropriate for UI Delivered on time at predictable cost Delivered on time at predictable cost

Or Simply Software Or Simply Software Engineering?Engineering?

“Fundamental differences [between hypermedia and other disciplines] however, make a pure transposition of techniques both difficult and inadequate. An important part of hypertext design concerns aesthetic and cognitive aspects that software engineering environments do not support.”

--Nanard & Nanard

Web Development vs. Web Development vs. Software DevelopmentSoftware Development

They are different due to the nature and They are different due to the nature and distinct requirements of Web-based distinct requirements of Web-based systemssystems

Even though Web-based systems often Even though Web-based systems often require programming and specialty require programming and specialty software, the development of that software software, the development of that software is often uniqueis often unique

Web-Based Systems vs. Web-Based Systems vs. Software (1/3)Software (1/3)

Web-based systems:Web-based systems: Are often document-oriented containing static or Are often document-oriented containing static or

dynamic contentdynamic content More emphasis on “look and feel”More emphasis on “look and feel” Are “content-driven” – process is driven by the Are “content-driven” – process is driven by the

availability of contentavailability of content Need to cater to users with diverse skills and Need to cater to users with diverse skills and

capabilitiescapabilities Are typically constrained to a short development Are typically constrained to a short development

time making it difficult to apply the same levels of time making it difficult to apply the same levels of formal planning and testing used in software formal planning and testing used in software developmentdevelopment

Web-Based Systems vs. Web-Based Systems vs. Software (2/3)Software (2/3)

Web-based systems:Web-based systems: Differ by means of their access and delivery Differ by means of their access and delivery

mediummedium Have different life spansHave different life spans Have differing development requirementsHave differing development requirements

Developers are vastly varied in terms of their Developers are vastly varied in terms of their background, skills, knowledge, and system background, skills, knowledge, and system understandingunderstanding

Developers differ in their perception of Web Developers differ in their perception of Web systemssystems

Web-Based Systems vs. Web-Based Systems vs. Software (3/3)Software (3/3)

Web-based systems:Web-based systems: Should be scalableShould be scalable Have varying performance requirements – Have varying performance requirements –

must be able to cope with uncertain, random must be able to cope with uncertain, random heavy demands on servicesheavy demands on services

Must be secureMust be secure Are subject to assorted legal, social, and Are subject to assorted legal, social, and

ethical scrutinyethical scrutiny

Categories of Web-Based Categories of Web-Based SystemsSystems

Category Examples

Informational Online newspapers, manuals

Interactive Registration forms

Transactional Electronic shopping

Workflow-oriented Status monitoring

Collaborative work Distributed authoring

Online communities Discussion groups

Web portals Shopping malls

Web services Enterprise applications

EngineeringEngineering

Discussing the term in the context of Discussing the term in the context of (software) engineering:(software) engineering:

““Engineering is about the systematic Engineering is about the systematic application of scientific knowledge in application of scientific knowledge in creating and building cost-effective creating and building cost-effective solutions to practical problems”, solutions to practical problems”,

Berry [Report No. CMU/SEI-92-TR-34]Berry [Report No. CMU/SEI-92-TR-34]

Why Engineering?Why Engineering?

Engineering is associated with “scale-up”Engineering is associated with “scale-up”

New issues arise when scaling by 2+ orders New issues arise when scaling by 2+ orders of magnitudeof magnitude

Skills of architects and all kinds of Skills of architects and all kinds of engineers are different from those needed engineers are different from those needed for designing and building simple structures for designing and building simple structures and simple systemsand simple systems

What is Web What is Web Engineering?Engineering?

There is still no common answerThere is still no common answer““Web Engineering is concerned with establishment and Web Engineering is concerned with establishment and use of sound scientific, engineering and management use of sound scientific, engineering and management principles and disciplined and systematic approaches to principles and disciplined and systematic approaches to the successful development, deployment and the successful development, deployment and maintenance of high quality Web-based systems and maintenance of high quality Web-based systems and applications”, SIGWEB Newsletterapplications”, SIGWEB Newsletter““Web Engineering is a discipline among disciplines, Web Engineering is a discipline among disciplines, cutting across computer science, information systems, cutting across computer science, information systems, and software engineering, as well as benefiting from and software engineering, as well as benefiting from several non-IT specializations”, IEEE Multimediaseveral non-IT specializations”, IEEE Multimedia““While Web Engineering adopts and encompasses many While Web Engineering adopts and encompasses many software engineering principles, it incorporates many software engineering principles, it incorporates many new approaches, methodologies, tools, techniques, and new approaches, methodologies, tools, techniques, and guidelines to meet the unique requirements of Web-guidelines to meet the unique requirements of Web-based systems. Developing Web-based systems is based systems. Developing Web-based systems is significantly different from traditional software significantly different from traditional software development and poses many additional challengesdevelopment and poses many additional challenges””, , IEEE MultiMediaIEEE MultiMedia

Web EngineeringWeb Engineering

The application of systematic, disciplined, and quantifiable approaches to the design, production, deployment, operation, maintenance and evolution of Web-based software products.

---Gaedke, 2000

Web Engineering (1/2)Web Engineering (1/2)

A holistic and proactive approach to Web A holistic and proactive approach to Web systems developmentsystems development

Offers systematic approaches and disciplined Offers systematic approaches and disciplined processes for developmentprocesses for development

Deals with the management of complexity and Deals with the management of complexity and diversity of Web developmentdiversity of Web development

Brings to Web-based system developmentBrings to Web-based system development ControlControl Risk minimizationRisk minimization Enhanced maintainability and qualityEnhanced maintainability and quality

Web Engineering (2/2)Web Engineering (2/2)

Other factorsOther factors Document orientationDocument orientation Navigational designNavigational design Changing technologyChanging technology Budget and time constraintsBudget and time constraints People and internal politicsPeople and internal politics Division between theory and practiceDivision between theory and practice Lack of understanding…Lack of understanding…

Goals of Web Goals of Web EngineeringEngineering

Develop high quality Web applicationsDevelop high quality Web applications EffectiveEffective EfficientEfficient Achieve desired applicationAchieve desired application

Maintain and evolveMaintain and evolve Plan for change – solution may change the Plan for change – solution may change the

problemproblem

Encourage the use of systematic, disciplined Encourage the use of systematic, disciplined and quantifiable approaches and process and quantifiable approaches and process modelsmodels

•Physical Layer•Internet Layer•Transport Layer•Performance

•Process•Design•Implementation•Test•Operation•Maintenance

•Design & StructureInformation Space

•Navigation•Visualization•Usability•Collaboration

•Data Design, ER,...•RDBMS•Query Languages•Strg.Devices: FS,...

Key Knowledge AreasKey Knowledge Areas

Web Engineering

NetworkEngineering

SoftwareEngineering

Hypermedia

InformationSystems

Others...

Web Engineering Web Engineering Activities (1/3)Activities (1/3)

Requirements specification and analysisRequirements specification and analysis

Web-based system analysis and designWeb-based system analysis and design

Web development methodologies and techniquesWeb development methodologies and techniques

Migration of legacy systems to Web environmentsMigration of legacy systems to Web environments

Web-based real-time applications developmentWeb-based real-time applications development

Web-based multimedia application developmentWeb-based multimedia application development

Testing, verification and validation techniques and toolsTesting, verification and validation techniques and tools

Quality assessment, control and assuranceQuality assessment, control and assurance

Web Engineering Web Engineering Activities (2/3)Activities (2/3)

Management of access to applications and privilegesManagement of access to applications and privileges

Configuration and project managementConfiguration and project management

““Web metrics” – metrics for estimation of development effortWeb metrics” – metrics for estimation of development effort

Performance specification and evaluationPerformance specification and evaluation

Update and maintenanceUpdate and maintenance

Development models, teams, and staffingDevelopment models, teams, and staffing

Human and cultural aspectsHuman and cultural aspects

User-centric developmentUser-centric development

Web Engineering Web Engineering Activities (3/3)Activities (3/3)

Graphics, animation, and streamingGraphics, animation, and streaming

Copyright, legal and social aspectsCopyright, legal and social aspects

Web Engineering is Web Engineering is MultidisciplinaryMultidisciplinary

Example – Web Project Management

Process Models and Web Process Models and Web Engineering (1/2)Engineering (1/2)

Code-and-fix model (ad hoc development)Code-and-fix model (ad hoc development)

The Waterfall modelThe Waterfall model

Prototyping modelPrototyping model

Evolutionary Development modelEvolutionary Development model

Spiral modelSpiral model

Rational Unified Process modelRational Unified Process model

Process Models and Web Process Models and Web Engineering (2/2)Engineering (2/2)

MSFv3 Process modelMSFv3 Process model

Agile ProcessesAgile Processes

Reuse-Oriented ApproachesReuse-Oriented Approaches

WebComposition Process modelWebComposition Process model

Agile Processes, XP, ScrumAgile Processes, XP, Scrum

And many, many more…And many, many more…

Approaches to Managing a Approaches to Managing a Project (1/2)Project (1/2)

From Wysocki & McGary

Approaches to Managing a Approaches to Managing a Project (2/2)Project (2/2)

From Wysocki & McGary

Thank You!

Questions? Comments?

bebo@slac.stanford.edu