Bypass Testing of Web Applications

Jeff Offutt, Ye Wu, Xiaochen Du and Hong HuangInformation and Software Engineering

George Mason UniversityFairfax, VA 22030, USA(+1) 703-993-1654 / 1651

{ofut,wuye,xdu,hhuang2}@ise.gmu.edu

Abstract

Web software applications are increasingly being de-ployed in sensitive situations. Web applications are usedto transmit, accept and store data that is personal, com-pany confidential and sensitive. Input validation testing(IVT) checks user inputs to ensure that they conform tothe program’s requirements, which is particularly im-portant for software that relies on user inputs, includ-ing Web applications. A common technique in Web ap-plications is to perform input validation on the clientwith scripting languages such as JavaScript. An insidi-ous problem with client-side input validation is that endusers can bypass this validation. Bypassing validationcan reveal faults in the software, and can also break thesecurity on Web applications, leading to unauthorizedaccess to data, system failures, invalid purchases andentry of bogus data. We are developing a strategy calledbypass testing to create IVT tests. This paper describesthe strategy, defines specific rules and adequacy criteriafor tests, describes a proof-of-concept automated tool,and presents initial empirical results from applying by-pass testing.

1 Introduction

The World Wide Web gives software developers anew way to deploy sophisticated, interactive programswith complex GUIs and large numbers of back-end soft-ware components that are integrated in novel and inter-esting ways. Web applications are constructed from het-erogeneous software components that interact with eachother and with users in novel ways. Web software com-ponents are distributed across multiple computers and

organizations, are often created and integrated dynami-cally, are written in diverse languages and run on diversehardware platforms, and must satisfy very high require-ments for reliability, availability and usability. Thesecharacteristics have the dual advantage of offering pow-erful new abilities and presenting new problems to soft-ware developers.

Analyzing, evaluating, maintaining and testing theseapplications present many new challenges for softwaredevelopers and researchers. Most Web applications arerun by users through a Web browser and use HTML tocreate graphical user interfaces. Users enter data andmake choices by manipulating HTML forms and press-ing submit buttons. Browsers send the data and choicesto the software on the server using HTTPrequests. Animportant point to note is that HTTP is a “stateless” pro-tocol, that is, each request is independent of previousrequests and, by default, the server software does notknow whether multiple requests come from the same ordifferent users.

The type of HTTPrequestdetermines how the user’sdata is packaged when sent to the server. AlthoughHTTP defines a number of request types, this paperonly considers GET and POST requests. GET requestspackage the data as parameters on the URL that arevisible in the URL window of most browsers (for exam-ple, http://www.buyit.com?name=george ).POST requests package the data in the data packets thatare sent to the user.

A common activity of Web applications is tovalidatethe users’ data. This is necessary to ensure that the soft-ware receives data that will not cause the software todo bad things such as crash, corrupt the program state,corrupt the data store on the server, or allow access tounauthorized users. This type of input validation is cru-cial for Web applications, which are heavily user inter-

active, often serve a very large user base, have very highquality requirements, and are always publicly accessible[14]. Because of the fundamental client-server nature ofWeb applications, input validation is done both on theclient and the server.

HTML pages (whether statichtml files or dynam-ically created) can include scripting programs that canrespond to user events and check various syntactic at-tributes of the inputs before sending the data to theserver. User events that JavaScript can respond to aredefined by the HTML document object model (DOM)and include mouse over events, form field focus events,form field changes, and button presses, among others.Client-side checking is used to check that required fieldsare filled in, inputs conform to certain restrictions oncharacteristics such as length, characters used, and sat-isfaction of syntactic patterns (such as email addresses).Client-side checking can be done as soon as a user eventis triggered or after the user clicks on a submit buttonbut before the data is submitted. Doing input validationon the client avoids the need for a trip to the server andallows the checking to be defined within the input form.

Server side checking is done by programs on theserver such as CGI/Perl, Java servlets, Java ServerPages, and Active Server Pages. Server side checkingcan perform all of the checks that client-side checkingcan, but not until after the user presses the submit but-ton. Server side checking cannot respond to user events,but has access to the state of the file system and databaseon the server. When a high level language such as Javais used on the server, server side checking also providesmore robust and flexible ways to check inputs and re-spond to invalid user inputs than does a scripting un-typed language such as JavaScript.

1.1 Running Web Application Tests ThroughHTML Forms

HTML forms expect users to type their values andmake their choices by using the keyboard and mouse.However, it turns out to be easy for users to bypass theHTML to send values directly to the server software. Forexample, if the GET request is expected, the users cansimply type the parameters into the URL box in theirbrowsers. If the POST request is expected, a simple pro-gram can be written on the client that creates and sub-mits the request. There are two reasons for bypassingHTML forms. One is for convenience, if a Web appli-cation is used a lot it might be more convenient to skipthe relatively slow FORM interface. Another reason isfor automation. When running multiple tests on a Web

application, the test execution can be automated by by-passing the forms.

This ability to bypass form entry allows another strat-egy to be used. If the Web application uses client-sideinput validation, then the userscan bypass the valida-tion. This technique is sometimes used by hackers. Oursuggestion in this research is to utilize the ability to by-pass client-side checking to create tests, thereby supply-ing invalid inputs to the software to test for robustnessand security.

An additional ability that is available when bypassingHTML forms is to override hidden form fields. HTMLallows data to be placed into a page with the tag “<IN-PUT Type="Hidden" ...> ”. These fields are notshown to the users in the browsers, but data in the fieldsare submitted to the server. Bypassing forms allow theadditional ability to change or remove the contents ofhidden form fields.

One of the most common ways to violate data secu-rity is through “SQL injection.” Many Web applicationsuse client-supplied data in SQL queries. However, if theapplication does not strip potentially harmful characters,users can add SQL statements into their inputs. This iscalled SQL injection, and Anley [2] claims that despitebeing simple to protect against, many production sys-tems connected to the Internet are vulnerable. SQL in-jection vulnerability occurs when an attacker inserts anSQL statement into a query by manipulating data inputs.

2 Types of Client-side Validation

Input validation can check both the syntax and the se-mantics of inputs. Client-side input validation can bedone by using the HTML input boxes to restrict thesize or contents of inputs (syntactic restrictions only), bywriting programs such as JavaScripts to evaluate the val-ues before submission (syntactic and semantic restric-tions).

2.1 Semantic Input Validation

In an initial attempt at categorization, we have identi-fied three types ofsemanticdata input validation.

1. Data type conversion. Most inputs to HTML formelements are plain strings that may be converted to othertypes on the server. The client can check whether thestring can be converted correctly. For example, if theinput is an integer, the client can check to ensure that allcharacters are numeric digits.

2. Data format validation. There are many more re-strictive constraints on inputs that can be checked, and

2

this is one of the most common ways to validate input onthe Web. This includes checking the format of money,phone numbers, personal identification numbers, emailaddress, and URLs.

3. Inter-value constraint validation. There are oftenconstraint relationships among input values. For exam-ple, when providing payment information, a check pay-ment should include a bank routing number and a bankaccount, whereas a credit card payment should include acredit card number and an expiration date. The combina-tion of a bank routing number and a credit card numbershould not be allowed.

2.2 Syntactic Input Validation

HTML can also be used to impose several types ofsyntacticrestrictions, all of which can be avoided by by-pass testing:

1. Built-in length restriction. Text boxes can include a“maxlength” attribute to restrict the length of text inputs.In the following text input box, only three characters willbe accepted:

<INPUT Type=text Name=Age Maxlength=3>

2. Built-in input value restriction. HTML can use se-lect boxes, check boxes, and radio boxes to restrict theuser to a certain pre-defined set of inputs.

3. Built-in input transfer mode. HTML forms definethe type of request (GET or POST). Because of the dif-ferences in these requests, this is effectively a way torestrict the user’s input. HTML links always generate aGET request.

4. Built-in data access. Web browsers manage twotypes of data, cookies and hidden form fields. Hiddenform fields can be viewed if the users look at the source,but are normally not shown. Cookies are automaticallymanaged by the browsers and server software and aresent to the server automatically. Cookies can also beviewed in most browsers (for example, in Mozilla by“Tools-Cookie Manager-Manage Stored Cookies”). Amajor difference is that cookies persist across multiplerequests, whereas hidden form fields are transient dataitems that only appear in individual HTML pages.

5. Built-in input field selection. An HTML form hasa pre-defined set of form fields that users can select val-ues for. Other values are normally not allowed, andclient-side scripting can also disable certain input fieldsby making them unavailable or hidden.

6. Built-in control flow restriction. HTML pages al-low the user to transfer to a certain, fixed set of URLs.These are defined byAction attributes inFORMtagsand by HTML links.

Illegal Character SymbolEmpty StringCommas ,Directory paths .. ../Strings starting with for-ward slash

/

Strings starting with aperiod

.

Ampersands &Control character NIL, newlineCharacters with high bitset

decimal 254 and 255

XML tag characters <, >

Table 1. Characters that sometimes causeproblems for Web applications

2.3 Generalizing to Input Validation

Some of the vulnerabilities (both on client and server)are due to the server not checking input and data fromthe client; but it would be a mistake to assume checkingdata is all that is necessary. By considering penetrationtechniques such as SQL injection, cross-site scripting,buffer overflow, embedded script attack, and shell es-cape vulnerabilities, Wheeler [18] gives a general solu-tion to user input validation from a security perspective.Any input accepted from user must be validated and anyillegal input data should be filtered out. Here are somegeneral rules that should be considered.

1. Filters: Set up filters in the Web application to pre-vent illegal characters from reaching the server’s datastore. Table 1 lists some specific characters that can beproblems for Web applications.

2. Numeric limits: Limit all numbers to the minimum(often zero) and maximum allowed values.

3. Email addresses: A full email address checkershould be enforced. Afull email includes a usernameand valid domain name. A complete email check shouldalso ensure that the email contains all expected informa-tion, including subject, and recipient addresses.

4. URLs: URLs (and more generally, URIs) shouldbe checked to ensure that they have a valid form and thedestination exists.

5. Character patterns: When possible, legal characterpatterns need to be identified. They can often be ex-pressed as regular expressions. Inputs that do not matchthe pattern should be rejected.

3

2.4 Feasibility Study: CyberChair

As an initial feasibility study, we applied some of thebypass testing techniques to CyberChair, a Web-basedpaper submission and reviewing system [17] that is usedby a number of conferences, including ISSRE. It hasbeen in use since 1996, and was opened as free soft-ware for downloading in 2000. The CyberChair web site(www.cyberchair.org) listed 242 users in April 2004.

CyberChair has multiple phases to support a confer-ence. Authors submit abstracts in the first phase, andthen full papers in the second. We manually tested thesubmission page of the second phase. Tests were per-formed on the ISSRE 2004 conference server. We didnot have access to the source code and did not down-load CyberChair. We started with a user id and accesscode from an abstract submission in the first phase. Af-ter logging in, CyberChair returns an HTML page witha form to submit papers. To implement bypass testing,we saved the page and then modified it.

In this early feasibility study, our test creation pro-cess was not formalized. We broke the inputs into threelevels; the control flow level, parameter level, and valuelevel. At the control flow level, we attempted to submita paper without logging in. At the parameter level, weremoved some parameters from the form and then sub-mitted. At the value level, we tried various values forparameters, including values that are normally not ex-pected by the software. This process revealed five typesof faults, all of which are potential security holes.

1. Submission without authentication: After correctlylogging in to CyberChair, a submission form is returned.We decided to attempt to use that form to submit withouta valid login. We saved the page locally, and changed theAction attribute on theFORMtag from a relative URLto a complete URL. (A relative URL does not includea domain name and only works within a single session.)Then we copied the modified form to a second computer,and used it to submit a file. The submission was allowed,implying that the semantics of a login is to send the sub-mission page,not to only allow authenticated users tosubmit. Whereas we used a valid login to find the sub-mission page, it would not be difficult for someone tofind or guess a valid URL for the submission, particu-larly since CyberChair is an open-source program.

2. Unsafe use of hidden field: The submission pageuses a hidden field to track the user. We customized thesubmission form by changing the value of the hiddenform field and were still able to submit the paper. Thisallows the possibility of overwriting another user’s sub-mission.

3. Disclosing information: We also tried removingthe hidden field and setting its value to empty. In thesecases, the software failed and returned messages that in-dicated in which file and which line of code the programfailed. This kind of information is confusing to validusers and potentially unsafe to show to malicious hack-ers.

4. No validation for parameter constraint: The serverdoes not check if the selected file type and the file sub-mitted really match. For example, it is possible to selectthe file type to be pdf, but submit an rtf file instead. Thislack of constraint checking can damage the state on theserver.

5. No data type or data value validation: CyberChairasks the user to submit the number of pages, whichshould be an integer value between 1 and some fairlysmall number such as 10 or 15. We tried submitting non-integer values, negative numbers, and extremely largenumbers, none of which were detected by the software.Similar problems are also found in other fields.

Although this experience is anectodal, and our pro-cess was fairly ad-hoc, it does demonstrate that bypasstesting can be effective on software that has reasonablywide use. The rest of this paper provides a first attemptto formalize these ideas.

3 Modeling HTML Input Units

Web application include static HTML files and pro-grams that dynamically generate HTML pages. HTMLpages can include more than one form, and each formcan include many input fields. For example, we iden-tified 169 HTML hyperlinks and 20 forms on ama-zon.com’s home page. This makes automatic input val-idation difficult to manage by hand, thus we take a firststep toward automation by constructing a formal modelfor HTML client-server inputs.

Each HTML page, whether a static file or dynami-cally generated, can have zero or more HTML links andforms that let users interact with the server. Aninputunit IU = (S, D, T ) is the basic element of interac-tion on the client side. The inputs are sent to a softwarecomponent on a server,S, and includes a set of inputelementsD. D is a set of ordered pairs,(n, v), wheren is a name (parameter) andv represents the set of val-ues that can be assigned ton. The set of values may beunlimited, as in a text box, or finite, as with a selectioninput. It is sometimes convenient to think of these sets ofvalues as defining atype. T is the HTTP transfer mode(GET, POST, etc.).

There are two types of input units. Aform input unit

4

is an HTML form that specifies the server componentas theAction attribute within theForm tag, and theinput data corresponds to all the input fields within theform. The transfer mode is specified within theMethodattribute of theForm tag.

A link input unit is an HTML link in an <A> tag.A link input unit’s server target can either be a staticHTML file or a server program such as a servlet, andthe server target is specified as theHREFattribute of the<A> tag. Link input units always generate GET requestsand only have input elements when URL rewriting isused. In the HTML link<A HREF="prog?val=1" ,S is prog andD is {(val, 1)}.

As an example, consider the screen shot of STIS inFigure 1. STIS helps users keep track of arbitrary textualinformation. The main part of the screen in Figure 1contains two form input units and 12 link input units,plus the menu bars on the top and the bottom have 5more link input units apiece. Key portions of the HTMLfor the search form input units and the delete link inputunits are shown in the callout bubbles.

3.1 Composing Input Units

Some Web pages may have large numbers of inputs,which can be difficult to manage. For example, it is com-mon to have identical forms for things like searching andlogging in. It is easy to eliminated this redundancy instatic HTML pages, but harder for dynamically gener-ated pages. Because the number of potential unique dy-namically generated pages is arbitrary, and which pagesare generated depends on inputs and processing on theserver, the problem of identifying all input units from aclient without access to the program source is undecid-able.

When possible, the following composition rules areused to eliminate redundancy. To simplify the dis-cussion, the following definitions assume two inputunits, each of which contains only one parameter:{iu1 = (S1, D1, T1), iu2 = (S2, D2, T2)}, andD1 = {(n1, v1)} andD2 = {(n2, v2)}. All threecomposition rules require the two input units to have thesame server component.

1. Identical input units composition. Two input unitsiu1 = (S1, D1, T1) andiu2 = (S2, D2, T2) are iden-tical iff S1 = S2, D1 = D2 andT1 = T2. The two unitsare merged into a new input unitiu = (S1, D1, T1).For example, it is common for a Web page to have thesame search form in two different places on the page.

2. Optional input element composition.Two inputunits iu1 = (S1, D1, T1) and iu2 = (S2, D2, T2)

have optional elements ifS1 = S2, T1 = T2 and oneinput unit has an input element name that is not in theother. That is, there exists(n1, v1) ∈ D1 such that thereis nov2 where(n1, v2) 6∈ D2 or (n2, v2) ∈ D2 suchthat there is nov1 where(n2, v1) 6∈ D1. The two in-put units are merged, formingiu = (S1, D

′, T1) where

D′

= {D1 ∪ D2}. This happens when a dymanicallygenerated page sometimes includes different input ele-ments, for example, if an order entry form sometimesincludes an input box to enter discount coupon code.

3. Optional input value composition.Two input unitsiu1 = (S1, D1, T1) and iu2 = (S2, D2, T2) haveoptional input values ifS1 = S2, T1 = T2 and thereexists(n1, v1) ∈ D1 and (n2, v2) ∈ D2, such thatn1 = n2 but v1 6= v2. Then the two input units aremerged, formingiu = (S1, D

′, T1) where D

′=

{D1− (n1, v1)}∪{D1− (n2, v2)}∪{(n1, (v1∪v2)}.This happens when a dymanically generated page some-times includes different input values, for example, in anonline grade entry form at our university, undergradu-ate courses have fewer choices for grades than graduatecourses do.

The two search forms at the top and the bottom of thescreen in Figure 1 are identical and are thus composed.The two search forms with buttons “Search” and “AllRecords” use the same server component, but have dif-ferent input elements, thus can be merged under the op-tional input element composition rule. Finally, the three“delete” link input units all reference the same servercomponent, so can be merged under the optional inputvalue composition rule.

4 Bypass Testing

Most input validation focuses on individual parame-ters. This works well for traditional software, where thepatterns of interaction between users and software arefixed and cannot be altered by the users. An interest-ing complexity is that the use of dynamic Web pagesmeans that the same URL can produce different formsat different times, depending on the parameters supplied,state on the server, characteristics of the client, and otherenvironmental information. Additionally, users of Webapplications can not only change the values of input pa-rameters, but can also change the number of input pa-rameters and the control flow. This makes it easier toviolate constraints among different parameters and be-tween software components. This section describes asystematic approach to identify constraints among inputparameters. Then rules are given to generate bypass testcases to test the Web application to ensure these con-

5

<form action="update_search_params.jsp" method=POST> <select name="infoCategory"> <option value="[*ALL*]">[All Records]</option> <option value="">[Uncategorized Records]</option> </select> Search:<input type="text" name="search" size="30"> <input type="checkbox" name="sname" value="name"> Name <input type="checkbox" name="content" value="content">Con… <input type="submit" value="Search"> </form>

<tr> <td> 1 </td> <td><a href="delete_record.jsp?rec_name=Downloads& rec_category=Computer">delete</a></td> …. </tr>

<tr> <td> 2 </td> <td><a href="delete_record.jsp?rec_name=JBuilder& rec_category=Computer">delete</a></td> …. </tr>

<form action="update_search_params.jsp" method=POST> <input type="submit" value="All records"> </form>

Figure 1. STIS initial screen

straints are adequately evaluated. According to the clas-sification of input validation types from Section 2 2, ourbypassing testing will be conducted at three levels, asdiscussed in the following subsections.

4.1 Value Level Bypass Testing

This type of bypass testing tries to verify whether aWeb application adequately evaluates invalid inputs. Itaddresses data type conversion, data value validation,and built-in input value restriction. This testing is basedon the restrictions described in Section 2. Given a singleinput variable, invalid inputs can be generated accord-ing to the 14 types of input validation that are specifiedin Section 2:

• Data type and value modification.HTML inputsare initially strings, but they are often converted toother data types on the server. Data type conver-sion testing uses values of different types to eval-uate the server-side processing, including generalstrings, integers, real numbers, and dates.

• HTML built-in length violation.The HTML tagin-puthas an attributemaxlength, as described in Sec-tion 2. Invalid values are generated to violate theserestrictions.

• HTML built-in value violation. Pre-defined inputrestrictions from HTML select, check and radioboxes are violated by modifying the submission tosubmit values that are not in the pre-defined set.

• Special input value. When data is stored intoa database or XML document, and under certainkinds of processing, some special characters, as de-fined in Table 1, can corrupt the data or cause thesoftware to fail. This data is often validated withclient-side checking, but sometimes with server-side checking. Thus, following Wheeler’s sugges-tions [18], values for text fields are generated withspecial characters such a commas, directory paths,slashes and ampersands.

4.2 Parameter Level Bypass Testing

This type of bypass testing tries to address issues re-lated to built-in input parameter selection, built-in dataaccess and inter-value constraints.

It is relatively easy to enumerate possible invalid in-puts for an input parameter. However, the restrictive re-lationships among different parameters are hard to iden-tify, hard to validate and are thus often ignored duringtesting. There are many kinds of relationships. One type

6

is invalid pair, where two parameters cannot both havevalues at the same time. For example, it is not reason-able to have a checking account number and a credit cardexpiration date in the same transaction. Another type isrequired pair, where if one parameter has a value, theother must also have a value. For example, if we havea credit card number, we must also have an expirationdate. Parameter level bypass testingtries to test Webapplication by executing test cases that violate restric-tive relationships among multiple parameters.

These relationships are very often difficult to obtainstatically and must be identified dynamically. They aresometimes described in English-language instructions,and sometimes simply assumed. Nevertheless, if we canidentify and follow all possible ways to send parame-ters to a server program, we can ensure conformance tothe restrictive relationships, and then find values to vi-olate the restrictive relationships. Thus, we define theinput patternto be the set of parameters that are sentto a server. If an optional input element compositionhas been applied to createiu, the input patterns iniuwill correspond to the parameters from the original in-put units.

The following algorithm is designed to derive all pos-sible input patterns in a Web application. As with find-ing input units, this is generally an undecidable prob-lem without access to the server program source. Thus,this algorithm creates an approximation that is limitedby the data that is supplied to existing forms in Step 2.The input patterns created by the algorithm are used togenerate parameter level bypass tests.

Algorithm: Identify input patterns of web applicationsInput: The start page of a web application,SOutput: Identifiable input patterns

Step 1 : Create a stackSTto retain all input units thatneed to be explored. InitializeSTto S. Create a setIUS to retain all input units that have been identi-fied. InitializeIUS to empty.

Step 2 : While ST is not empty, pop aninput unit (de-fined in Section 3) fromST, generate data for theinput unit and send it to the server. When a replyis returned, analyze the HTML content. For eachinput unit iu:

• if iu is a link input unit, andiu does not be-long to a different server, donot pushiu ontothe stack.

• if iu ∈ IUS (it has already been found), donot pushiu onto the stack.

• if there exists an input unitiu′ ∈ IUS such

that iu andiu′

have optional input elements,update the possible value ofiu. Do not pushiu onto the stack.

• Otherwise, a new input pattern has been iden-tified; addiu to IUSas an optional input unit,and then pushiu ontoST.

After executing the algorithm, we have acollection of input units IU = (S, D, T ),where D = {P1, P2, ..., Pk} and Pi ={(ni

1, vi1), (ni

2, vi2), ..., (ni

a, via)}. Each Pi is a valid

input pattern for the input unitIU . Based on theinput patterns, we generate three types of invalid inputpatterns to test the restrictive relationships amongparameters.

• The empty input patternsubmits no data to theserver component. Formally,IU1 = (S, φ, T ).The empty input pattern will violate allrequiredpair restrictive relationships.

• The universal input patternsubmits values for allparameters that the server component knows about.Formally,IU2 = (S, P1

⋃P2

⋃...

⋃Pk, T ). The

universal input pattern will violate allinvalid pairrestrictive relationships.

• The differential input patternsubmits appropriatevalues for all parameters in one input pattern, plusa value forone parameter that isnot in that inputpattern. For each pair of input patternsPi andPj ,generate an invalid input pattern in the followingway.x is a parameter fromPj−Pi, chosen arbitrar-ily. IU3 = (S, P

′, T ), whereP

′= Pi

⋃{x}. Theintent of the differential input pattern is to makesubtle changes that are not likely to be identified bychecks other than invalid input checking.

Parameter level bypass testing focuses on relation-ships among different parameters, therefore, all valuesof input parameters are selected from a set of valid val-ues.

4.3 Control Flow Level Bypass Testing

The previous two types of bypass testing assume usersfollow the control flow that is defined by the software.This is a safe assumption for traditional software appli-cations. However, users of Web applications can alterthe control flow by pressing the back button, pressingthe refresh button, or by directly entering a URL into a

7

browser. This ability adds uncertainty and threatens thereliability of Web applications.

Control flow level bypass testingtries to verify Webapplications by executing test cases that break the nor-mal execution sequence. As a first step, the “normal”control flow must be identified. The algorithm for find-ing input patterns in the previous section provides theneeded information. The input units that were identi-fied can be used to define all normal control flows fromthat unit. So we expand the algorithm to derive all nor-mal control flow for all input units. In the algorithm,an input unitiu is popped form the stack, data is sup-plied, then sent to the server. All the input units that arereturned from that submission are considered to be can-didates for the next step in the control flow. Given that,control flow bypass testing is carried out according tothe following two categories:

1. Backward and forward control flow alteration .Given a normal control flowiu1, iu2, ..., iuk, eachpair of input units (iui, iui+1) forms a transition.Model use of the back button by changing eachtransition (iui, iui+1) to (iui, iui−1). Model useof the forward button by changing each transition(iui, iui+1) to (iui, iui+2).

2. Arbitrary control flow alteration . Given a nor-mal control flow iu1, iu2, ..., iuk, for each inputunit iui, 1 ≤ i ≤ k, change the controliui to somearbitraryiut, such att 6= i.

4.4 Summary of Bypass Testing

The three levels of testing in this section, value level,parameter level, and control flow level, can be used in-dividually or combined together. Parameter level andcontrol flow level bypass testing focus on interactionsamong different parameters and different server compo-nents, thus can be run independently of value level by-pass testing.

5 Empirical Validation

As an initial validation, we applied bypass testingto a small but non-trivial web application, the SmallText Information System (STIS). STIS helps users keeptrack of arbitrary textual information. It stores all in-formation in a database (currently mysql) and is com-prised of 17 Java Server Pages and 5 Java bean classes.Eight of the JSPs process parameterized requests,lo-gin.jsp, browse.jsp, record edit.jsp, record delete.jsp,

record insert.jsp, categories.jsp, categoryedit.jsp andregister save.jsp. We extensively tested these eight JSPswith bypass testing.

When a Web application receives invalid inputs, thereare three possible types of server responses. (1) Theinvalid inputsare recognizedby the server and ade-quately processed by the server. (2) The invalid inputsare not recognizedand cause abnormal server behav-ior, but the abnormal behavior is caught and automati-cally processed by server error handling mechanism. (3)The invalid inputsare not recognizedand the abnormalserver behavior isexposeddirectly to the users. Abnor-mal server behavior includes responses like run time ex-ceptions and revealing confidential information to unau-thorized clients. A type 1 response represents properserver behavior, while type 2 and 3 responses representinadequate server behavior and are considered to be fail-ures.

Some inputs had to be created by hand for bypasstesting, including user names and passwords (STIS hastwo levels of access) and some very long invalid inputstrings. Other inputs were either automatically extractedfrom the HTML files or randomly generated. For com-parison, we generated four levels of tests, for just thevalue level, the parameter level but not control, the con-trol level but not parameter, and both the parameter andcontrol level.

Table 2 summarizes the results. For each group oftests, the number of tests (T) and the number of teststhat caused a failure (F) are shown. There were a totalof 158 tests, 66 of which caused failures. Of these 158tests, none of the parameter level or control level testscould be executed without bypass testing, and only 55of the value level tests could be executed without bypasstesting. These 55 tests only caused 9 failures.

6 Related Work

The bypass testing techniques are motivated bya combination of input validation and the category-partition method [15], a multi-step method to derive testframes and tests from specifications. The rest of thissection discusses the most closely related test ideas, in-put validation testing and testing of graphical user inter-faces.

6.1 Input Validation Testing

Input validation analysis and testing involves stati-cally analyzing the input command syntax as defined ininterface and requirement specifications and then gener-

8

Table 2. Failures found for each dynamic componentI: Value Level, No Parameter or ControlII: Parameter Level, No Control LevelIII: Control Level, No Parameter LevelIV: Parameter Level and Control Level

T = number of tests, F = number of failuresComponent I II III IV Total

T F T F T F T F T F

login 15 0 2 2 n/a n/a 17 2browse 7 4 1 0 1 1 1 1 10 6recordedit 17 9 5 2 1 1 5 5 28 17recorddelete 5 0 2 0 1 1 2 2 10 3recordinsert 13 9 3 1 1 1 3 3 20 14categories 12 2 2 0 1 0 2 0 17 2categoryedit 13 2 2 0 1 0 2 0 18 2registersave 25 11 6 3 1 0 6 6 38 19Total (#tests & #failures) 107 37 23 8 7 4 21 17 158 66

ating input data from the specification. Hayes and Of-futt [6] proposed techniques for input validation analy-sis and testing for systems that take inputs that can berepresented in grammars. Both IVT and bypass testingattempt to violate input specifications, so bypass testingcould be viewed as a special kind of IVT that addressesconcerns of Web applications.

6.2 GUI Testing

HTML forms can be considered to offer a graphicaluser interface to run software that is deployed across theWeb. Memon has developed techniques to test softwarethrough their GUIs by creating inputs that match theinput specifications of the software [13, 12]. This ap-proach focuses on the layout of graphical elements andthe user’s interaction when supplying form data. Bypasstesting relies on following the syntax of the GUI forms,but specifically finds ways to violate constraints imposedby the syntax. The two approaches are complementary,specifically, GUI testing could be used to develop valuesfor bypass testing.

6.3 Web Application Testing

Most research in testing Web applications has focusedon client-side validation and static server-side validationof links. An extensive listing of existing Web test sup-port tools is on a Web site maintained by Hower [7]. Thelist includes link checking tools, HTML validators, cap-ture/playback tools, security test tools, and load and per-

formance stress tools. These are all static validation andmeasurement tools, none of which support functionaltesting or black box testing.

The Web Modeling Language (WebML) [4] allowsWeb sites to be conceptually described. The focus ofWebML is primarily from the user’s view and the datamodeling. Our model derived from the software is com-plementary to the solutions proposed by WebML.

More recent research has looked into testing soft-ware from a static view, but few researchers have ad-dressed the problem of dynamic integration. Kung et al.[9, 11] have developed a model to represent Web sitesas a graph, and provide preliminary definitions for de-veloping tests based on the graph in terms of Web pagetraversals. Their model includes static link transitionsand focuses on the client side without limited use of theserver software. They defineintra-objecttesting, wheretest paths are selected for the variables that have def-usechains within the object,inter-objecttesting, where testpaths are selected for variables that have def-use chainsacross objects, andinter-client testing, where tests arederived from a reachability graph that is related to thedata interactions among clients.

Ricca and Tonella [16] proposed an analysis modeland corresponding testing strategies forstaticWeb pageanalysis. As Web technologies have developed, moreand more Web applications are being built on dynamiccontent, and therefore strategies are needed to modelthese dynamic behaviors.

Benedikt, Freire and Godefroid [3] presented Veri-

9

Web, a navigation testing tool for Web applications.VeriWeb explores sequences of links in Web appli-cations by nondeterministically exploring “action se-quences”, starting from a given URL. Excessively longsequences of links are limited by pruning paths in aderivative form of prime path coverage. VeriWeb createsdata for form fields by choosing from a set of name-value pairs that are initialized by the tester. VeriWeb’stesting is based on graphs where nodes are Web pagesand edges are explicit HTML links, and the size of thegraphs is controlled by a pruning process. This is sim-ilar to our algorithm, but does not handle dynamicallygenerated HTML pages.

Elbaum, Karre and Rothermel [5] proposed a methodto use what they called “user session data” to gener-ate test cases for Web applications. Their use of theterm user session data was nonstandard for Web appli-cation developers. Instead of looking at the data kept inJ2EE servlet session, their definition of user session datawas input data collected and remembered from previoususer sessions. The user data was captured from HTMLforms and included name-value pairs. Experimental re-sults from comparing their method with existing meth-ods show that user session data can help produce effec-tive test suites with very little expense.

Lee and Offutt [10] describe a system that generatestest cases using a form of mutation analysis. It focuseson validating the reliability of data interactions amongWeb-based software system components. Specifically,it considers XML based component interactions.

Jia and Liu [8] propose an approach for formally de-scribing tests for Web applications using XML. A proto-type tool, WebTest, based on this approach was also de-veloped. Their XML approach could be combined withthe test criteria proposed in this paper to express the testsin XML.

Andrews et al. use hierarchical FSMs to model poten-tially large Web applications. Test sequences are gener-ated based on FSMs and use input constraints to reducethe state space explosion [1]. Finally, our previous workon modeling of Web applications has led to the develop-ment ofatomic sections, which can be used to model dy-namic aspects of Web applications [19]. This approachis at the detailed analysis level and relies on access tothe code, unlike bypass testing.

7 Conclusions

This paper has presented four results. First, the con-cept ofbypass testingwas introduced to submit valuesto Web applications that are not validated by client-side

checking. Bypass testing requires a detailed model forhow to introduce inputs to server-side software compo-nents, thus we developed one. Third, this model sup-ports more general input validation testing, and rules aredefined for bypass and input validation. Finally, empiri-cal results from an open-source conference managementsystem and our own laboratory-built Web applicationwere shown.

Bypass testing is a unique and novel way to createtest cases that is available only because of the unusualmix of client-server, HTML GUI, and JavaScript tech-nologies that are used in Web applications. It is alsodeceptively complicated. Although the concept is rel-atively simple, to submit inputs that violate client-sideconstraints, the distributed and heterogeneous nature ofWeb applications brings in many complexities. Not sur-prisingly, the most complicated part is handling inputsto dynamically generated HTML forms. The algorithmpresented in Section 4.2 is a first attempt to approximatethe kinds of input forms that can be generated dynami-cally.

The existence of bypass testing may motivate Web ap-plication developers to check data on the server, obviat-ing much of the need for bypass testing. This may al-ready be a trend in the industry. Five years ago, manybooks on Web software advocated checking inputs withJavaScript as a mechanism to reduce network traffic;modern books and instructors usually advocate doinginput validation on the server. Nevertheless, major e-commerce and e-service sites still use client-side check-ing and hidden form fields. We found client-side check-ing onamazon.com andnetflix.com , and the useof hidden form fields to store sensitive information onfastlane.nsf.com . The long history of buffer-overflow problems leads us to be somewhat pessimisticthat developers will develop software well enough tomake bypass testing completely obsolete.

A major advantage of bypass testing is that it doesnot require access to the source of the back-end soft-ware. This greatly simplifies the generation of tests andautomated tools, and we expect bypass tests can be gen-erated automatically. Our current plan is to build toolsthat parse HTML, discover and analyze the form fieldelements, parse the client-side checking encoded in theJavaScript, and automatically generate bypass tests toevaluate the server-side software.

References

[1] Anneliese Andrews, Jeff Offutt, and RogerAlexander. Testing Web applications.Software

10

and Systems Modeling, 2004. Revision submitted.

[2] Chris Anley. Advanced SQL injection inSQL server applications. online, 2004.http://www.nextgenss.com/papers/advancedsql injection.pdf, last access February 2004.

[3] Michael Benedikt, Juliana Freire, and PatriceGodefroid. Veriweb: Automatically testing dy-namic Web sites. InProceedings of 11th Interna-tional World Wide Web Conference (WW W’2002),Honolulu, HI, May 2002.

[4] Stefano Ceri, Piero Fraternali, and Aldo Bongio.Web modeling language (WebML): A modelinglanguage for designing Web sites. InNinth WorldWide Web Conference, Amsterdam, Netherlands,May 2000.

[5] Sebastian Elbaum, Srikanth Karre, and GreggRothermel. Improving Web application testingwith user session data. InProceedings of the 25thInternational Conference on Software Engineer-ing, pages 49–59, Portland, Oregon, May 2003.IEEE Computer Society Press.

[6] J. H. Hayes and J. Offutt. Increased software re-liability through input validation analysis and test-ing. In Proceedings of the 10th International Sym-posium on Software Reliability Engineering, pages199–209, Boca Raton, FL, November 1999. IEEEComputer Society Press.

[7] Rick Hower. Web site test toolsand site management tools, 2002.www.softwareqatest.com/qatweb1.html.

[8] Xiaoping Jia and Hongming Liu. Rigorous andautomatic testing of Web applications. In6thIASTED International Conference on Software En-gineering and Applications (SEA 2002), pages280–285, Cambridge, MA, November 2002.

[9] D. Kung, C. H. Liu, and P. Hsia. An object-oriented Web test model for testing Web appli-cations. InProc. of IEEE 24th Annual Interna-tional Computer Software and Applications Con-ference (COMPSAC2000), pages 537–542, Taipei,Taiwan, October 2000.

[10] Suet Chun Lee and Jeff Offutt. Generating testcases for XML-based Web component interactionsusing mutation analysis. InProceedings of the 12thInternational Symposium on Software Reliability

Engineering, pages 200–209, Hong Kong China,November 2001. IEEE Computer Society Press.

[11] C. H. Liu, D. Kung, P. Hsia, and C. T. Hsu. Struc-tural testing of Web applications. InProceedingsof the 11th International Symposium on SoftwareReliability Engineering, pages 84–96, San JoseCA, October 2000. IEEE Computer Society Press.

[12] A. M. Memon, M. L. Soffa, and M. E. Pollack.Hierarchical GUI test case generation using auto-mated planning.IEEE Transactions on SoftwareEngineering, 27(2):144–155, February 2001.

[13] Atif M. Memon. GUI testing: Pitfalls and process.IEEE Computer, 35(8):90–91, August 2002.

[14] Jeff Offutt. Quality attributes of Web software ap-plications. IEEE Software: Special Issue on Soft-ware Engineering of Internet Software, 19(2):25–32, March/April 2002.

[15] T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generatingfunctional tests. Communications of the ACM,31(6):676–686, June 1988.

[16] F. Ricca and P. Tonella. Analysis and testing ofweb applications. In23rd International Confer-ence on Software Engineering (ICSE ‘01), pages25–34, Toronto, CA, May 2001.

[17] Richard van de Stadt. Cyberchair: A free web-based paper submission and reviewing system. on-line, 2004. http://www.cyberchair.org/, last accessApril 2004.

[18] David A. Wheeler. Secure Programming forLinux and Unix HOWTO. Published online,March 2003. http://www.dwheeler.com/secure-programs/, last access Feb 2004.

[19] Ye Wu, Jeff Offutt, and Xiaochen Du. Mod-eling and testing of dynamic aspects ofWeb applications. Submitted for publica-tion, 2004. Technical Report ISE-TR-04-01,www.ise.gmu.edu/techreps/.

11