Application and Website Security – Designer Edition:

Using Formal Specification Techniques To Discover Design Flaws

Daniel OwensIT Security Professional

INTRODUCTION

Introduction

• Systems generally come in two forms – Single Level Security Systems (SLSS) and Multilevel Security Systems (MLSS)– In an SLSS, the system processes information of the same

sensitivity or privilege level, ignoring the user’s need-to-know and clearance, but still enforce access controls

– In an MLSS, the system must take into account sensitivity levels, privileges, need-to-know, clearance levels, and enforce access controls

• This work is generalized enough to apply to both

Previous Works

• Many previous works have looked for specific flaws in specific systems– The works were never intended to be general,

simply to provide analysis of a specific flaw in a specific system

– This means that much of the works available are difficult to apply to current or more general problems

• Additionally, most works ignore that functional and security flaws stem from the same issues

Common Root Causes

• Most flaws only become problems after the flaw or data forged by the flaw has successfully passed through multiple checkpoints– Any of the checkpoints could have prevented the flaw– There are instances, such as faulty subsystems upon

which the current software relies• Even these can be tested for, but that becomes a never-

ending and very costly thing to do, so we assume that those we have no choice but to rely on have been heavily tested

• There is no such thing as a strictly ‘security’ or strictly functionality flaw

Common Root Causes (cont)

• In 1996, Alph1 [5] clearly showed how buffer overflows can be used to escalate privileges and execute arbitrary code on the victim– Buffer overflows cause visible functionality problems too – most

notably, the software being overflowed into and the software doing the overflow will likely crash, have inconsistent data, or operate in an unintended fashion

• More recently, Watchfire [32]made public an exploitation technique to use dangling pointers to execute arbitrary code on the victim– Dangling pointers may cause visible functionality problems too –

again, most notably, the software could crash, have inconsistent data, or operate in an unintended fashion

Common Root Causes (cont)

• Even weak passwords, often regarded as strictly a security problem can create undesirable functionality flaws– The Morris worm exploited weak passwords and caused a substantial

degradation of services– Conficker, which exploited a Windows bug and weak passwords, and

many other worms rely on bugs and weak passwords

• Even the most seemingly benign flaws, like off-by-one errors can cause disastrous security and functionality flaws– Such flaws can be exploited to execute arbitrary code, but also can be

triggered by mere use and cause the system to crash or have degraded services [13][15][19]

INPUT VALIDATION

Input Validation

• Input validation flaws arise when input (data egressing the process or thread from a source outside of the process or thread) is not properly verified and sanitized prior to use– These include come buffer overflows, format string flaws, integer

overflows, SQL injection, LDAP injection, SMTP injection, code injection, command injection, Cross-Site Scripting, and many other problems

• Proper and complete design documentation can help stop these

• The flaws can be discovered using a variety of different techniques, but no single technique can stop them all, only a ‘cocktail’

Finding Input Validation Flaws With UML

• Many input validation flaws can be found using UML diagrams and use case scenarios, or can be protected against using well-crafted OCL statements that are translated into code during implementation– This includes detecting or preventing certain

instances of injection attacks (SQL, XSS, command, code, LDAP, SMTP, JSON, etc) and overflows (buffer, off-by-one, etc)

UML Input Validation Example

• There is the potential that query could allow SQL injection (DBHandler performs the Database handling), or that the user could enter otherwise incorrectly formatted information– OCL and supporting documentation comes into effect here– context GUI

inv: query.matchesRegEx(“a-zA-Z0-9\ ”) and query.size()<40• Note that this requires an OCL extension be written that accepts regular

expressions

• This also forces the design team to clearly specify all data

Sidenote

• Regular expressions that dictate data content should be whitelists, not blacklists– Whitelists dictate what is allowed, blacklists dictate what is not

• All input should have OCL constraints that dictate the allowable content (a whitelist) and the maximum (and minimum in certain cases) size– This helps to stop overflows and injection, as well as providing carefully

crafted and examined whitelists that conform to the requirements, while providing input validation

• Of course if keywords and characters with special meaning are in the whitelist, the input will have to be properly escaped in the code – hard to convey in UML diagrams, but pre conditions and invariants can be used to do this with more OCL extensions

DATA FLOW

Data Flow

• Data flow flaws arise when data take paths that are untrusted or can present various problems– Many of these issues concern integrity and

confidentiality– The data may be modified while it traverses the less

trusted/untrusted path such that it causes problems – note that such data should be examined prior to use

• Examples include traversing a hostile environment, traversing faulty systems, and experiencing the data loss that is normal in any environment

Data Flow (cont)

• It is extremely important to know data flow in environments where there are levels of trust or in MLSSs– Most kernels implement a limited form of hierarchical

protection domains, or ‘protection rings’• This prevents problems in outer rings from impacting the inner

ring (both functionality and security benefits)• The military and systems requiring high fault tolerance benefit

• Limiting access points by having ‘gates’ can help prevent or limit the scale of compromises, as well as contain damage should a fault occur

Finding Data Flow Flaws With UML

• Certain data flow flaws can be exposed by UML diagrams, but mostly in Use Case diagrams– This involves ensuring that data re-entering the

system has not be tampered with or is not malicious (in some cases, both should be checked)

• This is similar to input validation, since the data reentering the system is assumed to be data from an external source

Use Case Data Flow ExampleUse Case Number 1

Use Case Name Initial Data Distribution

Overview Follows the distribution of initial data in the distributed system

Type Primary

Actors Master (P0) and Slave Processors (P1-PN)

Properties

Preconditions All slaves are up and waiting for the master to communicate

Flow Main Flow:1. Master prepares each matrix segment (one per slave)2. Master sends each matrix segment to the responsible slave3. Master enters waiting state pending responses4. Slaves receive matrix segment5. Slaves transmit go-ahead to master

Performance O(n)

Security Data crosses system’s boundaries prior to use

Other

Use Case Data Flow Example (cont.)

• What happens if the slaves never receive the matrix segments sent to them?– The master will wait forever– This should be handled as an Alternate Flow in the Use Case

• What happens if some of the data in the matrix segment is lost or modified?– At best the result will be inaccurate– At worst the slave may become compromised or crash, resulting in the master waiting

forever for the slave to respond– This should be handled as subflows – the master should hash the matrix segment and

send the hash along with the segment• The hash should also be encrypted or signed if the environment is hostile

• What if someone injects a slave response (affirmative or negative)?– Perhaps random network chatter– Perhaps a malefactor– This should be handled as a subflow (if it is a concern) – the master and slave should

use encrypted channels or traffic

• All environments outside the software (the network and even the system’s bus) should be assumed buggy/malicious in most cases

UML Data Flow Example

• Data flowing between the two packages (VideoLibrarySystem and Users) should be protected– This is easiest by providing ‘chokepoints’ – normally making a single class on each side

(frontend, backend, middleware, etc) that handles interprocess/internetwork communication and protecting all methods in that class

• Classification should actually be placed in each package– This not only serves to decouple the interface and backend, but also helps to serve as

another sanity check and reduce the likelihood of a fault in one package effecting the other package

Finding Data Flow Flaws With Models

• UML cannot be easily used to find many of the more complex data flow flaws– Alloy, ORC, and SMV, as well as similar modeling languages

can find many of the more complex data flow flaws• Examples include verifying that algorithms effectively protect

against or detect data tampering, spoofing, denial of service (to the extent possible), and information disclosure

• More than anything, modeling can help to verify the adequacy of controls, but (as a consequence) also detect when the data flow could potentially expose the system to threats

Alloy Data Flow Example/** Program to verify enforcement of the x86 protection rings **/abstract sig Ring{}one sig RingZero, RingOne, RingTwo, RingThree extends Ring{}one sig Policy{order:Ring->Ring}{order=(RingThree->RingTwo)+(RingTwo->RingOne)+(RingOne->RingZero)+(RingZero->Ring)+(RingOne->Ring)+

(RingTwo->RingTwo)+(RingTwo->RingThree)+(RingThree->RingThree)}sig Program{onRing: one Ring}sig State{programs: set Program,connections: programs->programs}{ -- Note that we are allowing programs to connect to themselves (this is common)

no prog: Program | prog not in programs -- Ensures all Programs are in our stateno progA, progB: Program | progA->progB in connections and progA.onRing->progB.onRing not in Policy.order -- Enforce Policy.order

}assert verifyDataFlow{all state: State | not getAnInvalidDataFlow[state]}pred getAnInvalidDataFlow[state: State]{ -- Attempts to find an invalid data flow (one that violates Policy.order

some progA, progB: Program | progA->progB in state.connections and progA.onRing->progB.onRing not in Policy.order}

• Models can be used to verify control of data flow• They can also show that algorithms to protect or contain

damage are accurate• Boolean logic models can even check fault tolerance, like

examining mechanisms that automatically adjust data flow to route around faulty paths– This ensures that the algorithm works

ACCESS CONTROL

Access Control

• Access control flaws occur when access to a resource is improperly granted or denied– Can exist under numerous circumstances:

• Race conditions, improperly assigning access control restrictions, improperly protecting resources, shared resources, etc.

• A tremendous amount of work has been done concerning access control violations and modeling– The works tend to pick on specific access control

violations, like time-to-check-time-to-use and breaking Bell-La Padula model

Finding Access Control Flaws With UML

• Many access control flaws can be exposed by UML diagrams– Sequence diagrams can expose race conditions, some

instances of starvation and deadlock, and time-to-check-time-to-use

– Use cases and design diagrams can expose some instances of improperly protected resources and instances where shared resources can lead to access control flaws

• Note that UML will expose these flaws in the design and may not expose flaws in algorithms or internal working mechanisms – for that models must be used

Use Case Access Control Example

• In this example of a basic medical record system, there are four actors – the Doctor, Nurse, Records Office, and Front Desk– The Front Desk can create a record (person) and delete a record– The Records Office can only view records– Doctors can view and modify records– Nurses can view and modify records

• There are multiple potential access control flaws– Deletion is often as good as modification– Modification can include deletion (erase all of the data in the record is the same as

deleting the record and creating a new one)• In fact, the two can replace most of each other’s functionality and capability, with regard to the

destructive force/results

UML Race Condition Example

• In this example, a parallel Branch and Bound Knapsack solver written in Java spawns multiple threads that then attempt to solve the knapsack problem– Each of the thread (ParallelKnapsackWorker) works for a period of time, say 5 seconds

and then the thread notifies ParallelBranchAndBound the best knapsack found thusfar• If the thread has finished, it posts a final notification and is joined

– The threads sleep immediately after sending notification so that ParallelBranchAndBound can then grab the current data from the thread (to prevent a race condition where the data is changing as the parent retrieves it)

– The parent then calls notify() to wake the thread• This can, itself, create race conditions, deadlock, and starvation because Java doesn’t allow the

program to specify which thread to wake

UML Race Condition Example (cont)

• Note that we now have two threads notifying ParallelBranchAndBound at the same time– This means that one of the notifies is either going to go

unheard (image on the left) or…– …We have a very serious race condition where what we were trying to

protect from (ParallelBranchAndBound grabbing data while it is being updated) occurs and worse – we have the complete loss of data as one thread wakes another instead of ParallelBranchAndBound, who then performs another notify() – potentially accidentally waking another thread and causing a complete loss of control

UML Access Control Example

• Without any OCL to prevent it, a Student can rent a scientific video – something that the requirements clearly stated should not be possible– If implemented as the diagram dictates, the system does not meet the

required access control capabilities… OCL should be included

UML Access Control Violations

• Carefully examine privileges– Fields should always be private unless they are marked ‘final’– Methods should be private unless there is good reason

• In most cases only constructors, getters, and setters should be anything other than private– Getters and setters should be limited and only available when required

• Carefully examine sequence diagrams for potential starvation, deadlock, and race conditions

• Examine OCL to ensure that it prevents instances where access to certain resources is inadvertently granted by the diagram were the OCL incorrect or missing– Common examples include bypassing the login to gain access to

protected data and using input validation flaws to gain access

Finding Access Control Flaws With Models

• UML can find many of the access control flaws, but may also create some– As in the example where the lack of OCL allowed a violation of

the requirements and a breech• Modeling, however, can help to discover complex or

covert access control flaws– Models trivially discover covert channels and side channels– Models also can be used to define the rules that will enforce

intended controls• They can also prove that intended controls fail to achieve the desired

effect

– Models have even been used to discover previously unknown access control flaws

Prolog Access Control Example/** Excerpt from [14] -- a model to ensure database subtransactions are atomic, do not conflict, and can be rolledback/recovered from **/haveto(Node1, Action1, Node2, Action2) :-

catch(not(maynot(Node1, Action1, Node2, Action2)), _, true).cdefs(a, A, B) :-

cdefs(t, A, B), haveto(A, abort, B, abort).

cdefs(b, A, B) :- checkorder(A, begin, B, _).

cdefs(ba, A, B) :- cannot(A, commit, B, _), checkorder(A, abort, B, _).

cdefs(bc, A, B) :- cannot(A, abort, B, _), checkorder(A, commit, B, _).

cdefs(c, A, B) :- checkorder(A, commit, B, commit).

cdefs(ex, A, B) :- haveto(A, commit, B, abort).• The full model identified several new subtransaction conflicts as well as the

previously unknown state of ‘ignored dependencies’ that were easy to discover using a logic model– Conflicts include instances where race conditions are present, the database cannot

rollback/recover if something fails, access control mechanisms are violated, or two subtransactions are attempting to perform conflicting operations like committing and strongly committing the same link

• Using Prolog, similar flaws are trivial to detect and correct– Alloy can also be used to find certain access control flaws – Prolog is generally

better, though

ENCRYPTION

Encryption

• Cryptographic flaws exist in all known crypto schemes and their implementations– Some flaws are far more egregious than others– Flaws include poor randomness, poor entropy collection,

poor keys, weak S-boxes, key exchange flaws, key sizes, and reversibility of the algorithm• Evil oracles have always been the Achilles’ heel and allowed

the reversing of the entire MD5 family (MD5, SHA, etc)

• Modeling and diagramming have uncovered previously undiscovered flaws in common software– Web-browsers have been shown to be trivial to hijack

sessions from because of the way they handle SSL

Finding Encryption Flaws With UML

• UML can be used to find some common cryptographic flaws– Most of the flaws are found in state and sequence diagrams– Encryption flaws found in state diagrams are usually just for

that ‘oracle’• If a state diagram shows a flaw in OpenSSL’s handling of SSL, a

similar flaw may or may not exist in other implementations of SSL

– Flaws discovered in sequence diagrams are likely flaws in the actual algorithm

• These flaws, of course, exclude mathematical errors, other than to point out that the particular implementation relies on a library that provides weak entropy or randomness (e.g. rand)

UML Encryption Flaw Example (cont)

• Note the Trusted Third Party– What if this trust were undue?

• The entire algorithm could be broken• SSH suffers this problem as well

• What if the attacker spoofed the client, rather than the server?– The client never posts a certificate, so can

be trivially spoofed• SSHv1 suffered this flaw; SSHv2 and

TLSv1/TLSv1.1 normally ‘authenticate’ the client (weakly)

• What if the attacker downgraded the cipher spec?– This would allow the attacker to more

easily retrieve the session key• This is common and normally done by

spoofing the client• In many cases the client or server

downgrades as a matter of course– Compatibility being the major reason– The result is a much weaker algorithm

Finding Encryption Flaws With Models

• UML can only find encryption and cryptographic flaws that are ‘surface level’– Flaws in S-boxes, the underlying mathematical assumptions,

etc, are invisible in any UML diagram• Modeling, however, can be used to discover many of the

flaws that UML and OCL are unable to find– Models easily evaluate truth statements, such as those about

trust and who knows what when• As well as who trusts whom and when

– Models can also help determine why levels of trust are what they are

– Most importantly, models can verify the math behind the algorithm

Prolog Encryption Flaw Example/** Excerpt from a model to ensure the proper sharing of cryptographic keys**/trust(Node1, Level1, Node2, Level2) :-

catch(not(violatesTrust(Node1,Level1, Node2, Level22)), _, true).keyExchange(Key, A, B) :-

trust(A,_,B,high).keyExchange(Key, A, B, C):-

trust(A,high,C,high),trust(B,high,C,high).

sessionKeyExchange(Key,A,B):-trust(A,high,B,high),keyExchange(Key,A,B).

• The full model explains how most systems perform key exchange– Or more correctly how most systems should perform the key exchange

• Models like this can help prevent a system from performing a poor key exchange or one that leaves a system vulnerable to hijacking before, during, or after the key exchange

• Using Prolog, logic statements like those governing trust and security decisions can prevent the mistakes of SSHv1 and SSL– The mathematics could be proven in Prolog, but there are much better

tools for that…

Coq Encryption Flaw Example/** Excerpt from a model to prove mutual exclusion of a cryptographic algorithm**/Lemma correct:forall memo:Memo,Cycle memo->Oracle->CorrectTrace.

refine (cofix choice : forall memo:Memo,Cycle memo->Oracle->CorrectTrace := fun (memo:Memo)(cycle:Cycle memo)(oracle:Oracle)=> SCons _ _ memo _ (choice (proc (hd oracle) memo) _ (tl oracle))).

• Numerous languages exist to help prove the correctness of cryptographic algorithms– Many of them rely on Prolog to provide the knowledge base and logic

• Coq and PVS are commonly used to prove the underlying mathematical theorems– In the example above, Coq is proving the lemma that illustrates a key

property of mutual exclusivity– Failing this lemma means that the algorithm fails to provide mutual exclusion

• By using a mixture of Prolog-based languages and theorem solvers, most common encryption flaws can be eliminated or reduced– Alloy generally fails to provide the capabilities and is much more difficult in

this realm than Prolog-based solutions

IN EXIGO

Conclusion• Formal Specification Techniques can be used to detect design flaws

– In some cases, after-the-fact models and diagrams have detected flaws in implementations• These are more costly and difficult to remedy than had they been caught in the

design phase

• In most instances, the flaws can be caught through both modeling and to some degree with diagrams– Many of the flaws can be seen through sequence diagrams and use

cases– Other flaws can be caught with class diagrams

• Prolog is a powerful language that enables most modeling languages to discover some of the more complex flaws– This is particularly true with flaws that require changing states or heavy

logic and a powerful knowledge base

Acknowledgements

• This work relied heavily on previous works of mine for examples, to include the modeling examples provided

References[1] Alan Shaffer, Mikhail Auguston, Cynthia Irvine, and Timothy Levin, “A Security Domain Model to Assess Software for Exploitable Covert

Channels,” Proceedings of the ACM SIGPLAN Third Workshop on Programming Languages and Analysis for Security, ACM Press, 2008.[2] Alan Shaffer, Mikhail Auguston, Cynthia Irvine, and Timothy Levin, “A Security Domain Model for Implementing Trusted Subject

Behaviors,” Unpublished.[3] Alan Shaffer, “A Security Domain Model for Static Analysis and Verification of Software Programs,” Proceedings of the 20th International

Conference on Software Engineering and Knowledge Engineering, Redwood City, California, 2008, pp. 673-678.[4] Alan Shaffer, Mikhail Auguston, Cynthia Irvine, and Timothy Levin, “Toward a Security Domain Model for Static Analysis and Verification of

Information Systems,” Proceedings of the 7th OOPSLA Workshop on Domain-Specific Modeling, 2007.[5] Aleph1, “Smashing The Stack For Fun And Profit,” Phrack, 7(49), 1996.[6] Aliaksei Tsitovich, “Detection of Security Vulnerabilities Using Guided Model Checking,” Lecture Notes in Computer Science, Springer-

Verlag, Berlin, 2008, pp. 822-823.[7] Andrei Lapets, “Formal representation and reasoning approaches in modeling cryptographic protocols”, Unpublished.[8] Catherine Meadows, “Formal Verification of Cryptographic Protocols: A Survey,” Lecture Notes in Computer Science, Springer, 1995, pp.

135-150.[9] Chad R Dougherty, “Vulnerability Note VU#395412: Apache mod_rewrite contains off-by-one error in ldap scheme handling,” US-CERT, US

Department of Homeland Security.[10] C.R. Ramakrishnan and R. Sekar, “Model-based analysis of configuration vulnerabilities,” In Journal of Computer Science, IOS Press, 10(1-

2), 2002, pp. 189-209.[11] Daniel Jackson, Software Abstractions: Logic, Language, and Analysis. The MIT Press, Cambridge, Mass, 2006. [12] Daniel Owens. Application and Website Security - Fundamental Edition, National Aeronautics and Space Administration, 2009. [13] Daniel Owens. Application and Website Security – Developer Edition, National Aeronautics and Space Administration, 2010.[14] Daniel Owens. Automated analysis of dependencies in advanced transaction models, SystemSecurities, 2006.[15] Daniel Owens. Certification Test & Evaluation Report for the Common Link Integration Processing (CLIP) Release 1.2.1.26, Certification

Test and Evaluation Report, Booz Allen Hamilton, 2008.[16] Daniel Owens. Common Link Integration Processing (CLIP) Integrity, Whitepaper, Booz Allen Hamilton, 2008.[17] Daniel Owens. Integrating Software Security Into The Software Development Lifecycle, SystemSecurities, 2008.[18] Daniel Owens. IT Security Threats: Improper or Inadequate Change Management, National Aeronautics and Space Administration, 2009.

References (cont)[19] Daniel Owens. Fault Tolerance In Sensitive Embedded and Real-Time Information Technology Systems, SystemSecurities, 2009.[20] Jim Arlow and Ila Neustadt, UML 2 and the Unified Process: Practical Object-Oriented Analysis and Design, 2nd ed, Addison-

Wesley Professional, 2005.[21] Junfeng Yang, Paul Twohey, Dawson Engler, Madanlal, Musuvathi. “Using model checking to find serious file system errors,”

USENIX Symposium on Operating Systems Design and Implementation, 2004.[22] Michael Howard and D. LeBlanc, Writing Secure Code. 2nd ed, Microsoft Press, 2002. [23] Mikhail Auguston, Alan Shaffer, “Security Domain Model and Implementation Modeling Language Reference Manual, version

2.0,” Unpublished, May 2008.[24] Musab AlTurki and Jose Meseguer, “Reduction Semantics and Formal Analysis of Orc Programs,” Electronic Notes in Theoretical

Computer Science, 200 (3), 2008, pp. 25-41.[25] O.H. Alhazmi, Y.K. Malaiya, I. Ray, “Measuring, analyzing and predicting security vulnerabilities in software systems,” Computes

& Security, 26 (3), 2007, pp. 219-228.[26] R. Ritchey and P. Ammann, “Using model checking to analyze network vulnerabilities,” In Proceedings of the IEEE Symposium

on Security and Privacy, 2001, pp. 156-165.[27] Shuo Chen, Jose Meseguer, Ralf Sasse, Helen J. Wang, Yi-Min Wang, “A Systematic Approach to Uncover Security Flaws in GUI

Logic,” in IEEE Symposium on Security and Privacy, Oakland, California, 2007.[28] Shuo Chen, Zbigniew Kalbarczyk, Jun Xu, Ravinshankar K. Iyer, “A Data-Driven Finite State Machine Model for Analyzing Security

Vulnerabilities,” IEEE International Conference on Dependable Systems and Networks, 2003, pp. 605-614.[29] Shuo Chen, Ziqing Mao, Yi-Min Wang, Ming Zhang, “Pretty-Bad-Proxy: An Overlooked Adversary in Browsers’ HTTPS

Deployments,” 2009 30th IEEE Symposium on Security and Privacy, 2009, pp. 347-359.[30] SMV. SMV: A Symbolic Model Checker. http://www.cs.cmu.edu/~modelcheck/.[31] Stuart McClure, Joel Scambray, and George Kurtz, Hacking Exposed: Network Security Secrets and Solutions. 6th ed, McGraw-Hill

Osborne Media, 2009.[32] Watchfire, “Dangling Pointer: Smashing the Pointer for Fun and Profit,” Blackhat Convention, 2007.[33] Y. Shin and L. Williams, “An Empirical Model to Predict Security Vulnerabilities using Code Complexity Metrics,” Proceedings of

the 2nd International Symposium on Emperical Software Engineering and Measurement, Association for Computing Machinery, New York, 2008, pp. 315-317.