Turning the Tide Driving Software Security in the Enterprise

John B. Dickson, CISSP

2

Turning the Tide: Driving Software Security in the Enterprise Intended Audience Stakeholders involved in the creation of the software within an organization have a vested interest in understanding and contributing to the process of building more secure software. Although roles can differ between company “headquarters” managers responsible for policy and software development teams in the field, a common set of fundamental practices can provide a starting point for process improvement. This document targets the two groups most central to the creation of software in large organizations:

3

Executive Summary Creating a software security initiative in a large organization is difficult by any measure. Many times organizational culture or politics can present more daunting challenges than purely technical issues when implementing a software security initiative. Status quo bias reinforced by deadlines for new features and functionality can also provide development managers a strong counterargument for implementing software security concepts. Unfortunately, attacks are more sophisticated and applications have become a central avenue of attack. The option to accept the status quo – building software without a consideration for security – is becoming a less viable option given compliance pressures and public data breaches involving software. As a result, organizations are now increasing efforts to tackle the issue in a meaningful, systemic way. The vast majority of information on software security either focuses on technical means to write more secure code, or strategies for putting controls around the software development process. Unfortunately, there is a dearth of information on how one might go about the process of leading a software security initiative within a large organization. What most understand though, is that is where software security initiatives are most likely to fail or lose steam due to organizational issues, not technical issues. On top of that, most literature on the topic assumes away the issue of corporate buy-in, leaving managers to determine for themselves the most likely path to success. To address this perceived gap, this white paper focuses on what senior leaders need to do to successfully conduct an enterprise-wide initiative to build more secure software. To change the way large organizations build software, an enterprise-wide initiative is required. This is at its heart a reengineering of existing development processes. A set of common best practices that large organizations have used exists to make software security initiatives successful. At the most basic, these include taking a disciplined approach by characterizing the landscape, securing champions, defining standards and strategy, executing, and then sustaining the effort. These steps, tailored to your organization, will help ensure that your corporate-wide efforts to secure applications are as productive as possible. It is the experience of this author that nearly every successful software security initiative has included most of these common strategies.

4

Background Large organizations with unique operational requirements typically find themselves building custom software systems to address their specific needs. Unfortunately, building custom software on time, on budget, and without bugs is difficult. Building software that satisfies the aforementioned requirements and complies with an organization’s software security policies presents an even more difficult problem. Although the tools and practices to build secure software are maturing, most organizations find internal hurdles to be more daunting. Organizational impediments including culture, differing software development approaches, and short-term business drivers make it more difficult to effect meaningful change. Those interested in leading secure development initiatives within their organizations face myriad challenges. If a development manager faces these challenges head on, there is a reasonable chance of making a positive impact. Without question, the larger and more complex the organization is, the more likely varying cultures within the organization will present a potential first roadblock to build more secure code. For starters, certain business units or divisions may have missions that drive vastly different approaches to developing software. In organizations that value openness and collaboration, the rationale for secure code will be openly questioned. Higher education and pure research and development organizations come to mind as examples that are likely to reflect this type of culture. By contrast, highly regulated industries that must protect sensitive financial or customer data may be more comfortable with internal initiatives aimed to enhance the state of software security. To further complicate matters, large organizations may have groups with cultures that span the spectrum of openness, thereby presenting a more difficult challenge for those attempting to create a consistent approach for delivering secure software to internal clients. When you combine differing approaches to building software with the culture factor, the problem is exacerbated. Sometimes shared philosophies, such as what software development methodology an organization embraces, are driven by factors such as geographical proximity of software development teams. I have seen organizations – both private sector and public sector – that have a variety, an entire ecosystem, of different software development lifecycle (SDLC) approaches, from Waterfall, Extreme Programming (XP), Scrum, to Microsoft Solution Framework (MSF). Most organizations have several SDLCs – not one. Unfortunately, many organizations do not arrive at this situation by design. Instead, this situation exists due to distributed operations and years of corporate mergers and acquisitions that left a hodgepodge of SDLCs.

Finally, there is a strong status quo bias for not changing the way software is built in certain organizations. Tradition, for one, exerts an unspoken pressure on many, because software development teams have built software for years without consideration for security. Put simply, if an organization hasn’t experienced a breach via its applications there is a strong chance that managers will not take the threat seriously. The argument of “it hasn’t happened to us” is particularly compelling to C-level executives and senior leaders that deal with a variety of serious business risks on a day-to-day basis. At its core, an initiative to change the way an organization builds software is a fundamental business process improvement effort. Such efforts are never easy, which explains why many fail. It is no coincidence that many successful software security initiatives have been led by managers

5

who were able to navigate the troubled political waters of their organizations and wielded informal power to move the effort forward. Given these realities, how does one go about leading the effort to build secure software within large and complex organizations? Essentially, managers tasked with driving a software security initiative are being asked to reengineer software development lifecycles (SDLC) to ensure more secure software is produced on a repeatable basis. Put simply, they are being asked to revamp one of the most tedious processes – that of developing custom software for large organizations. Existing professional literature on the topic of software security falls into two camps. The first camp focuses on the technical aspects of writing and deploying secure code – code assessments, dynamic (read “black box”) testing, tools, and practices surrounding this creation of secure code. The majority of books on software security focus on “greenfield” software systems – namely, those built from the ground up. The second camp includes books written about tactical process improvement steps that organizations can implement to “bake in” certain security improvement activities within their software development lifecycle. Although technical and process improvement steps are critical to building secure software, there is an underlying assumption in this body of literature – namely, that the organization has already bought into the idea of revamping their SDLCs to build more secure software. This white paper picks up where others have left off and focuses on “how” one should approach a secure software program, recognizing fully that in complex, highly political organizations, internal hurdles might be more of a roadblock than technical or process hurdles. I have had the opportunity to work with numerous senior security officers and software development manager who have initiated and followed through with the process of undertaking organizational change to build more secure applications. What follows represents an approach that includes many of the common denominators that have worked for others who have been successful influencing organizational change.

6

Proper Approach In order for a software security initiative leader to successfully implement a successful secure software program, he or she must take a phased approach that considers organizational culture at each step. Like any ambitious undertaking, the effort to change the way an organization builds secure software is difficult. Some decide to take a “shortcut approach” by deploying a web application firewall (WAF) or running an automated scanner against applications, but this does not solve the problem of securing software in most organization. It only highlights that deeper process improvements involving the SDLC must be undertaken, particularly for higher level business logic or authorization vulnerabilities.

7

Characterize the Landscape The first step, Characterize the Landscape, is to understand the task ahead and to be able to craft a realistic strategy for adoption within your organization. This includes five critical components:

Compliance Framework Step one in the process is to understand the compliance framework to which the organization must comply. What are the known regulations or directives from headquarters that will exert influence on how security should be applied to software within the organizations? Who is responsible for monitoring changes in these regulations and what is the process that the organization has in place to gauge their impact? Who audits these regulations, both inside the organization and externally? Given that nonconformity with certain regulations is the first and fastest way for security managers to get fired, understanding the organization is a starting point of any discussion. It also gives the security initiative leader “ammunition” to make the business argument with nontechnical leaders as to why secure software initiatives are important to the organization.

Cultural Norms After understanding the regulations and rules that govern your organization, the second thing a security initiative leader should do is analyze and understand the culture of the different operating units that are building software. Break down the operating units or development teams into logical groups – teams that build independent projects and are viewed more or less as standalone entities. Once you’ve broken down the organization into different groups, write down terms that best describe them (note: if your organization is large enough, you might have to drill down to a couple of layers in the organizational chart). For example, is the group research-driven with a significant number of Ph.D.’s on staff, or is it loose-knit and works in informal teams? Perhaps it deals with highly sensitive information – nuclear secrets or acquisition opportunity information that is not available publicly. You should fully understand the unit that you will be interacting with as you will likely have to tailor your approach to its culture.

Understand SDLCs in Place Another critical building block of the approach is to understand what Software Development Lifecycle (SDLC) - or software development approaches - the organization uses. Through focus groups or informal surveys, try to quickly catalog the different approaches the organization uses to build software. Again, if your organization is large enough, you might have to organize this by division or operating location. There are numerous development approaches that range from the

8

most precise to very flexible. How many recognized approaches do your organizations have? How different are they within each division or operating location? Why are their approaches different and what projects/products are they working on that influence the choice of development approach? Cataloging the methodology that each group uses will be more critical when the time comes to engineer change, but for now, just knowing what approaches are in place is a must.

Identify Existing Software Security Artifacts After identifying and understanding the development environment, the next step is to identify any existing artifacts of software security, regardless of how effective they have been. Is threat modeling or peer reviews being conducted in certain groups? Are external penetration software tests being conducted in others? Are policies in place regarding software and data security? Have any of the groups bought tools or put open source tools in place, even in an ad hoc fashion? Regardless of how much evidence of past activity you find, it’s important to identify what might be in place. This area will be the beachhead for future application security initiatives and a first candidate to hold up as good examples to others of what can be done to improve the state of security in applications.

Identify the Gap Between Stated Policy and Practice The final step involved in Characterizing the Landscape is to identify the existing gap between stated policy and practice. Why should you do this? Simply put, there is almost always a gap between what organizations say they do, and what they actually do. This is particularly true in the case of software development, where few have visibility into the “sausage making process.” Understanding where a gap might exist also will help you better understand the risk profile of your organization given that an organization might be more liable should it be aware and not do anything, versus knowing nothing at all.

9

Secure Champions The second phase, Secure Champions, focuses explicitly on the fact that you will need clear support of executive sponsors as well as other key influencers in the organization in order to be successful.

Confirm Support from Senior Leadership The first facet of this phase will be to engage senior leaders in your organization to confirm support for attacking the status quo of building software insecurely (you could make a strong argument that this step precedes the “Characterize the Landscape” phase). The starting point for this discussion, many times, is the Chief Information Officer, or the equivalent. If outside regulatory pressures are strong enough, consider engaging the Chief Financial Officer or even the Chief Operating Officer. Put your arguments in terms of business risk. These senior leaders will not understand the minutia of software vulnerabilities, nor will they care. They will understand the business impact of a breach, though, and have probably read about TJX, Hannaford, and other businesses that had to deal with very public breaches. Tailor your pitch to the senior leader, and remember: you probably only have one shot to secure this support.

Validate the Necessity for Building Secure Software with Internal Audit Perhaps as important as executive support is getting the support of your internal audit – or in the case of government groups, the Inspector General – to validate that the necessity for building secure software is critical. Given that you’ve already done the research to understand the compliance framework relating to applications in the earliest efforts, this should be straightforward. In certain organizations, having internal audit publish an opinion about secure coding has been an effective way to get the attention of key managers. It might be just enough of a business rationale for software groups to pay attention to the necessity of application security.

Line Up Internal Champions within Development Groups Finally, you will have to line up internal champions at the divisions or operating locations at which software development activity takes place. Go back to your research in the early phase of your project to identify champions in the field in development groups. They are likely to carry the title “Senior Developer,” “Architect,” “Software Lead,” or the equivalent. There are many different ways to identify these secure coding ombudsmen. One Fortune 500 semiconductor company used lunch and learns at different corporate locations as a way to identify likely candidates to serve as champions.

They are local influencers, they define the local culture, and they are likely to be the ones most important when you get to the implementation phase of your initiative. One thing to remember here: developers usually look for career opportunities to distinguish themselves from their peers. Becoming an application security “guru” can be very attractive to the up and coming developer who might seize on the opportunity to become the local guru on everything relating to software security.

10

Define a Strategy and Initial Standards Now that you have awareness of the state of software development and have support in several areas, the next logical step is to Define a Strategy and Initial Standards. This may seem like an obvious step, but I am constantly amazed to find clients who take the “ready, fire, aim” approach. Given that you may have only one opportunity to successfully roll out a secure development initiative, you should not overlook this step in the process.

Identify the Most Vulnerable Applications To that end, the first thing you should do is begin to conduct a “lightweight” risk assessment of applications owned by your organization to identify the most vulnerable applications to provide some qualitative ranking for decision-making. The initial step in the risk quantification process is an inventory of applications. In my experience, this is easier said than done. In certain environments the amount of fielded applications that exist in production might number in the hundreds (or thousands for that matter). You may have to reach out to numerous development managers and support staff to obtain a list that reflects reality.

Rank Applications From Most Critical to Least Once you have a list of what you think best represents production applications, conduct a brief risk analysis and rank order the applications from most critical to least critical. Factors that might influence the rank order include whether or not the application:

o processes a large amount of financial transaction critical to business o is Internet-facing o includes personally identifiable information o contains other very sensitive data o Is written in a legacy language that might be easily exploited

Conduct Assessment on Critical Application Then pick one of the critical applications and conduct an application-level assessment via dynamic, static, or hybrid testing approaches. Perhaps most importantly, this test will identify the most egregious vulnerabilities that currently exist in the code base. Making some qualitative attempt to measure application risk is critical, but really only one facet of a comprehensive application security initiative. You now have a rough starting point for future improvement and you’ve been able to quantify the existing risk. Make sure to have a sharp developer with application experience to conduct this assessment; otherwise, you run the risk of getting a superficial scan of the application. Second, by assessing a live application you will dispel pre-existing perception that “it can’t happen here” and will move the discussion from the practical to the real. This is the most important part – the developers must understand and embrace the fact that the applications they built are not perfect and, in fact, application vulnerability might exist in their code base. This takes the discussion from the abstract to the concrete fairly quickly.

Define Standards for Future Improvement After you’ve done a high-level assessment of a handful of the critical applications, the next step is to define technical standards for future improvement. The key concept at this phase is to set reasonable goals and to demonstrate future progress. Realistically, the process of changing the way an organization builds software is not an overnight activity, as you have probably learned in the earlier part of this paper. Setting aggressive goals to transform your SDLC to an NSA-like process overnight is unrealistic and ultimately will lead to failure in your initiative. And, given that you likely have only one opportunity to roll out this process, it is critical to begin with simple goals

11

that will allow you to capture easy wins and show progress to senior leaders. Examples of modest goals for the first part of your execution phase include the following:

1. Conduct a ½-day classroom overview of application security concepts for your development leads, architects, and project managers.

2. Publish a “Top 10” list of coding standards that reflect fundamental security values for your organization. For example, define how your organization handles data at rest with encryption.

3. With the help of internal or external resources, fix one of the applications you assessed earlier in the process by eliminating identified vulnerabilities

4. Implement threat modeling at the early stages of a handful of big projects so you can begin to develop an internal competency to review applications at the earliest stages of development.

5. Improve one facet of the SDLC to inject a software security “control.” This might include security peer review, black box testing, or a similar activity that improves the security of code being published.

For ideas on some of the best practices that could comprise this initial set of goals, there are several resources from which you can pull ideas. A starting point for defining initial software security standards might include the OWASP Testing GUIDE, or one of the National Institute of Standards and Technologies (NIST) publications covering software security if you are a Federal agency. The SANS institute has published its “Top 25” list of most dangerous programming errors and practices to identify and remediate them. At this point, consider engaging consultants who have done this for numerous organizations or engage a peer organization slightly further ahead of this process to mentor you (assuming, of course, they are not a competitor). Regardless, having a baseline set of practices and procedures that are well thought out, realistically implemented, and reflect what is realistic in your organization is absolutely critical. A successful first set of policies will ultimately define the lowest common denominator between your disparate development groups, and you can consider improving software security practices from there. Finally, now is the time to circle back to ensure that what you’ve laid out dovetails with existing security policies and procedures that you captured in Phase One of your process. What you are doing in the development work has to align with what the security stakeholders in your organization hold to be self-evident truths.

12

Execution Phase So you’ve done your homework, secured supporters up the chain of command and in the field, and laid out your strategy and goals. Now comes the hard part. Without question, the most challenging part of the initiative is the Execution Phase. The focus on this phase is to grab the attention of key development managers to begin the herculean effort to improve the software development process in order to build secure software. Sophisticated leaders have found a variety of creative ways to approach this phase, from “traveling road shows” to innovative awareness campaigns that bring the issue of software security to the forefront. There are several common denominators that I have seen work in the field that helped managers make a strong push at the outset.

Empower Security Champions First, the Execution Phase is when you put the security champions in the field to work. Make them part of the plan, and include them in your planning process. At this stage, formal recognition of these secure development ombudsmen might also enhance their standing within their groups and provide them enough authority to actually move some of these efforts forward. Brief them on your plan – they are your evangelists and need to know what you want them to evangelize! Keep them updated on what is working in certain groups, and push down business justifications so they can continue to justify their spending time on the software security initiative. A common wiki or blog to capture tactics is helpful in the largest organizations. The key point here is to have a common approach, and “over-communicate” to your supporters in the field so they embrace the message that you want them to carry to their managers.

Tailor to Your Audience Second, tailor the delivery to the audience. As discussed in an earlier part of this paper, culture plays a big part in how these types of initiatives are received in the field. Perhaps a training approach that encourages self-learning is better received in one organization, where auditing and the up-front demonstration of weaknesses that exist might be more valued in other organizations. It is critical that you consciously implement a rollout strategy tailored to the approach of the organization.

Set Qualitative and Quantitative Metrics for Success Since you’ve defined initiative goals, the final important facet of the execution phase is to measure goal accomplishment via qualitative or quantitative metrics. The old cliché “what is measured matters” comes in to play here, as development teams will begin to adjust their development approaches based upon what security weaknesses are measured. For example, if an organization measures software quality by bug count, counting vulnerabilities by OWASP Top 10 vulnerability types might be an attractive approach. Make what results you have internally public, and capture positive case studies that highlight positive behaviors. Should a development group use threat modeling in a way that prevents a serious vulnerability, tell that story to other groups and quantify the cost that the vulnerability might have been to the organization, had it been released to the world. Bottom line: Show quick wins, highlight positive behaviors, and do it over and over again, ratcheting up expectations and software security with each iteration.

13

Sustainment Phase Once you’ve finished the execution phase of your initiative, which ideally will likely take between one and two years, a Sustainment Phase will inevitably take place. Ideally, the sustainment phase should not exist until many, if not most, of the key facets of software security have been adopted by your organizations. However, in practice, the execution phase may take longer and include various components, all of which will probably need some level constant care and feeding. This is why defining success criteria up front is so important. Managers will expect this from you. Be cognizant of the fact that this initiative is a business process reengineering effort, and as such will not be conducted in one fell swoop.

Monitor Regulatory Environment Changes In the sustainment phase there are several key operations that need to occur. First, a regular, disciplined update of the regulatory framework must occur. Statutory laws and administrative code are not static. They are constantly being updated and need to be reviewed on a regular basis to determine if any new, or different, regulations apply to your organizations. Engaging a broad set of stakeholders including IT, HR, legal, and audit managers to confirm regulatory environment changes will help ensure an important update is not missed. These stakeholders are likely to keep closer tabs on regulatory updates on their areas of expertise, thus ensuring depth as well as breadth in compliance matters relating to applications. Finally, by keeping open lines of communication you will have the added benefit maintaining a pulse on compliance issues that are driving the business. I would recommend this process occur on an annual basis, at a minimum.

Identify New Tools to Improve Software Security In addition, an organization should continue to keep an ear to the ground in the commercial tools arena, as rapid innovation is changing how organizations measure and manage the risk of custom applications. The marketplace for static and dynamic assessment software has evolved dramatically, and new approaches, such as software as a service, may largely change the way we assess applications in the future. This is but one example of the types of changes occurring in the market. You will be well served if you keep track of new, and better, ways to improve the security of software.

Observe New Risks within Your Organization Finally, you need to continue to observe what’s occurring within your organization to determine whether there are new technology risk areas emerging. For example, more and more critical business is conducted on mobile applications or via SaaS. If you focus exclusively on well-trodden areas (e.g., web applications), you are likely to miss emerging software risks in areas on the periphery of your focus.

14

(844) 572-4400 denimgroup.com | @denimgroup

@johnbdickson

1354 North Loop 1604 E, Suite 110 San Antonio, TX 78232

© Denim Group, Ltd. 2015 All Rights Reserved.