From Requirements through Risks to Software Architecture for Plan-based and Agile Processes James D. Kiper Dept. of Computer Science and Systems Analysis Miami University Oxford, OH 45056
[email protected]
Abstract The choice of software architecture of a system plays a vital role in its successful implementation and maintenance, especially for large, complex systems. An appropriate architecture must meet functional and nonfunctional requirements and must satisfy resource constraints. We advocate the use of a risk-based approach to making software architecture decisions. In this paper, we describe an adaptation of a risk management process and accompanying tools to this purpose.
1. Introduction The primary theme of this paper is that risk can be used to guide decisions about software architecture and design. Design of complex systems that include software is difficult. Some of the first decisions made in the design process, those concerning software architecture, are the most critical. These decisions shape the software design in profound ways. Changing these architectural decisions later in the design process generally requires extensive rework. The search for a good architecture has two dimensions. To be considered good, an architecture must match system requirements and must be capable of being implemented within the resources allocated to it. Use of risk-based reasoning about requirements and design allows software engineers and managers to make wise choices of software architecture that can fulfill both dimensions: matching system software requirements and meeting resource limitations. We have conducted our research in the context of spacecraft software development taking place at NASA and JPL, a context that favors plan-based development rather than an agile development method. However, we believe that the risk-based approach also has relevance in use of an agile development model. Kent Beck, one of
Martin S. Feather Jet Propulsion Laboratory California Institute of Technology 4800 Oak Grove Dr Pasadena, CA 91109-8099
[email protected]
the early proponents of extreme programming, an agile method, has written that “The basic problem of software development is risk.” [2] We take the view of agile versus traditional software architecture and design expressed by Boehm. [3] “But what is most needed is …efforts to synthesize the best from agile and plan-driven methods to address our future challenges of simultaneously achieving high software dependability, agility, and scalability.” Each has its own strengths and areas of appropriate application. We should find ways of adapting techniques from one approach to compensate for weaknesses in the other. To quote Boehm and Turner [5], “…both agile and planned approaches have situation-dependent shortcomings that, if left unaddressed, can lead to project failure. The challenge is to balance the two approaches to take advantage of their strengths in a given situation while compensating for their weaknesses.” A risk-based approach can help to provide this balance. Developing spacecraft (which is the primary business of JPL) is inherently a plan-driven activity. However, need for agility during software development arises even in this setting. Concurrent engineering is employed to meet aggressive schedules imposed by upcoming launch windows. Some of the spacecraft hardware itself may be new, a first-of-its-kind development, so its performance characteristics may not be known precisely until late in the development cycle. Finally, post-launch the hardware cannot be retrieved for repair or rework, so in the event of changes from the nominal expectations (unanticipated hardware characteristics, unanticipated environmental conditions, and the inevitable degradation to hardware from radiation, thermal swings, vibration, etc.), software is sometimes adjusted to compensate. We note that willingness to embrace change is one of the primary tenets of agile design [2]. The second principle of the agile manifesto is to “welcome changing requirements, even late in development.” [http://agilemanifesto.org/principles.html]
inspection of requirements, use of static code analyzers, adoption of coding standards, etc. The DDP tool is typically used in interactive sessions involving experienced engineers and domain experts to collect and maintain decisions about the system and design. The process used in these DDP sessions is explained in figure 2. [11] First the engineers and domain experts collect and weight requirements. Weights reflect the relative importance of each requirement to the overall system success. (For example, a guidance and control subsystem that can guide a spacecraft to its mission site is weighted more heavily than a small experiment that a scientist may want to add to the mission.) The engineers and domain experts use the set of requirements to determine associated risks. Risks are assigned an a priori likelihood of occurring. Each of the risks is scored against each requirement to indicate the impact that of that risk on a requirement. That is, a score Inchof 1 indicates that, if that risk Adaptive Milestone Milestone pebble SW occurs, the associated requirement risk-driven plan-driven ironbound Hackers XP development models models is completely lost. A score of 0 contract indicates that that risk has no impact on that requirement. If the score assigned is r where 0 < r < 1, Agile methods the meaning is that, if this risk Software CMM occurs, a proportion r of that requirement is loss. An example CMM would be that if the risk to the software controlling the spacecraft’s pointing at a particular star occurs, requirements to Figure 1. Boehm’s planning spectrum perform scientific experiments concerning that star will be lost, but other experiments may be unaffected. 2. Using risk to guide design After risks are determined and scored against requirements, the engineers and experts list activities that We begin by describing the risk-based design process can mitigate these risks. Each mitigation is scored with called “Defect Detection and Prevention (DDP)” and its respect to its ability to reduce each risk. Additional accompanying tool. [7] This process and tool have been information is also elicited about each mitigation, most developed and used at JPL to help engineers manage importantly its cost. choices in the large space of possible designs of After this information is obtained (typically over spacecraft and associated technology. several sessions), the DDP tool can add up the cumulative DDP has three first class concepts, instances of which impact of all risks, determine the total impact of a it captures and tracks: requirements, risks, and particular risk, compute the amount of requirements that mitigations. Both functional and nonfunctional are attained, compute the net benefit of a mitigation’s use, requirements are included, and both requirements on the etc. Since cost information is included for each artifact being developed (software for spacecraft), and on mitigation, managers can use the DDP tool to make the development process itself (e.g., schedule, budget choices about which mitigations to select. This has profile) are included. We use the term risk to mean proven especially useful with complex systems, since the anything that gets in the way of fulfillment of one or more total cost of all possible mitigations will typically exceed requirements. A mitigation is then something that lowers the budget allocated by a large amount. a risk, by reducing the risk’s likelihood and/or impact. The preceding description, and illustration in figure 2, For example, mitigations include unit level testing, indicate the major steps of the nominal DDP process as Boehm has identified a continuum of levels of planning, from hacking through XP and other agile methods to rigorously plan-based methods. (These are points on the planning spectrum of [4] that is reproduced in figure 1 below.) Some of JPL’s spacecraft developments would fall towards the “milestone plandriven” point in Boehm’s spectrum, while some would fall around the “milestone risk-driven” point. For the latter category especially, JPL has developed a risk-driven model to support early-lifecycle design decisions. This risk-driven model will be described in the next section. This paper is organized as follows: section 2 describes our risk-based design process and the accompanying tool developed to support this process; section 3 describes an adaptation of this process to explicitly capture software architectural decisions; section 4 briefly discusses the application of this tool and process to agile methods of design.
we have seen it used. Temporary deviations from this process do occur in practice. For example, when scoring risks against requirements, the experts may recall an additional requirement, so temporarily may return to the earlier process steps to add in that requirement and score it against the already considered risks. Determine and Weight Requirements
Determine Resources (Budget, etc)
List Potential Risks
List & Cost Relevant Mitigations
Score Risks against Requirements
Score Mitigations against Risks
Descope/ Reprioritize Requirements
Select Cost-Effective Mitigations
Revise Budget, etc.
Development Plan
Figure 2: Standard DDP Process The DDP process has been applied to spacecraft hardware, software and systems (including both hardware and software). Engineers from the hardware field are used to thinking about risks of various technologies. These fields have much published data about component reliability. In the software field, less is known (or published) about typical software risks. To help in this area, we have preloaded a set of software risks into a DDP database. This is a taxonomy of software risk from the Software Engineering Institute. [6] With this initial seeding, the engineers and domain experts do not have to begin with a “blank slate.” (They are, of course, not limited to these software risks. They can down-select from this set to just those that they deem applicable, and can add others easily.) From observation of engineers and domain experts using the DDP process, it has become apparent that there
is some additional structure to the design process that we were not capturing. This observation has led us to propose that architecture be included as a first-class concept in DDP. In the next section, we describe this inclusion of architecture.
3. Risk based approach to traditional design and architecture. Obviously, some choices of software architecture are more appropriate for a particular system than are others. By this, we mean that one software architecture satisfies the functional and nonfunctional requirements better than some other architectures. In DDP terminology, we would say that this architecture reduces the system risks, resulting in a higher level of requirement attainment. In that sense, an architecture can be treated as just another a mitigation. However, several factors have led us to propose lifting architecture to the level of a first class concept in DDP rather than coding it as a mitigation: • In use of the DDP process and tool, it has become apparent that architecture and design choices also induce new risks and/or aggravate existing risks. For example, suppose that our engineers and domain experts have decided that either a pipes-and-filters architecture or a data-centered blackboard architecture [14] will be able to meet the software requirements for a system. As soon as a choice is made between these, a whole set of risks are induced. For example, the pipes-and-filters architecture has the characteristic (or risk) that an interactive user interface will be difficult to implement. A blackboard architecture has the risk that multiple simultaneous updates to the blackboard may cause anomalies. Note that the risk identified for the pipes-and-filters architecture is not generally one associated with the blackboard architecture, and vice versa. • In addition, it is apparent that the degree to which the detailed design achieves the architecture in turn influences how well that architecture mitigates the risks that motivated its selection in the first place. For example, a pipes-and-filters architecture imposes a simple communication mechanism among otherwise independent computational units, thus diminishing the difficulty of understanding a large, overly complex system. The less the implementation adheres to this architectural style, the less well it will diminish the risk of poor understanding.
Determine Resources (Budget, etc)
Determine and Weight Requirements
List Potential Risks
List Relevant Architectures
course, there may be multiple architectures that will reduce risk to acceptable levels. To make this step easier, we propose to seed a DDP database with some of the classic software architectural styles e.g., pipe-and-filters, repository, object-oriented, etc. ([1, 14]) These architectural styles can serve as starting points for the selection of an architecture. Engineers and designers may, of course, insert their own hybrid architectures. Controller
Score Risks against Requirements
Score Architectures against Risks
Active component of robot
Revise selection of architecture
Actuators
Select Software Architecture
Sensors
Feedback
Action List Potential Risks
List & Cost Relevant Mitigations
Score Risks against Architecture
Score Mitigations against Risks
Descope/ Reprioritize Requirements
Select Cost-Effective Mitigations
Revise Budget, etc.
Risk Mitigation Plan
Figure 3: Revised DDP Process
We propose to incorporate software architectures into the DDP process and tool as shown in figure 3. In this modification to the process, we first go through the process in a typical DDP session focusing on requirements, risks, and architecture rather than requirements, risk, and mitigations. In this pass, we are explicitly capturing alternative software architectures that will reduce certain risks, and will produce another set. Of
Environment
Figure 4: A control-loop architecture for mobile robots In the second pass through this process, the architectures from pass one become the starting point. In a sense, the architecture from this first step becomes an induced requirement for pass two. The better this architectural requirement is attained, the better riskmitigating effects that architecture will achieve. For example, if a pipe-and-filters style of architecture is only partially adhered to, then its beneficial risk-reducing effects will to some extent be compromised. This pass deals with architectures, risks, and mitigations. Note that risks are important only as links between requirements and mitigations and design. However, the software architecture is an important output of this process, not merely an intermediary. The incorporation of a notion of architecture rests in turn on some extensions to the requirements-risksmitigations model that underpins DDP. Recall that in DDP requirements are quantitatively linked to the risks that threaten them (the quantitative information indicating by how much an occurrence of a risk would detract from attainment of a requirement), and risks are quantitatively linked to the mitigations that reduce them (the quantitative information indicating by how much a
mitigation, if applied, will reduce a risk). To support the notion of architecture, we: 1. Extend the effectiveness links between mitigations and risks to allow for the possibility that adoption of the mitigation increases a risk. The way we use this for architectures is as follows: an architecture is encoded as a mitigation. The risks it induces (assuming those risks were not already present) are added to the DDP set of risks. Each such risk is given an a-priori likelihood of zero (meaning that by default it will not occur). Each such risk is linked to the mitigation representing the architecture so that selection of the mitigation increases the risk’s likelihood from zero to one. It will then be the responsibility of other mitigations to reduce such risks’ likelihoods. 2. Allow for the option to declare a mitigation representing an architecture to be equated with a new “induced” requirement. An induced requirement comes into play only if there is one or more mitigation selected that induces it. It has no intrinsic “weight” of its own (i.e., it doesn’t vie for importance with the standard requirements.) However, the extent to which that requirement is attained determines the effectiveness of the linked mitigation. If the induced requirement is only 50% attained (because of risks against it), then the mitigation will have only 50% of its declared effectiveness at reducing risks. The first of these capabilities is already present in “standard” DDP. It has been used to capture the effectiveness of traditional mitigations that exacerbate risks (e.g., for hardware testing, a vibration test may induce new problems as well as detect present problems). In the use we propose, it serves to encode not simply a mitigation, but a major design choice, that of an architecture. We have experienced some use of such mitigations in several of the spacecraft applications of DDP to date. The second of these capabilities has been prototyped within the DDP software, but has yet to see application in a real-life study. The aforementioned capabilities provide the means to implement the affects of architectural choices. Our proposal goes beyond simply encoding them within the existing framework – given their importance, we also wish to expose them as distinct concepts to the end user.
respond in real time. Here are the primary requirements for the architecture as described in [14]: 1. Must accommodate deliberative and reactive behavior. 2. Must allow for uncertainty. 3. Must account for dangers in the environment. 4. Must give the designer flexibility. In the DDP process each of these requirements would be given a weight to indicate their importance to success of the system mission. (Note that, as written, these are requirements on the architecture. We may assume for this example that these have been generated from corresponding system requirements. Also note that the fourth is actually a non-functional requirement.) Supervisor Global planning
Control Navigation Real-world modeling Sensor integration
Sensor interpretation Robot control
Environment
Figure 5: A layered architecture for mobile robots
3.1 Examples: To help illustrate this interaction between risk and architecture, consider the following simple example: a mobile robotics example as described in [14] and attributed to Marco Schumaker. The purpose of this software system is to control mobile robots. It must handle signals from external sensors and actuators, and
Let us assume that the engineers and domain experts agree that requirement #1 is the most important – with a weight of 100; requirements #2 and #3 are next most important with weights of 50 a piece; and that requirement #4 has a weight of 10. In the next step we would determine risks that may pertain to this set of requirements. Here is a partial lists of risks as given in [14].
Map
Captain
Pilot Lookout
Blackboard
may conclude that the blackboard architecture does reasonably well against all of the risks, that is, it is not particularly susceptible to those risks. However, the layers architecture is weak with respect to task coordination and flexibility; the control loop is weak in dealing with uncertainty. (These were the conclusions of Shaw and Garlan in [14].) In the DDP process, we would push our experts and engineers to value these linkages between risk and architectures quantitatively. For example, suppose that our experts conclude that the blackboard architecture is likely to do well in task coordination because of the central repository. Conversely, they decide that the lack of task coordination in the layers architecture is likely to be problematic. Table 1 illustrates the linkage data that these DDP session participants might produce in their analysis of these architectures with respect to the six risks.
Risks
Table 1: Risk – Architecture matrix
Perception subsystem
Figure 6: Blackboard architecture for a mobile robot
1. 2. 3. 4. 5. 6.
Task coordination Dealing with uncertainty Fault intolerance Safety Performance Flexibility
Note that some of these are written as negatives and some as positive terms. It is left to the reader to translate these into the form of risk statement although the meaning is obvious in each case. Next (or perhaps in parallel), we consider possible architectures. Let us consider three that are presented in the above reference. The first is the control loop architecture of Lozano [12] as displayed in the diagram of figure 4. Figure 5 diagrams a layered solution due to Elfes. [8] Shafer’s blackboard architecture ([13] is displayed in figure 6. At this point in the process, we ask our experts to score each architecture against the risks. For example, they
Task coordination Uncertainty Fault intolerance Safety Performance Flexibility
Architectures Layers Contr ol Loop 0.3 0.1
Blackboard 0.9
0.1 0.3
0.3 0.7
0.9 0.7
0.3 0.7 0.7
0.7 0.3 0.7
0.9 0.9 0.9
The numeric entries in table 1 are in the range 0 to 1, where 1 means that this architectural choice completely eliminates this risk and 0 means that it has no effect on that risk. This matrix has been kept small for ease of presentation. In practice, larger matrices are required, and the DDP tool allows such, and provides other views of this linkage data to accommodate scaling up [10]. At this point, the team of experts with the assistance of the DDP tool can select a tentative architecture. (We call this tentative since the entire process is iterative.) This begins the second phase of this DDP process. We begin by adding to the list of risks those risks associated with this particular architecture. The risks of the previous phase were those associated with the requirements regardless of which architecture is chosen. Here we add those design and implementation risks that are specific to this architecture. Note that many of these can be preloaded since the general risks and weaknesses of most architectural styles are well known. What stands in the way of design and implementing a system satisfying these requirements with this architecture? For a system that is intended to be highly interactive, a pipes-and-filters
Risks
architecture is risky. Conversely, an event-driven system may have much lower risk in this area. An objectoriented design (OOD) may be susceptible to adverse consequences of programming staff inexperience with OOD because of the well-known paradigm shift needed to move from procedural to OO design and programming. Given the set of this newly amended list of risks, we now ask our experts to identify those activities, i.e., mitigations, that we can perform to reduce the impact of, avoid, or completely eliminate these risks. For example, if we have chosen an OOD and have a staff that is inexperienced in this method, we may chose to give them additional training, or to augment their team them with more experienced designers. Each such mitigation has an associated cost. For example, additional training may require the purchase of training materials or tuition for courses. Hiring additional, experienced staff requires additional salary and benefit costs. Our experts now evaluate each mitigation against each risk to score its effect at reducing that risk. For example, the effect of experienced designers is likely to be greater against the risk of an un-experienced staff than is training of existing staff. These scores are recorded as illustrated in table 2. Numeric values in the range 0 to 1 are used again with 0 meaning that this mitigation does not effect that risk at all; 1 means that this mitigation completely eliminates that risk. Finally, this collection of information (risks x architecture, risks x mitigations) is combined with budgeting information to make decisions about which set of mitigations will achieve the system requirements using the tentative architecture and within budget and resource constraints. This is typically a complex decision given the enormous number of interactions among requirements (with their relative weights), risks (with their likelihoods), the tentative architecture, mitigations (with their costs), and linkages among these. DDP provides graphical displays of this information that helps the design team explore this complex trade space. An optimizer is available that uses simulated annealing to find near optimal choices of mitigations within a specified cost bound. Table 2: Risk – Mitigation matrix
Inexperienced staff Inconsistent requirements
Mitigations Hire Provide experienced OOD training OOD staff 0.7 0.9 0.0
0.1
Perform formal inspections 0.0 0.9
As mentioned previously, this is an iterative process. In these activities, it is common for the design team to discover additional requirements or learn of the
infeasibility of certain requirements (resulting in the need for descoping [9]). Additional risks of a particular architecture choice may not be apparent until very late in the process. Thus, the entire DDP process may be iterated to capture these changes. However, note that subsequent iterations are likely to be more efficient because of the leverage of information derived during previous iterations. The reader may be struck by the length and complexity of this process. We assert that this is the nature of the task, not a side effect of our process. Design of a complex software system is difficult.
4. Risk based approach to agile design and architecture. In [4], Boehm has identified a fundamental distinction between plan-driven and agile methods of software development. “The main difference is that agile methods derive much of their agility by relying on the tacit knowledge embedded in the team, rather than writing the knowledge down in plans.” He goes on to point out that a risk of agile methods is that incompleteness of this tacit knowledge may allow the team to “make unrecoverable architectural mistakes.” The DDP tool and process has made system design more agile, moving it toward a “milestone risk-driven model” in Boehm’s planning spectrum. [4] (See figure 1.) In the business of spacecraft software development, we will most likely never move to the extreme programming range of his spectrum. However, we do believe that the DDP tool can be used in a lightweight way to capture the requirements, risk, mitigations, and architectural decisions that accumulate over the lifetime of an agile development. In this paper, we have described a plan-driven approach using DDP. However, the DDP tool already has built into it some lighter weight functionality. For example, at each point that we discussed entering numbers to score impacts of risks on requirements, and mitigations on risk, the DDP tool allows a user to enter an asterisk to give a qualitative rather than quantitative indication of an impact. Obviously, some of the calculations that DDP performs on the numeric values cannot occur on a symbolic value. Nonetheless, this is an excellent forum for capturing important, but previously tacit team knowledge.
5. Conclusions and Status The primary argument of this paper is that risk can be used as an effective guide to architectural design decisions for a software system. We have arrived at this conclusion though observation of a risk-based process for decision-making in real-world design activities. As this DDP process has evolved over time, it has led us to the
point where we believe that architecture should be considered a first-class concept along with requirements, risks, and mitigations. We illustrated this process and these points through a small example – a mobile robot. The status of this ongoing work is that the DDP process and tool described in section 2 and figure 2 has been used is actual spacecraft technology studies. Our encoding of architectural design decisions and associated risks has not seen actual customer use yet. Additional information about DDP can be found at the Defect Detection and Prevention website: http://ddptool.jpl.nasa.gov.
6. References 1. 2. 3. 4. 5.
6.
Acknowledgements The research described in this paper was carried out at the Jet Propulsion Laboratory, California Institute of Technology and at Miami University, Oxford, Ohio, both under a contract with the National Aeronautics and Space Administration. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not constitute or imply its endorsement by the United States Government, the Jet Propulsion Laboratory, California Institute of Technology, or Miami University. Dr. Steve Cornford leads the DDP project at JPL. DDP technology has been developed with NASA's Software IV&V, Code R and Code Q support, managed in part through JPL's Assurance Technology Program Office.
7.
8.
9.
10.
11.
12. 13.
14.
Bass, L., P. Clements, et al. (1998). Software Architecture in Practice. Boston, Addison-Wesley. Beck, K. (2000). Extreme Programming Explained: Embraced Change. Boston, MA, Addison-Wesley. Beck, K. and B. Boehm (2003). "Agility through discipline: a debate." IEEE Computer 36(6): 44-46. Boehm, B. (2002). "Get ready for agile methods, with care." IEEE Computer 35(1): 64-69. Boehm, B. and R. Turner (2003). "Using risk to balance agile and plan-driven methods." IEEE Computer 36(6): 57-66. Carr, M. J., S. L. Konda, et al. (1993). TaxonomyBased Risk Identification. Pittsburg, PA, Software Engineering Institute: 78. Cornford, S. L., M. S. Feather, et al. (2001). DDP – A tool for life-cycle risk management. IEEE Aerospace Conference, Big Sky, Montana. Elfes, A. (1987). "Sonar-based real world mapping and navigation." IEEE Journal of Robotics and Automation 3: 249=265. Feather, M. S., S. L. Cornford, et al. (2002). Descoping. The 27th IEEE/NASA Software Engineering Workshop, Greenbelt, MD, IEEE Computer Society. Feather, M.S. & S.L. Cornford, (2003). “Quantitative risk-based requirements reasoning”, to appear in Requirements Engineering (Springer), published online 25 February 2003, DOI 10.1007/s00766-0020160-y. Available from: http://eis.jpl.nasa.gov/~mfeather/Publications.html Kiper, J. D. and M. S. Feather (2003). Requirements, Architectures and Risks. Second International Workshop “From Software Requirements to Architectures” (STRAW’03) at the International Conference on Software Engineering, Portland, OR. Lozano-Perez, T. (1990). Preface. Autonomous Robot Vehicles. New York, Springer-Verlag. Shafer, S. S., A. Stentz, et al. (1986). An architecture for sensor fusion in a mobile robot. IEEE International Conference on Robotics and Automation, San Francisco, CA. Shaw, M. and D. Garlan (1996). Software Architecture: Perspectives on an Emerging Discipline. Upper Saddle River, NJ, Prentice-Hall.
