Improving Software Quality with Agile Testing

©2010 International Journal of Computer Applications (0975 - 8887) Volume 1 – No. 22

Improving Software Quality with Agile Testing Shubhangi A.Deshpande

Anand N. Deshpande

Manisha V. Marathe

Lecturer , PVG’s College of Engineering Pune. India

Principle Software Engineer Symantec Ltd. Pune India

Lecturer , PVG’s College of Engineering Pune. India

ABSTRACT Software engineering can be complex and hence has the risk of project delays, defective product due to time constraints that lead to the risk of loosing projects from the customer. To succeed it demands the use of IT methodologies. But off the shelf methodologies are not flexible. QA is an important part of software and ensures the quality of deliverable. Agile development completely redefines quality assurance work—from formal roles to day-to-day activities—making some traditional QA responsibilities and outputs irrelevant. In this paper, we describe application of Agile testing to software quality management that will ease development and testing. Also we will explain ho it helps in better planning and time management and how it enables clients to achieve improved coordination of their test resources with the agile development team by allowing automated tests to be developed in tandem with code development on the same set of requirements. We’ll summarize with existing agile testing applications, most notably by

describing how to effectively use professional testers and how to thoroughly acceptance-test a system that’s too large and complex for a single customer to specify and the future of agile testing. Agile development methodologies: Extreme programming, Crystal, SCRUM, Pragmatic programming, Feature driven development, Dynamic Systems Development Method (DSDM), XBreed

Keywords SCRUM, Sprint meeting, Test plans, User stories, Test scripts. TDD (Test Driven Development)

1. INTRODUCTION Agile testing is the close collaboration between the test writer and the developers to ensure test scripts can both be rapidly created and are robust and ensures flexibility and adaptability. It is an iterative process. We’ll begin with short description of Extreme programming and Scrum methodology.

1.1 Typical classical Test approach

G. V. Garje , Assistant Professor, PVG’s College of Engineering Pune. India

Test plan. Start reviewing requirements and specifications. Start writing test cases by test engineers. Wait for software, until it is almost too late to execute the test plans. Start test execution phase; solve problems found and 1st retesting. Involvement of test team in the project is very late. Classical approach doesn’t work in SCRUM model because there is no test team, no time to write test plans and do extensive retesting and there is no test manager within Scrum. Now let’s discuss on disadvantages and limitations of traditional model.

1.1.1 Waterfall Software Development In waterfall development environments, there are well-defined phases, each having a thorough review and authorization process that should be completed before moving on. On the surface, this process is not inherently bad. In fact, it’s quite logical and reasonable: First, figure out everything you need. Then design it. Then code it. Then test it. And repeat. Problems arise when the project requires adjustments and modifications that were not anticipated in the early stages. So, for projects that can be specified out with precision ahead of time, waterfall may be the proper choice. Unfortunately, history has shown that a calm, unchanging set of requirements is a rarity – but why? The answer varies depending on who you ask. Upper management blames the product team for not being thorough enough. Product blames sloppy coding by development. QA blames management for not enforcing tighter deadlines and cutoffs (―You signed off on the spec two months ago, we can’t add this feature now!‖) The real blame does not fall on any single department or manager, but rather with the basic nature of software itself. Though it may always seem like the fault of another department, the real issue is principle, not persona. In other words, the customer’s needs are always changing, and so too should your software. Looking back, it’s no wonder why these projects became the source of such frustration; basically, we delivered software that lacked the most important features! At the same time, we were up to our necks in fancy-looking documents - MRD, SRS, Architecture diagrams, APIs and Test Plans – that didn’t provide any real value to customers.

Test team is asked to test a project. Test Manager starts with Project Test Plan. Test Manager or Test Leads creates Phase 68


1.2 The Emergence of Agile In response to the problems highlighted above, development teams began shifting to frequent releases. At first, these iterative methods went with small cycles, but remained with a fixed schedule ahead of time (i.e. fixed budget, strict plan, and predetermined feature sets). So instead of the one major release, many minor releases would be scheduled. Agile would eventually take this concept one step further: Iterative releases, with no fixed plan! Here, each release adds a bit of value, allowing for better decisions as to what features would be included next. Development now works in small cycles – planning and designing only far enough ahead to keep the flow working. With this increased flexibility, the burden of excessive formal documentation was greatly alleviated. More importantly, we now incorporate testing into each step of the process, not just at the end of each release.

1.2.1

The Importance of Testing within Agile

If Agile is to succeed, testing must be a central pillar of the development process. The buzz surrounding this idea initially sounded like a dream come true: Cut out the wasted time of QA from the end schedule; move the delivery date forward and toss out the costly documentation. This, however, proved to be a fallacy. When executed well, Agile is not chaotic at all – in fact, it is are letting the developers run wild. But the truth is just the opposite. Any methodology that has programmers build unit tests before writing code is orderly by definition! It’s just a different kind of order – one that focuses on code rather than paper, and on value as opposed to blame. In a traditional project, ―everyone is responsible for quality,‖ but in agile projects, everyone actually writes tests. This includes all developers, business analysts, and even the customer. Having

everyone test offered several project advantages. First, it eliminates the team’s reliance on the single, assigned project tester, who would have otherwise constituted a major bottleneck. This proves highly important because—for reasons unrelated to the project— the tester might be replaced during different release cycles. Second, because developers are responsible for writing tests for each new feature, their testawareness increases and they can prevent or quickly catch more edge cases while they work.

Finally, because people know that they are required to test their specifications and code, they designed them for testability.11 That is, where appropriate, developers can add special features or design considerations to the code to make hard-to-test features testable. Such coordination tasks are often difficult and neglected when developers and testers work as separate teams.

1.3 Agile vs. Waterfall – It Isn’t One vs. the Other Although Agile is gaining the upper-hand in terms of popularity, this piece is not an endorsement of one over the other. Many of the tips we’ll discuss in this whitepaper are based in Agile concepts, but are relevant to both types of development

processes. So, if your organization uses traditional waterfall methodologies, you can still benefit from Agile concepts to improve quality. If you’re operating in an Agile development environment, or just want to adopt more real-time, real-world QA processes, here are ten tips that should enable you to improve efficiency and quality, while further incorporating QA into the development process. This will help break the cycle of QA being viewed as merely end-of-the-line testers and the bearers of bad news.

2. AGILE TESTING WITH EXTREME PROGRAMMING (XP) AND SCRUM 2.1 Extreme programming 2.1.1

XP practices: Test Driven programming, Pair programming, Short iteration and releases, Refactorings, User stories and Acceptance testing 2.1.1.1 The Role of Testing Testing is the headlights of the project. It indicates the current status of the project and where it is heading. Testing provides information to the team. This allows the team to make informed decisions.

2.1.1.2 Pair programming Each given task is handled by a pair of programmers. One programmer writes code while other one reviews code and highlights / comments the problems.

2.1.1.3 Test Driven Development A ―bug‖ is anything that could bug a user. Testers don’t make the final call. Testing does not assure quality. Find ways to set goals, rather than focus on mistakes. Developers write unit tests before coding. So it motivates coding, improves design (reducing coupling and improving cohesion), Supports refactoring. Many open-source test tools like xUnit have been developed to support this

2.1.1.4 Refactoring: It is a way to improve the Design of Existing Code. i.e., changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure. Make the simplest design that will work. Add complexity only when needed. Refactor as necessary. Refactoring requires unit tests to ensure that design changes (refactorings) don’t break existing code.

2.1.1.5 Acceptance Testing User stories are short descriptions of features that need to be coded. Acceptance tests verify the completion of user stories. Ideally they are written before coding. XP is exciting and will improve the practice of testing in the industry. XP developers are writing unit tests and unit testing tools. XP projects are finding new ways to build in testability and support automated tests.

69


2.2

SCRUM

2.2.1

What is Scrum?

Scrum is an iterative incremental framework for managing complex work (such as new product development) commonly used with agile software development.

2.2.2

Scrum Roles

SCRUM contains predefined roles and set of processes. Main roles are SCRUM master (who maintains processes), Product owner (who represents stake holders) and Team (ideal size of 7 people). The complete project is broken into small shippable product increment deliverables that can be tested at the end of each sprint.

2.2.3

How Scrum works

meetings, debriefings, interviews, questionnaire, previous project learnings document Conversational Test Creation – It tells who should write tests. Customers are often too busy. Defining tests is a key activity and it should include programmers and customer representatives. It should not be done be alone by tester. Coaching Tests – This is a way of thinking about Acceptance Tests. User stories should be mapped to tests. Tests provide - Goals and guidance, Instant feedback and Progress measurement. Tests are to be specified in a format - That is clear enough that users/customers can understand. That is specific enough that it can be executed. Specification by Example should be used. Providing Test Interfaces

The set of features that go in sprint come from product backlog (prioritized list of high level requirements). These backlog items are identified during sprint planning meeting. The team commits the tasks for the current sprint. During sprint no one is allowed to change backlog items. After the sprint is over team demonstrates the use of developed software. Testers also get actively involved in daily sprint meeting during which team discusses the current progress, any bottlenecks or impeded tasks. Testers keep testing in coordination with developers.

Exploratory Learning - Developers are responsible for providing the fixtures that automate coaching tests In most cases XP teams are adding test interfaces to their products, rather than using external test tools

So to make testing work effectively in Scrum organization, test activities and regression testing approach needs to change.

Look for bugs, missing features and opportunities for improvement.

Test team won’t work as separate team. Instead tester is part of each Sprint and asks questions, bring people together, does testing and communicates with Scrum team about issues and acts as team’s quality conscience. Tester uses risk analysis to identify test risks. Testing capacity is finite. Tester has to come up with a set of documents like High level test document (used for all projects), Test strategy and project test plan. The unit test case created by developer should include entry and exit criteria for each functionality tested and test process. At the end of sprint Sprint review meeting is conducted and testers also participate in this meeting. The customer issues are handled by prioritizing tasks, add issues to product backlog.

3. HOW CAN WE IMPROVE AGILE TESTING 3.1 Practices for Agile Testing To manage risks and provide full and timely information about any large scale project’s progress, the team needs to develop a set of metrics, consistently collect and analyze the data, and hold regular meetings and formal debriefings. Prepare quantitative and qualitative data for a predefined period. Quantitative data can be same as the existing team is using for decision making. Qualitative data includes recording of formal

Test Interaction Model – Human, Testers, Software, Product, Automation, Fixtures, Manual Testing, Test Descriptions and test interface forms test interactive model The team should plan to explore the product with each iteration.

The main thing is we don’t understand software until we have used it.

3.2 Improvement guidelines 1. Choose a realistic goal that

aligns with company strategy People often think QA’s goal should be to simply find all the bugs. Well, that’s not entirely realistic (Ok, it’s impossible). Rather, the goal of QA should be to improve the software. That’s hardly an admission of failure, as it focuses on the true realworld goal: producing better software. It also accepts the simple truth of quality, as brilliantly articulated by software testing expert James Bach: ―Testing is the infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous.‖

2. Unify SRS and test plans This tip requires that you define your testing strategy early on. This does not have to be a burden. Understand that an SRS is quite similar to a test plan. So team up, between Product Management and QA, and compile the two documents into one. From an optimization perspective, it eliminates the wasted process sometimes referred to as ―Test Plan Alchemy‖, in which separate test plans are created by merely copy-pasting from the SRS, and then search-and-replacing ―The software should do …‖ with ―Make sure that the software does…‖

70

©2010 International Journal of Computer Applications (0975 - 8887) Volume 1 – No. 22 It’s important to note that optimization also improves quality. QA experts know how to find errors. They also know that errors don’t just appear at the end, they appear throughout the process. Finding errors in logic or in usability is a must for Product Managers during the spec phases, yet their skill sets are often geared towards other areas. Looking at the unified SRS/Test Plan with a tester’s eye, therefore, adds tremendous value in the early stages.

3. Review your test plans Agile doesn’t mean ―no planning‖ or ―no documentation‖ so review your test plans to make sure they’re up to date. It may seem like an obvious point, yet it’s often overlooked. Overworked, understaffed QA managers are sometimes tempted to cut corners when it comes to reviewing existing plans, but here again, the merging of QA and PM tasks can alleviate this review bottleneck. With QA involved in the spec process early on, PM can return the favor in the test plan review process. Define your testing coverage matrix (don’t let it define you) : The multi-dimensional matrix of what needs to be tested should be defined early. The importance of this has grown exponentially the past few years, as the matrix has become increasingly complex:

example, if a requirements spec for a particular project is 120 pages, and 52 defects were collected relating to this spec, then the Defect Density is 0.43/pg. Similarly, you can measure Defect Density per kilo line of code or per object or module design. Containment: This measures the success of preventing errors from propagating. As we all know, the earlier a bug is identified, the lower the cost of fixing it. Measuring containment, therefore, can tell us where we can save money and time. Measure containment by counting the errors (i.e. mistakes caught within the step) versus defects captured in a later step downstream.

6. Self-measurement Some statistics, specifically the containment rate, require a certain degree of self-measurement. Using a technique such as Evidence-Based Scheduling, therefore, can save you a great deal of time, effort and money. You can find an excellent overview of this approach to Evidence-Based Scheduling 7. No broken windows allowed ―Fixing broken windows’ is a perfect metaphor for this issue. In the 1980’s, an experiment was run to measure the impact that broken windows have on crime in urban neighborhoods. Now, we wouldn’t normally think of broken windows as a cause of crime. But indeed, it was shown that just fixing broken windows in a neighborhood helped decrease the crime rate. The reason for this is that if you give the appearance that you don’t care, then others won’t care either. It’s the same with your testing efforts. If you continually say ―Ignore this bug, it’s a known issue‖ or ―Ignore that test, it’s not yet relevant‖, your QA team will begin to listen to you: They’ll be more willing to ignore bugs, and they’ll miss the ones that you don’t want ignored.

Figure.1 . Year Vs. Compliance Matrix Complexity.

4. Tell a story – not a use case Forget the words ―use case‖ and instead say ―storytelling.‖ In theory, it’s the same thing, but in reality it’s much different. To illustrate this difference, try searching for each of these terms on Google image. ―Use case‖ brings images of dry and intimidating flowcharts and diagrams, while ―story‖ brings friendly and colorful pictures. Testers will have a similar subconscious reaction: Ask a tester to build a use case and they will think in dry and uninspired terms. But ask them for a story, and you will see the creative juices start to flow. This way, you encourage them to be more aggressive and not settle for simple screen touching. It empowers the testers, without actually using the cliché word ―empowering.‖ Testing For Agile - Ten Best Practices for Agile QA Managers

5. Capture valuable data Defect density is one such valuable data Defect Density: Information that shows number of bugs in the context of the size of the particular project module or phase. For

8. Regression testing: Divide and conquer In our study project, regression testing was considered as global (rather than personal) overhead and allocated as a global time period. This works because regression tests are usually run on each iteration’s last day, with the testing work divided among team members. Figure 2a plots the net number of hours for running and maintaining regression tests in each iteration. As the figure shows, during the first two iterations, the variance between iterations was small enough to make this global approach the correct choice. The net time required of the entire team at each iteration was about half a day; this changed over time, owing mainly to substantial growth in team size in the third release. 9. Allocate bug-fix time globally The project also allocated a global time pool for fixing defects; experience and available data thus far indicates that this was a good choice for two reasons. First, estimating the time to fix a defect is extremely difficult 10. Testing is your feedback loop The testing process not only eliminates bugs, it can also serve as source of customer feedback. If you build a community of testers surrounding your product that truly represent your user base (in terms of demographics, geography, level of knowledge, etc.), you 71

©2010 International Journal of Computer Applications (0975 - 8887) Volume 1 – No. 22 can add feedback into the system before you even launch your products. This is especially important for public web and mobile applications, as opposed to turnkey custom projects. Feedback generated by a community of testers - whether through beta testing or crowdsourcing - can affirm (or refute) any preconceptions you might have regarding your software. You will get to see your product used under real-world conditions, something that often does not occur in many QA labs. Better yet, this information can be shared with other key decision makers in your organization, notably the sales and marketing departments, who’ll often spend great amounts of money on surveys and research.

11. Timing is everything The short release cycles of Agile development can often place QA teams under tight time constraints. Naturally, there are gaps in your staff’s schedules. The two day weekend is almost always a period of low development and testing activity, and this is a good thing – people need their rest to maintain productivity. However, this two day gap need not go to waste. By adding a crowdsourced community to your testing process, you can deliver software releases for testing each Friday and receive the results by Monday morning, thus compressing the development cycle without adding pressure to in-house staff. It’s one thing to be able to get a release out on schedule, but if you and your staff are working 60 hours a week to get it done, what’s the point? With crowdsourcing, there’s finally an easy, costefficient way to augment your in-house QA team. Measuring progress is also crucial; the project team tackles it by defining the project’s product size metrics as the number of regression test steps run at each iteration. The project’s leaders chose this approach for two reasons. First, because test size is more highly correlated with complexity than are lines of code or specifications, it’s a better approximation of product size. Second, such a metric sends a strong message to the team: only features that have full regression testing at each iteration are counted as delivered product size. Reaching and increasing the product size metric is a nontrivial team goal.

12. Easing bottlenecks: Code less, test more Dealing with testing bottlenecks is relatively straightforward: developers simply code less and test more. comparing the product-size metric points attained by tests that the entire team ran to the points attained by the team’s testers alone. The graph lines show the various phases the project went through. At first, only the tester was running tests. This became impractical within a few weeks as the software grew in size; at that point, most tests were handed over to other developers. The only iteration in which the total size decreased was when the tester was unexpectedly removed from the team (owing to an emergency in another project). However, the team recovered and took over testing the following week. Initially, the project tester didn’t work with other team developers but instead wrote crossfeature tests and automated the customer- defined use cases. The testing personnel manager—who wasn’t an XP team member— required this separation for two reasons.

First, common wisdom states that testers must be independent of programmers to prevent implementation details from affecting their test designs. This contamination argument maintains that testers can’t write adequate tests for a feature if a programmer has already explained the testing requirements. Second, one of testing’s aims is to verify that detailed specifications are up-to-date. So, testers traditionally write tests

according to the specifications, not to what someone has ―unofficially‖ told them. An unspecified feature is viewed as a kind of bug. These two claims form many QA professionals’ core objection to agile testing.13 The claims target basic XP principles—whole-team and developer testing, in particular— and insinuate that they’re unsuitable for producing quality software. The counterargument isn’t that these claims are false but simply that the alternative is better overall.

13. Activity planning Planning for quality activities involves time-allocation challenges in feature testing, regression testing, and repairing defects. 14. Integrate feature testing and coding In agile development, feature testing is always an integral part of each feature’s development and time estimation. Testing and coding time are usually equal: if feature A is estimated to require five hours of specifications and 10 hours of coding, for example, it would typically be allocated 10 hours for writing and running tests. This estimation can change on a per-task basis, such as when a team knows in advance that a given feature is more difficult to test than others or requires refactoring of existing tests. As a rule, features are timeestimated to completion. Feature A, for example, would be planned as a single, 25-hour task. This again stresses that no task is considered complete until its tests are written and running. 15. Defect management Defect management involves two major challenges: managing workflow and selecting and scheduling the defects to fix. 16. Use a team-centered defect-management approach Defect management workflow is much simpler in agile projects than in traditional ones in three ways: ■ Anyone can open a defect (once its developers declare that the defective feature is ―done,‖ it’s considered a defect). ■ Anyone can close a defect, after fixing it and running the related tests. ■ Anyone who finds a defect also assigns it to someone to be fixed. Team members routinely view their own defects list and fix the defects. If they’re assigned a defect by mistake—for example, a defect related to a subject they know nothing about— they can reassign it. This eliminates a sizable overhead from team leaders,4 who traditionally are responsible for this task. For example, the team leaders on two comparable traditional projects at the IAF estimated that they devote one to three hours per week to defect management, compared to zero for the agile project’s leaders. Team-centered, XP-based 72

©2010 International Journal of Computer Applications (0975 - 8887) Volume 1 – No. 22 development is key to enabling project members to successfully assign defect fixes to each other. XP’s wholeteam and sit-together practices require that everyone, including the tester, sit in the same room. The informative-workplace practice requires that everyone participate in the planning game and daily stand-up meetings, so people know each other and everyone’s respective knowledge areas. Given this, team members can usually identify the right person to fix a given bug. This doesn’t occur when the testing team is separate from the development team. The same practices also reduce false defect reports. Such reports either duplicate existing defect reports or are created owing to developer misunderstanding.

17. Fix defects as soon as possible Letting team members assign bug-fixing responsibilities requires that the average bug-fix time is short. This speed is crucial to ensure that precise load balancing of bugs among team members is unimportant. When a bug could require a day or more to fix, team leaders must assign such work to prevent bottlenecks and ensure that deadlines are met. In this case, the average bug-fix time remains relatively constant, even as the application grows significantly in size and complexity. This agrees with the flat cost-of-change curve, a central benefit of agile development. The average time is based on developer reports of the total net time they spent working on defects (developers complete daily reports on time spent on each task). The team achieved this very low average—an order of magnitude or more lower than most projects, including those in the same organization—for two reasons. First, there’s little overhead to reproduce defects because testers and programmers work on the same baseline. Second, team members fix defects as soon as they’re found, when the relevant code is still fresh in the developers’ mind. In traditional projects, a feature-complete version is sent to the QA group, which sends back a large stream of defects of varying importance. Because the effort to repair defects always greatly exceeds the available resources, a lengthy process of prioritizing defects begins. The team fixes the most urgent defects first, during time slots preallocated for this activity during project planning. Agile projects replace this process with a simple rule: fix every defect as soon as you discover it. This is required because the team releases software to its customers at every iteration, and releases shouldn’t, in principle, contain known bugs. In our case study, the rule was that defects should be fixed as soon as possible— this does not necessarily mean immediately, However, simply because the right person to fix a defect wasn’t always immediately available. In addition, team sometimes finds defects during regression testing, at the end of an iteration, that they can’t fix in time. Given this, project’s team devoted the beginning of each iteration to fixing the previous iteration’s open defects.

Fixing all defects as soon as possible has several advantages. First, defects require far less time to fix with this approach. Second, working on a clean and highly stable code base makes new development faster. Third, besides avoiding the overhead of prioritizing and planning defect fixes, this rule avoids unpleasant customer negotiations over which defects to solve. Such negotiation is often forced upon both sides when a project’s delivery date approaches and—because traditional development projects test late in the process—it becomes clear that not all defects will be fixed before the deadline. In the project, the team fixed all defects routinely and given its perceptions about product quality, planned no ―stabilization period‖towards the official delivery date. Another significant difference from common practice that often surprises traditional practitioners is that agile projects have no concept of defect severity. This relieves developers of both the pressure to stabilize the system just before the delivery date and the need to coerce the customer into dictating which defects to fix (and which to not fix). So, the only consideration regarding a given defect is whether to fix it. If the answer is no, the defect isn’t opened. If the answer is yes, it’s fixed when it’s cheapest to do so—right away.

4. CONCLUSION Agile testing is very effective and proven in small as well as large scale industries. The main advantage following agile process is quick adaptability to changing requirements of the customer. It increases discipline, confidence and satisfaction of developer and tester. By properly planning and applying above guidelines while working on a project, we can be sure to achieve productivity, good coordination and ensured quality of end product.

5. REFERENCES [1] International Software www.istqb.org.

Testing

Qualifications

Board,

[2] Agile Software Testing in Large Scale project, IEEE publication, July/Aug 2006, David Talby and Arie Keren, Orit Hazzan and Yael Dubinsky [3] ― Testers Should Embrace Agile Programming ―, Bret Pettichord , Quality Week, September 2002, San Francisco. [4] QFS, www.qftest.com [5] Scrum, www.controlchaos.com [6] TestComplete, www.automatedqa.com

73