Paper Title (use style: paper title)

9 downloads 236 Views 355KB Size Report
It is a forewarning system against .... that defines system and software requirements; development ... found by test team are stored in an industrial ticketing.
Applying EVM in a Software Company: Benefits & Difficulties

Pınar Efe

Onur Demirörs

Informatics Institute Middle East Technical University Ankara, Turkey [email protected]

Informatics Institute Middle East Technical University Ankara, Turkey [email protected]

Abstract—Earned Value Management (EVM) is a broadly used performance management and feedback tool for project management. EVM enables depicting the progress in terms of scope cost and schedule. In simplest form, Earned Value concept shows what we have in a project at a specific time against what we spent. Based on the trends, it provides the prospective progress of the project and enables forecasting actual cost at completion in addition to possible completion date. In spite of its widespread use in project management, software projects do not utilize it as much as the other industries. The main objective of this study is to identify the benefits of EVM for software projects as well as the difficulties of its application. Two exploratory case studies have been performed on two industrial software projects. In the light of the application results, the benefits and challenges of EVM implementation for software projects are discussed. Keywords-Earned Value Management, Software Project Management, Software Project Tracking, Performance Measurement.

I.

INTRODUCTION

Earned Value Management is a quantitative project management tool to measure and track the performance and progress of a project objectively. It integrates scope, cost and schedule dimensions of a project and enables tracking cost and schedule variance between planned and accomplished work of a given project at any time [17]. EVM allows management to predict the total cost at completion and the possible completion date based on the patterns and trends in the past. EVM is called as “Management with the lights on” since it sheds light on where the project is and where it is going comparing to where it was supposed to be and supposed to be going [18]. It is a forewarning system against possible cost overruns and delays in the schedule. Even though EVM is commonly accepted in project management world and has been employed to a wide variety of projects of different sizes and complexities around the world, software project managers rarely apply this powerful technique during tracking of the software projects. The tools, techniques and methods used in the traditional project management have been utilized on software projects for years. However, software projects have numerous differences with respect to traditional projects. In a general way, traditional project management approaches cannot be

sufficiently used for software projects [16]. Using tools, techniques and methodologies of project management in its basic form may not be so powerful without considering the challenges of software projects and may not meet the needs of software projects adequately. The adversities of applying EVM in software projects should be investigated and clearly observed in order to improve and tailor EVM for software projects. The objectives for this study are to conduct case studies with EVM and looking in detail to the application of EVM in software projects, observing the benefits and difficulties, gathering all the information useful for improving the EVM for software projects and identify the possible reasons for its lack of use. The paper is structured as follows. Section 2 presents the related research on EVM. Section 3 proposes the case studies, providing the background of the projects and results of the execution, as well as the problems encountered and deficiencies identified during EVM implementation. Finally, Section 4 summarizes the findings and prospects for future works. II.

BACKGROUND

A. Earned Value Concept The “Earned Value” (EV) concept has been used in industrial manufacturing starting with the early American factories in late 1800s in its most fundamental form. EV was formally introduced as a project management tool by US Department of Defense (DoD) in 60s. ANSI/EIA-748 Guide for EVM [1] was officially issued to the public in 1998. Project Management Institute (PMI) has published “Practice Standard of Earned Value Management” as a supplement to PMBOK Guide to facilitate its role in effective project management [17]. The usage of method has already been spread out to the private industry, other governmental agencies and the other nations such as Australia, Canada, and Sweden [12][2][3]. In 2000, a staged, 5-level maturity model, called as Earned Value Management Maturity Model (EVM3), is proposed to assess the capability of an organization in applying EVM [24]. EVM has two major key practices, which are establishing a performance baseline and measuring the performance against the baseline. It has three key data elements: Earned Value, Planned Value and Actual Cost. The last two are already available in our project plans and earned value also

can be extracted easily based on the plans. In the daily routines, the earned value concept is already used intuitively even without calculating explicitly. Planned Value (PV) is the sum of all the budgets for all planned work at any given time in the project schedule. It represents also established Performance Measurement Baseline (PMB), also known as Budgeted Cost of Work Scheduled (BCWS). The project performance is measured against PV. PV is typically plotted with S-shaped curve as cost versus time. Earned Value (EV), also known as Budgeted Cost of Work Performed (BCWP), is the value of the work progress at a given point in time. The earned value is expressed in terms of PV, representing the amount of work accomplished. Actual Cost (AC) is the summation of the resources expended and recorded in accomplishing all work performed for the time period. It is also known as Actual Cost of Work Performed (ACWP). These three key data elements basically ground a base for EVM. All the other EVM metrics including variances, indexes and forecasts are derived from these three basic elements. The variances show the project current status clearly comparing these data elements. The indexes are the indicators of how cost and schedule efficiently used and represent the trends of the progress in the project. Based on the fundamental principle that trends and patterns in the past determine the future, the indexes are used to predict future of the project and project completion metrics are predicted. Earned Schedule (ES) is an extension technique to EVM, which is introduced by Lipke to overcome schedule related deficiencies in traditional EVM method [15]. Lipke defined a new metric “Earned Schedule” based on the idea of using time instead of using cost for measuring schedule progress. The related schedule variance and schedule performance index are calculated based on ES. B. EVM in Software Projects The practice of EVM in software projects has been the subject of the researches in the literature due to the different characteristics of software projects and to increase the use of EVM for software projects. Fleming and Koppelman, who are the authors of the one of the fundamental EVM books [11], claim that earned value project management can be applied to software projects as it is applied to the other projects [12]. They do not point out any variances in the method. Instead, they present a general guideline to apply EVM on all projects successfully and propose “Ten Musts to Implement Earned Value on All Projects”. They state that when those ten earned-value “musts” followed, the critical fundamentals of the earnedvalue concept are captured and the management of all projects, large and small, from any industry is succeeded. Ferle [10] focuses on implementing software project management and underlines the factors affecting project success. He mentions the difficulties specific to software project management. He also asserts that implementing EVM on software projects is not trivial and investigates the complexities in planning, monitoring and reporting of software projects in the view of EVM. He states that

estimating the efforts of tasks in software projects is extremely difficult, even impossible in certain situations and explains various reasons for this uncertainty and also stressed out that one of significant difficulty of the software estimation is totally being human oriented. The effort is very rely on the skills of the developers. Brooks [7] show that the ratio between the best and least developer may be as much as 1 to 10. Also, the significance of initial estimate, no matter how accurate or inaccurate, is emphasized. In monitoring, the subjective assessment of task progress makes the visibility of progress challenging [10]. Hanna discusses the challenges specific to software management and their effects on EVM [13]. He classifies these challenges under three categories as “Innovation and Prototypes”, “Defect Discovery and Resolution” and “Architectural Changes”. He also proposes several approaches to make a software project more appropriate for an EVM and then explains how these solutions could be applied to the challenges of applying earned value management to the development of software applications. He provides a comprehensive overview of the challenges, which are mostly related to a large degree of uncertainty, either in schedule, quality factors, or design issues of the software projects, and also the techniques, which help quantify the nature of the uncertainty and differentiate the uncertainty in project schedule. In 2001, Solomon initially introduces his own method Performance-Based Earned Value (PBEV), defined as “PBEV is a lean, cost-effective means of implementing EVM to minimize administrative costs and to focus on the big picture” [19]. In this study, he concentrates on the existing shortcomings of EVM and possible EVM improvement opportunities based on the best practices and lessons learned by the Northrop Grumman team in developing weapons system software. Later Solomon published details of PBEV in a book and in different articles [20][21][22][23]. There are several limitations of standard EVM discussed by Solomon [20]. Firstly, EV is a measurement of quantity, not quality, of work accomplished. A project manager should ensure that EV also measures the quality and technical performance of work products instead of just the quantity of work. Secondly, EVM addresses only the project work scope and disregards the product scope. Thirdly, EV is a derived measure. Consequently, its effectiveness to integrate technical and cost performance depends on reliability and accuracy of its base measures. Fourthly, although EVM is not designed to manage risk, EVM is perceived to be a risk management tool. Finally, EVMS does not require precise, quantifiable measures. It states that objective EV methods are preferred but management assessment may be used to determine the percentage of work completed [21]. In PBEV, those shortcomings of standard EVM are aimed to be handled. Mainly using CMMI, PMBOK Guide, EIA 632, IEEE 1120 and author’s experiences, the PBEV guideline defining activities step-by-step is established. The distinguishing feature of PBEV is its focus on the customer requirements. PBEV addresses the product scope and quality requirements. In the project plan, product quality

requirements in addition to scope requirements are incorporated. The performance-based measures of progress for satisfying product quality requirements are specified. They are the base measurements of EV. The progress is measured based on both product scope and quality requirements using those performance-based measures [22]. Additionally, EVM is used together with agile project management as Agile EVM [25]. Using EVM together with agile practices was first proposed by Steven H. Lett from Lockheed-Martin in 1998 [14] and then Agile EVM methodology has been evolved with the contributions of Alleman, Henderson and Cockburn [4][5][8]. Agile EVM is adaption from EVM using values defined in Scrum and it has a simplified set of earned value calculations. In their paper, Sulaiman et al. describes Agile EVM in detail, compares the traditional EVM applications with Agile EVM applications and explains the terms and hypothesis [25]. Also, they present the results of empirical validity test of the Agile EVM by applying it on two projects. III.

CASE STUDIES

Two exploratory case studies have been performed to apply EVM on software projects, to spot the difficulties of applying EVM in software projects, to assess the usefulness of EVM in software projects and to observe possible improvement areas. The following research questions are asked during these case studies: 1. What are the benefits of using EVM in software projects? 2. What are the challenges of applying EVM in software projects? 3. What are the deficiencies of EVM for monitoring software projects? To look into the answers of these questions, two different project types have been selected: The first project is the software product development project with iterative approach and ongoing for years and the second one is the software application development project with waterfall development methodology and has relatively short development time. Projects with different characteristics were selected to reflect different drawbacks in applying EVM and to provide extensive look to the possible issues. The challenges and difficulties explored during EVM application and the specific deficiencies of the method for software projects would bring into light the improvement opportunities. A. Case Study I 1) Organizational and Project Context The case project is the development of sale management tool that enables selling exceedingly complex hardware systems with many peripherals. The software development organization is an independent supplier with ISO 9001:2000 certification. The annual revenue is around 7 million Euros, employing close to one hundred engineers. The project is a product development project, started in October 2003 and is still on-going. It follows iterative and incremental development approach. Every major release of

the product is managed like a new project. The case study has conducted on the latest major release on the date it was conducted, from May 2011 to April 2012. In the project, in a major release, there are monthly-based minor releases, lasting approximately one month, and released at the end of every month. In addition to the high-level coarse grained major release planning, there is a fine-grained iteration planning for minor releases. There are three main teams in the project, definition team that defines system and software requirements; development team that is responsible for implementation of the requirements and test team that is responsible for testing activities. The definition team is located in another region and test team is a distributed team. Hence, the case study here only covers the project management of software development team and shows their performance. The development team consisted of 7 people, including software developers, team leader and project manager in average 5 years experience. The total development effort is given in Table I. The project is developed using organization-specific, very high-level modeling tool. The requirements specified by definition team are kept in a requirements database, developed by the organization itself. The changes and errors found by test team are stored in an industrial ticketing system. The development team has a close collaboration with both teams. TABLE I.

TOTAL EFFORT

Item

Effort (personhour)

Total Planned Effort for the new features

6341

Actual Effort for the new features

6665

Rework Cost spent in this Release

1704

Rework Cost for this Release*

1444

*. Includes later reworking of the new features

2) Applying EVM to the Case Project The case study conducted in May, 2011 after a major release completed. The project related data, basically project plan, timesheets and error reports were provided by project manager. The project data were kept in excel sheets, including the effort, responsible and timelines of the new features, as well as details of found errors including dates, responsible and effort spent to solve it. The effort data were mainly collected from these excel sheets and subsequently the conflicts in the documents discussed and clarified with project manager and team leader to gather data as accurately as possible. During the application of EVM, the effort (person-hour) was used as cost unit since it is in-house software development project and the main budget consideration is the effort of the developers. All projects are tracked in effortdriven way in the organization. We checked the project progress on every minor release using EVM metrics, so the month was used as time unit.

The following figure, Figure 1, displays the planned performance baseline of the major release.

Based on this data, EVM was applied in a different way. It is not exact implementation of EVM, since the errors have been performed later than the releases completed but we just want to have an idea on reworking effects of the releases. The results are displayed in Figure 3.

Figure 1. Planned Value

First, we implemented EVM on the project ignoring any rework effort including errors and changes; considering new tasks in every minor release and their completion (Figure 2).

Figure 2. EVM key elements for the new features

As a result of first EVM implementation, the picture for the new tasks was looking good. The tasks were mostly implemented on time by the developers assigned to them. The deadlines almost met without delay. The assigned tasks were in general implemented in the planned time interval. All minor releases were published on time including the planned features almost done. Nevertheless, after we checked the error reports, we discovered that these tasks had been significantly reworked again and again with the errors coming from system test and the other parties. In their process, during a minor release, system testers are already testing the completed features and change the task status to “completed” afterwards. After release, system testers perform release test, and additionally another test group performs additional level of testing in a more integrated way. The errors found before release were counted in the iteration and the fix effort by the developer was already counted in the feature implementation effort. Only the efforts found after release is considered as rework effort in this study. Then, we categorized the errors and their efforts for the releases to see how much effort actually spent for the tasks in the releases including later bug-fixing efforts.

Figure 3. EVM key elements including reworking

As seen in Figure 3, there is a major effort gap between planned value and actual cost in this case. We can interpret these two figures as the following by choosing a month randomly; according to first EVM results before reworking, at the end of December, the 8th minor release of this major release was available and the project seemed almost on track with a little cost overrun and schedule delay. However, in a few months, some changes occurred and errors found and it is understood that the earned value calculated at that time was not precisely correct; cost and schedule variances were significant. Even though for a specific task in this release, project manager thought it was completed at that time, afterwards it is observed that it was not completed and additional hours, corresponding to %20 of the initial effort, spent for the task later. B. Case Study II 1) Organizational and Project Context The case project is a middleware application which enables data collection from the various products in the product portfolio of the company in order to serve data for the different management applications. These applications are web based high-level applications used by technicians in service department to configure these products. The project is from the same software development organization that Case I conducted. The project development started in January 2012 and completed at the end of July 2012. Before January, the product management already prepared product backlog and system architects shaped the architecture. The project followed waterfall development methodology. The case study only covers the project management of software development team till the end of system test phase since the organization was responsible for these phases. There were three teams in the project, definition team that defines system and software requirements, development team that is responsible for implementation of the requirements and system test team that is responsible for

testing activities. The definition team was located in another region. The test team was distributed team in three different regions and only one of the test team members was colocated with the development team. This case study covers development teams’ activities during development phase and also provides an extension to the activities of development team in system test phase. The system test team performed tests during June and July and the development team already worked for bug-fixing in parallel. The development team staff consisted of 8 people, including software developers, team leader and project manager in average 6 years experience. During the development of the project, Service Oriented Architecture (SOA) and Java based, organizational web application development framework were utilized. The requirements were kept in requirements database as use cases and wireframes. The errors were stored in an industrial ticketing system. 2) Applying EVM to the Case Project The case study conducted in December, 2012. The project plans with Microsoft Project and additionally some excel sheets displaying progress were the main source of EVM application. The conflicts in the documents were discussed with team leader and project manager. During the application of EVM, the effort (person-day) was used as cost unit since the project manager already tracked the project in this way. The monthly project progress reports were used basically during EVM implementation. Therefore, the month was used as time unit. First, EVM was implemented on the project without including system test phase; just concentrated on development phase. At the beginning of the project, 4months development phase planned to complete the features of the application and during 2 months-system test phase, two additional developers will support bug-fixing and reworking. In this case, rework was already estimated and included project plans. Figure 4 shows the initial performance baseline of the project.

Figure 4. Planned Value for development phase

Then, EVM was applied to the project during development phase monthly. The development phase completion delayed one month and completed at the end of May. Table II shows the project effort monthly basis and

Figure 5 displays the picture of the progress in terms of time and the EV key elements. TABLE II. Efforts for (in person-days) Planned Value Earned Value Actual Cost

TOTAL EFFORT IN DEVELOPMENT PHASE

Jan

Feb

Mar

Apr

May

52.5

180

300

407.5

407.5

42.5

137.5

222.5

367.5

407.5

55

162.5

265

416.25

470

Figure 5. EVM key elements during development phase

In the first three months, there is a considerable gap between the planned and earned values. The main reason for this gap is the infrastructural problems. In the project, the organizational framework developed by another region of the company was used but learning curve for the framework was quite high for the team. Additionally, they encountered many problems regarding framework infrastructure. Due to these unexpected infrastructural problems, the project was very behind the schedule at that time and the estimations have become unrealistic. Also, there were some uncertainties in some work packages and their estimations were not so correct. They were in general underestimated. It is apparently seen from the table and figure that teams’ earned value velocity is quite high during last two months. It is the result of high overtime during April and May to catch the milestone. At the end of the belated development phase, the project seems that it is over-budget but on-schedule based on the EV metrics. After development phase, additionally EVM has been applied to the development team activities during 2 months system test phase (see Table III and Figure 6).

TABLE III. Efforts (persondays) Planned Value Earned Value Actual Cost

TOTAL EFFORT IN DEVELOPMENT AND TESTING PHASE

Jan

Feb

Mar

Apr

May

June

July

52.5

180

300

407.5

407.5

450

490

42.5

137.5

222.5

367.5

407.5

450

490

55

162.5

265

416.25

470

553

603

Figure 6. EVM key elements during development and system test phase

In testing phase, there was a task scheduled for bugfixing and simply two developers were assigned to this task initially. There are 152 issues found by system test team and reported bug-fix time for those issues is 133 person-days during system test phase. The project manager stated that during system test phase, they needed to assign more developers than estimated and five developers worked on bug-fixing. The error ratio was much higher than estimated. During system test phase, EVM has nothing to reflect correctly. The features already implemented had been reworked during 2 months with more developers than estimated and no metric gives us any clue about the real progress or the status objectively other than low cost performance index. The low quality of the development phase misled project manager at the beginning of the system test phase and prevented to see the possible quality issues. C. Results & Discussion This section summarizes the observations including benefits of the method, challenges encountered and the deficiencies identified during applying EVM to these case projects. EVM is a beneficial for software projects like the other projects since it reflects the project status at a time point. For both case studies, in particular for new features, it demonstrates how the project is progressing. In case study II, during development phase, it obviously reflects the latency in the project beginning from early months and so give an opportunity to project manager to estimate this one-month milestone delay. However, there are also major problems of using EVM specially for reworking of the implemented features.

The first difficulty was related with data gathering for EVM. It is very important to have a mature project management process to apply EVM correctly. Since both case projects were already completed, the only sources were the available documents and the support of project managers. Therefore the backward data gathering for the actual efforts was difficult and not so obvious. The efforts for implementing new features, changes to the existing features and errors were kept in different sources and not so up-todate. Mostly, the documents were inconsistent with each other. Hence, it was not easy to gather actual implementation efforts precisely. If the project management process is more mature and EVM is applied during project execution, the case studies would have generated better results and may let us apprehend other difficulties of applying EVM. The most significant finding of this study is the effect of rework effort on EVM in the software projects. Reworking is a major issue and indispensable part of software projects. Based on the essential characteristics of the software projects and the quality related issues, reworking is in general accepted as natural consequence. For projects in other industries such as construction projects, it is not very common and acceptable in particular after some milestones. As a result this problem in most cases is not visible. In the first case study, in every release, there are some changes and errors for the features implemented in the previous releases. Those errors are coming from testers, external testers, sales groups, definition team, and product management and so on. Therefore significant effort is spent on errors on the previous features but it is counted in the current release. In that case, the gap between earned value and actual cost seems huge (see Figure 3). However, if we exclude rework effort of the existing features, the figure is quite reasonable (Figure 2) to observe the progress of the new features. In this project, reworking is not calculated or planned systematically, only some reserves are put for that effort since it is already accepted. However, this approach makes EVM results unacceptable, and the correct value earned at some specific time cannot be calculated. In other words, at a specific time, even though it seems 100% of a task completed, after sometime more effort needs to be spent for bug-fixing of the same task. Therefore previously calculated earned value is not correct or something wrong here. EVM does not represent the figure and the status of the project correctly. So, the quality aspect should be considered to estimate reworking in order to see the clear picture of the project. In the second case study, rework is already accepted at the beginning of the project based on the past-experiences and visibly put into the project plan, as bug-fixing. When the development phase is completed with 1-month delay, EVM shows us the project is over budget but on time. This is also another problem of the EVM for late projects which are already observed before by Lipke and Earned Schedule method is proposed to resolve this drawback of EVM [15]. During system test phase, much more effort is needed for bug-fixing comparing to original plan but based on the EVM metrics, we could not observe any sign before that happens.

There are major quality problems during development phase causing significant reworking. It considerably affects project’s progress but there is no systematic way to plan and measure it in the software projects. We conclude that applying EVM as it is could give software project managers wrong direction in both case studies. Even at a specific time the project is supposed to be on track, additional cost/effort would be needed for the features completed. Since it is known as rework costs may approach 50 % of the total software project cost, this fact should not be ignored and carefully considered [9][6]. The third challenge encountered during EVM application is the ambiguity of effort and value relation of a software task. In both case studies, EVM cost metric is considered as effort, which is person-hour and person-day since the organization tracks all their projects based on effort. It is also very common in software organizations to track projects, based on effort as the main cost driver is the effort. However, it is very subjective to quantify the value created for a specific effort. It is more or less known in the projects of other industries, for example it is very clear in construction projects how much value can be created in an hour for a worker. In software projects, the effort and size prediction models are not enough to state precisely the effort and value relation. There is no standard size measure used as a base to progress measurement in the industry yet. Is it Line of Code (LOC)? Is it Function Point (FP)? Is it Use Case Point?... Furthermore the effort estimates are still based on the expert opinions and usually are not directly related with the size that progress can be measured. Both case projects used expert opinions during effort estimation. Since the first one is already ongoing for years, the estimations are not problematic but Case II suffers a lot from underestimation, some tasks double the effort comparing to initial estimations. As a result, the 4-months development period is extended one month more. Additionally for software projects, it is unclear how to state the value of an ongoing task objectively and precisely and give an exact statement about that considering the fact that there is no objective and concrete size/effort measure. In Case II, it is observed that especially the huge tasks spanning more than a month may get lower complete percentage than previous month. It means after one month work, the completeness percentage is decreased. It basically points to the incorrect and subjective measurement of physical work progress. Additionally, it also brings us the problem of managing huge work packages. It is quite subjective and hard to decide work progress ratio in particular for such a big work packages and may lead to incorrect earned value calculations. IV.

CONCLUSION

This study presents the results of two case studies practicing EVM on software projects. We identified three significant problems while applying EVM on these projects. First of all, the mature project management process is fundamental during EVM application; otherwise EVM would not be applied properly and only produces incorrect values. Secondly, it is observed that the most considerable reason of the gap between planned and actual is reworking,

which is inevitable part of software projects considering the fact that the reworking cost may approach 50% of total cost. EVM is not dealing with any reworking specific issues. It must be taken into consideration in the software projects in order to see the project progress objectively. The last but not least issue is uncertainty of effort and value relation of a software task. There is no standard size measure of software projects. It is very subjective and human-being dependent. So, incorrect effort estimation is a fundamental problem. Also, measuring/estimating physical progress of the tasks properly and objectively is another challenge of the software project managers. EVM is a powerful technique to reflect project progress in terms of scope, time and cost. Basically it focuses on the quantity of the tasks rather than the quality. The quality is supposed to be implicitly included in the scope dimension. Even though it is correct for the traditional project management, for software projects the status is different. Implementing EVM as it is without considering the characteristics and challenges of software projects might mislead software project managers at a specific time. Though project manager thinks the project is on track, this quality issues and subsequent reworking efforts make project over budget and delayed in an unexpected way using EVM. Software projects have substantial rework and quality factors are affecting project progress significantly. To demonstrate the project progress correctly and to provide project managers more accurate estimation, EVM needs to consider quality factors in the method and so could tell more about the progress and performance of the project accurately. V.

REFERENCES

[1] ANSI/EIA -748A (1998), American National Standard Institute / Electronic Industries Alliance/ Standard for Earned Value Management Systems [2] AS 4817-2003, Australian Standard 4817-2003: Project performance measurement using Earned Value. [3] AS 4817-2006, Australian Standard 4817-2006: Project performance measurement using Earned Value. [4] Alleman, G. B. (2003). Project Management = Herding Cats, A Field Report, Agile Project Management, PMFORUM, PMFORUM.ORG, February [5] Alleman, G. B. & Henderson, M. (2003). Making Agile Development Work in a Government Contracting Environment – Using Earned Value to Measure Velocity, Agile Development , Salt Lake City, Utah, June 25 – 27 [6] Boehm, B. & Papaccio, C (1988). Understanding and Controlling Software Costs, IEEE Transactions of Software Engineering, 14, 1462 - 1477 [7] Brooks, F. P. (1995). Mythical Man Month: Essays on Software Engineering, Reading, MA: Addison-Wesley. [8] Cockburn, A. (2005). Crystal Clear: A Human-Powered Methodology for Small Teams, Pearson Education Inc. Upper Saddle River, NJ, USA [9] Dion, R. (1993). Process Improvement and the Corporate Balance Sheet, IEEE Software, 28-35.

[10] Ferle, M. (2006). Implementing Earned Value Management on IT Projects, 19th International Cost Engineering Congress, Ljubljana, Slovenia [11] Fleming, Q.W. & Koppelman, J.M. (2005). Earned Value Project Management, 3rd Ed., Newtown Square, PA, USA : Project Management Institute. [12] Fleming, Q.W. & Koppelman, J.M. (1998). Earned Value Project Management: A Powerful Tool for Software Projects, CrossTalk, The Journal of Defense Software Engineering, 4, 1923. [13] Hanna, R.A. (2009). Earned Value Management Software Projects, Proceedings of the 3rd IEEE International Conference on Space Mission Challenges for Information Technology [14] Lett, S. H. (1998). An Earned Value Tracking System for Self-Directed Software Teams, Proceedings of European SEPG 98 [15] Lipke, W. (2003). Schedule is Different, The Measurable News, 10-15. Retrieved from http://earnedschedule.com/Docs/Schedule%20is%20Different.pdf. [16] Plekhanova, V. (1998). On Project Management Scheduling where Human Resource is a Critical Variable. Proceedings of the 6th European Workshop on Software Process Technology (EWSPT-6), Lecture Notes in ComputerScience series, SpringerVerlag, London, UK, pp. 116–121 [17] Project Management Institute. (2005). Practice standard for Earned Value Management. Newtown Square, PA, USA: Project Management Institute (PMI). [18] Project Management Institute. (2008). A Guide to the Project Management Body of Knowledge (PMBOK® Guide). 4th ed. Newtown Square, PA, USA: Project Management Institute (PMI). [19] Solomon, P. J. (2001). Practical Software Measurement, Performance-Based Earned Value. CrossTalk, September, 25-29 [20] Solomon, P. J. (2004). Integrating Systems Engineering with Earned Value Management, Defense AT&L, 33, 42-46 [21] Solomon, Paul J. (2005) “Performance-Based Earned Value.” CrossTalk , The Journal of Defense Software Engineering, August, 22-26 [22] Solomon, Paul J. (2006) “Performance-Based Earned Value” CrossTalk The Journal of Defense Software Engineering, May, 2024 [23] Solomon, P. J. & Young, R. (2007). Performance-Based Earned Value, Hoboken, NJ:Wiley & Sons [24] Stratton, Ray W. (2006). The Earned Value Management Maturity Model, Vienna, VA, USA : Management Concepts Inc. [25] Sulaiman, T., Barton, B. & Blackburn, T. (2006). AgileEVM – Earned Value Management in Scrum Projects, Proceedings of AGILE 2006 Conference (AGILE'06), IEEE Computer Society, Minneapolis, Minnesota, USA