EuroAsiaSPI 2015
Software Development Practices in SMEs : an ISO29110-based Survey Annick MAJCHROWSKI – Christophe PONSARD – Jacques FLAMAND – Jean-Christophe DEPREZ
Ankara Turkey – 01102015
Centre d’Excellence en Technologiesde l’Informationet de la Communication www.cetic.be
Structure of the presentation • • •
Motivation of ISO29110-based survey Scope and design of survey Survey results • • • • •
• • • www.cetic.be
Characterisation of the participating SMEs Ranking of the reported problems Correlation between problems and SME characteristics Top of ISO29110 recommended activities Top of ISO29110 recommended tasks
Threat to Validity of the survey Comparison with other surveys/studies Conclusions and future works
Standard ISO 29110 • ISO 29110 - Systems and Software Engineering — Lifecycle Profiles for Very Small Entities (VSEs) • A framework for software implementation and project management in Very Small Entities (VSE) • Allows to assess the process maturity level of a VSE • Benchmarking against an “average” performance • Identify measures that maximize ROI • Become reference for certification in some countries
• Considers four profiles according to the maturity of the organization • Entry level-Basic level-Intermediatelevel–Advanced level
www.cetic.be
ISO29110 – Project Management • In ISO29110 definition, the project management process covers 4 practices: • Project Planning • Project Plan Execution • Project Assessment and Control • Project Closure • To allow to manage the projects tasks and to reach the project’s objectives • With expected quality, time and costs
www.cetic.be
ISO29110 – Software implementation • In ISO29110 definition, the software implementation process covers 6 practices: • • • • • •
Software ImplementationInitiation Software RequirementsAnalysis Software Architectural and DetailedDesign Software Construction Software Integrationand Tests Product Delivery
• For a new software product, or a new release of an existing one • According to specific requirements
www.cetic.be
Motivation of ISO29110-based survey •
Developing software is challenging for SME • •
•
SME evolve in an increasingly complex, changingandcompetitiveenvironment SME have to ensure the qualityin timeand in budgetto satisfy the customer
SME are faced to problems : cost/time overruns, failure to address functional or non-functional requirements •
•
customer is not satisfied failure of IT projects
Idea of conducting a survey •
What are the main problems leading to customer dissatisfaction?
•
What are the main problems leading to SW development team dissatisfaction?
•
Which kind of SME and how SME are impacted by the problems ?
•
Which are the main IT practicesinvolving the problems ?
www.cetic.be
Scope and design of survey •
Focus on software development practices (6 activities) and project management (4 activities) based on ISO29110 standard Self-Assessment Questionnaire
•
•
•
Provide recommendations to improve practices
Questions asked about 1) The characteristics of SMEs 2) The causes of customer and team dissatisfaction 3) The way the practices are performed • •
www.cetic.be
Identify unperformed practices that lead to the dissatisfaction of customers and teams Identify practices already performed in order to not recommend them
Characterisation of the participating SMEs •
•
Online survey started in December 2014 and ended in February 2015 (still accessible in https://www.cetic.be/Enquete-sur-les-pratiques) 36 participating SMEs based in Wallonia or Brussels Size repartition Sector repartition Other 16%
TIC 23%
Manufacturing 12% www.cetic.be
Banking 14%
Public 16%
Health 19%
Characterisation of the participating SMEs Used development methods
Most of SMEs have integrated RUP and SCRUM methods More than half of SMEs have IT projects of a relatively small duration, the mean duration is about 6 months
Project duration repartition More than 1 year 14%
Less than 1 month 11%
Between 6 months and 1 year 17%
Between 1 and 6 months 58% www.cetic.be
Survey results : Main problems encountered by SMEs •
How to succeed an IT project ? by Balancing cost and time and scope to deliver product quality (triple triangle constraints)
•
Questions asked from a set of 12 common problems leading to dissatisfaction of the customer and the development team •
•
4 possible responses : never, sometimes, often, always
Problems ranked by Severity Score (PSS) •
www.cetic.be
PSS = weighted sum over all answers used to establish the order of problem severity (weights for negative answers are 0 and 1 and for positive answers are 4 and 6)
Survey results : Reported problems ranked by severity Short problem description
Type
Delay overruns (>5% duration time) in the project execution
Customer
14%
47%
31%
8%
3,0
Inefficiency in tasks related to project management and SW impl
Team
11%
33%
56%
0%
2,6
Cost overruns (>5% initial budget) in the project execution
Customer
14%
31%
39%
17%
2,4
Customer satisfaction w.r.t. project documentation (ex:user guide) Customer Customer satisfaction w.r.t. project follow-up (ex : clarity of project status) Customer
25%
42%
28%
6%
1,9
11%
58%
28%
3%
1,9
Customer
3%
17%
58%
22%
1,4
Discussion with customer of adjustment to timing, cost or functions Customer
44%
31%
22%
3%
1,4
Many bugs reported by the customer
Always Often Sometimes Never PSS
Tasks not clearly defined for team
Team
3%
14%
61%
22%
1,3
Maintainability and capability of evolution of software Customer satisfaction w.r.t. project non-functional features (ex : performance, reliability,..)
Team
39%
42%
19%
0%
1,2
Customer
17%
72%
11%
0%
1,2
Team satisfaction w.r.t. development environment available
Team
33%
58%
8%
0%
0,9
Customer satisfaction w.r.t. project functional features
Customer
33%
61%
6%
0%
0,8
www.cetic.be
Global problem severity score for each SME • •
Benchmark of each SME against other participating SMEs Benchmark of each SME against the average GPSS (19.97) Benchmark of participating SMEs 60 56 50
50 43
40
37 32
30 26 20
20
17 13
10
19
15
24
22
26
16
15
15 8
5
10
15
24
21
18
20
21
14 4
20
25
24
9
3
0
• • •
17
11 5
0
24
22
25
2
1 30
35
40
The highest rank SME is 56 The two thirds of SMEs score are between 10 and 30 This confirms that the SMEs are really faced to problems and really need help
www.cetic.be
Correlation between SME size and reported problems • •
Smaller is the company, higher is the occurence of the 3 top problems SMEs with less than 10 people reports problems about 5 times more than SMEs with more than 50 people about cost overruns 5,56%
Plus de 50 employés
8,33% 8,33% 2,78%
Entre 25 et 50 employés
0,00% 5,56%
11,11%
Entre 10 et 25 employés
16,67% 16,67%
25,00%
Moins de 10 employés
19,44% 30,56%
Cost overruns Inefficiency tasks in project Delay overruns www.cetic.be
Correlation between project duration and problems • •
Based on the average of Global Problem Severity Score = 19,92 Impact of the project duration on the average GPSS more than 1 year
35,2
between 6 and 1 year
21,17
between 1 and 6 months less than 1 month
• • www.cetic.be
18,29
8
Longer is the project, higher is the reported GPSS Which is coherent with a higher risk
Top of the most recommended practices # Practice (ISO29110 activity) 1 2 3 4 5 6
Establish the description of customer requirements Document progress reports during project execution Control regularly project progress vs planning Establish project plan Organize project kick-off Formalize the delivery and acceptance of SW …. 9 Describe tests 10 Realise tests
• •
• www.cetic.be
ISO29110 recommanded activities 97% 92% 92% 81% 64% 56%
Activities performed by SME 77% 67% 82% 69% 87% 40%
3% 3%
100% 100%
The most recommended activity is about the requirements ; this activity is already performed by more than 75% of participants The next 3 activities related to project management are also widely recommended ; they are not yet performed by 20% to 30% of participants Testing practices seems to be under control
Top of the most recommended tasks # Tasks related to activities 1 2 3 4
5 6
35 36 37 38 39
Evaluate progress against the project plan Evaluate and track changes from the client Assign tasks to project team for analysis of software requirements Establish measures to correct deviations or to solve problems and monitor them Document or update the specification of requirements Validate the requirements specification with the customer and obtain approval …. Assign tasks to project team for delivery of software Establish or update the test cases and test procedures Perform unit testing and correct defects until success Perform integration testing and document the results in a report test Deliver the software according to the project plan
www.cetic.be
ISO29110 Tasks recommanded performed by tasks SME 75% 85% 75% 85% 75%
85%
75%
78%
70%
80%
70%
92%
8% 3% 3%
33% 100% 100%
3%
100%
3%
100%
Threat to Validity of the survey #
1 2
Threat of validity Sample of participants not representative Truth of the participants with positive answers
Mitigation actions to limit
Sample is representative for the French Community of Belgium Recommendations are sent to improve inneficient practices
3 Different people , different appreciation , different answers
Standardizing questions and answers helps to reduce the degree of subjectivity
4 Participants answer several times with different email addresses
IP filtering performed at the beginning of the test
5
Survey based on the Entry profile of ISO29110 with a selection of activities and tasks related to the reported problems ; 1015 minutes needed to answer Survey begins with the common encountered problems The Entry profile concepts fairly simple
Too long and complex questionnaire 6
SMEs not well familiar with the jargon of SW process standards
www.cetic.be
Comparison with other studies/surveys CHAOS GARTNER report (2012) report (2011)
SAUER study CETIC survey (2003) (2015)
Project outcome Successful Failed Challenged Key dimensions Time overruns Cost overruns Features coverage www.cetic.be
39%
25%
16%
20%
18%
25%
31%
10%
43%
50%
53%
70%
74%
44%
45%
92%
59%
42%
49%
83%
69%
57%
41%
100%
Conclusions : What can we learn from this survey? •
The main causes of dissatisfaction are confirmed : • • •
•
Additional cost Inefficient tasks Exceeding time limit
The size and duration of the project significantly correlated to the problems The main recommendations activities and tasks concern :
•
• •
•
www.cetic.be
The description of the customer's requirements The mastery of project monitoring
50% of SMEs are in trouble and need help
Conclusion and future works •
Our survey • • •
•
confirms the problems previously reported by others international studies : cost/delay overruns Shows some significant correlations (as size and project duration) Shows the poor maturity level of SMEs
Extend our survey to other countries and, based on the larger set of answers, see if the same conclusions are confirmed or need to be revised. Encourage you to complete the survey (about 15 minutes ) to receive quick win recommendations
•
• • www.cetic.be
Survey available both in French and English at this URL: http://survey.cetic.be/iso29110/survey.php . We warmly encourage you to spread the information to any SME you know that might be interested.
Aéropole de Charleroi-Gosselies Bâtiment Éole Rue des Frères Wright, 29/3 B-6041 Charleroi
Questions ?
Tel: +32.71.490.700 Fax: +32.71.490.799
www.cetic.be
[email protected]
Annick MAJCHROWSKI Senior R&D Project Manager
linkedin.com/company/cetic
Tel:+32.496.073.807
[email protected]
twitter.com/@CETIC www.cetic.be
