eChallenges e-2014 Conference Proceedings Paul Cunningham and Miriam Cunningham (Eds) IIMC International Information Management Corporation, 2014 ISBN: 978-1-905824-46-5
Best Practices for Validating Research Software Prototypes - MARKOS Case Study Alicja LASKOWSKA1, Jesus GORROÑOGOITIA CRUZ2, Paweł KĘDZIORA1, Ilaria LENER3, Bartosz LEWANDOWSKI1, Cezary MAZUREK1, Massimiliano DI PENTA4 1 Poznań Supercomputing and Networking Center, ul. Noskowskiego 12/14, 61-704 Poznań, Poland Tel: +48 61 8582001, Fax: +48 61 8585954, Email: {alicja, kedziora, bartekel, mazurek}@man.poznan.pl 2 ATOS Spain SA, Calle de Albarracín, 25 28037 Madrid Spain, Email:
[email protected] 3 T6 Ecosystems srl, Via Genova 30, Rome, 00184 Italy Tel: +39 06 47823286, Fax +39 0647882798, Email:
[email protected] 4 University of Sannio, Piazza Guerrazzi, 1, 82100 Benevento, Italy, Email:
[email protected] Abstract: A common approach in research projects is to plan the evaluation and in particular the validation phase In the last months of the project. When done late in the project, validation cannot provide valuable inputs to the development of the research idea, because there is no space to bring about changes, and very often ends up as confirmatory tests. Software validation can and should occur since creating the first lines of code until the completion of the system in test. Thus, we propose the approach where research project can adopt the agile paradigm used by startups “release early, release often”, not only to development, but also to validation: “validate early, validate often”, to improve the quality of projects results on one hand and to be in line with potential users on the other.
1. Introduction The term of validation refers to the evaluation of the prototype from users’ perspective, and aims at answering the question: “Does the system meets users’ needs?” The validation process can provide valuable feedback in research projects when working with users on an ongoing basis. There is no sense in building a technically correct system that is not useful for the intended users. The increasing importance of user collaboration and involvement in driving innovation is recognized. Moreover, validation of research results becomes extremely important in the context of Horizon 2020. The work programme explicitly states that focus areas should support activities that “meet the demands from consumers, policy makers and other users of the results” [1]. However, a common approach in research projects is to perform the evaluation and in particular the validation phase during last months of the project. In such a case, very often, the validation takes a very soft form of acceptance tests, focusing on single aspects of the prototype, and makes the purpose of the activity ceremonial [2]. There are many reasons to postpone, limit or even omit validation. Ambitious plans and limited resources for testing in research projects result in an anxiety about showing immature system and expecting a lot of criticism. When done late in the project, validation cannot provide valuable inputs to the development of the research idea, because there is no space to bring about changes. Copyright © 2014 The Authors
www.eChallenges.org
Page 1 of 9
Therefore software evaluation, and in particular validation, can and should occur since the early development stages until the completion of the system in testing phase. This paper describes the validation approach defined and used in the MARKOS (the MARKet for Open Source) European Project [3], a FP7-ICT-2011-8 STREP project, which aims to realize a service and an interactive application, which provides an integrated view on the open source projects available on the web. The project focuses on functional, structural and license aspects of a software source code [4]. Many services available on the Web often provide textual-based code search. MARKOS differs from such services in the sense that it provides the possibility to browse the code structure at a high level of abstraction, what facilitates the understanding of the software from a technical point of view. It also shows relationships between software components released by different projects, giving an integrated view of the available open source software at a global scale. Last, but not least, MARKOS is able to track potential legal issues resulting from license incompatibilities, and provides argumentations for these issues to support developers in the search for alternative solutions to their problems. Along with research objectives, one of the MARKOS aims is to involve end-users in order to allow to practice its results in scenarios coming from industrial and open source communities.
2. Objectives The objective of this paper is to briefly present an approach for validating research software prototypes. This paper outlines the approach, highlighting its advantages and lessons learnt based on MARKOS experience. Holistic testing and validation requires a lot of effort. This is a very wide area where different approaches and standards are provided. However, such comprehensive and complex methods, are very time and effort consuming, and therefore are not feasible to be applied in a research project with restricted resources. The overall goal of the paper is to convince the reader that the presented approach is lightweight enough to be used in research projects, following agile development methodologies.
3. Background and related work The approach used in MARKOS project and presented in the paper was not defined entirely from the scratch. Instead, the authors considered and reused parts of existing research and industry methodologies, methods, practices and standards related to requirements validation [5], requirements reviews, prototyping, acceptance testing, beta testing [6], or user perceived service quality [7] and usability [8], considering their applicability for MARKOS software prototype validation purposes. The final approach was highly inspired by rapid prototyping ideas typical to design-thinking [9], with its “human-centered” ethos, as well as agile developments methods. The quality attributes were adopted from standard ISO 25010 [10] and methods for validating web portals, further tailored to the research nature of the project. Finally, some shared characteristic of participatory design and validation observed in the living lab methods [11] were derived, in particular the shared characteristic to involve users early in the process of validation in real-life environments and incorporating the validation results into products development and services. Particularly interesting is the FormIT method [12], evolving in spiral throughout three stages of product/service development: the design of concepts, the design of prototypes, and the design of the final system, where real-life environment validation is maintained through the process as much as possible.
4. Approach description Below the authors highlights best practices being the key characteristic of the approach adopted in MARKOS project for validating research software prototypes. The validation Copyright © 2014 The Authors
www.eChallenges.org
Page 2 of 9
process is meant to be lightweight and seamlessly integrated within the development life cycle. In particular it tries to adopt agile and start-up practices to research projects. It consists of several cycles and combines different validation activities throughout the development process (see Figure 1).
Figure 1: An overview of validation approach
The first and most important best practice of this approach is to start the validation very early and treat validation as a continuous process throughout the whole development life cycle. First validation activities should be started in the project conceptual phase when initial requirements are specified for most important actors. 4.1
Practice 1: Start Very Early and Follow an Iterative Process
Validation should start even before writing first line of code, and shall be drawn from the requirement elicitation and specification phase. When the validation starts before development, validation can evaluate whether or not there is a need for a particular feature. Once an implementation of the feature has been accomplished, validation can evaluate the implementation, but starting from the assumption that initial feature expectations were already satisfied. This approach is quite convenient, because it minimizes the risk of implementing features that do not satisfy end-users' expectations, reducing the risk of wasting resources on developing useless features. Funded research projects usually have defined the main concept of the solution at the proposal writing stage, which is further elaborated in the beginning of project, therefore, users can be involved very early in the validation of that concept, providing real-life context and helping in the specification of their needs and their prioritization. At this stage the proposed activities are a focus group workshop and a field survey. A focus group is highly efficient way for capturing background information from users and capturing user needs. In turn, a survey can be used as a mean to gather priorities on planned functionalities. Moreover, the validation needs to be done throughout the development life cycle of the system, because the system changes as new user needs arise. Gradually, the requirements give way to working prototypes. Following the agile paradigm, validation activity is meant to verify the implemented sets of requirements, materialized in the form of a series of prototype’ releases, and feed any findings back into the development process as issues to be fixed in the next cycles. The proposed approach foresees validation of several major Copyright © 2014 The Authors
www.eChallenges.org
Page 3 of 9
releases of the prototype during the project lifetime, every few months. The exact timing needs to be aligned with the project dynamics and resources. 4.2
Practice 2: Focus on Key Aspects
The overall goal of validation is to make the system usable in a broad sense i.e. “the user can do what he wants to do the way he expects to be able to do it, without hindrance, hesitation, or questions” [12]. However, a broad range of characteristics can influence the quality of system, as defined in industry standards (e.g. ISO 25010). From the broad range into account the purpose of usage of the software product, a narrow set of aspects has been chosen to drive the evaluation process. For a prototype of a service atop an information resource, or simply a software prototype delivered in SaaS model, such as MARKOS, the approach allows to cover, at least partially, the following aspects of the validated system: ● benefits - to assess the perceived value of a system and benefits it provides to users, ● functionality - to assess if the capabilities of prototype meet users’ needs, ● information - to assess if the content/information resource is of the kind the users look for, ● UI usability - to assess if the prototype meets expectations on the easiness and intuitiveness to access and use MARKOS features, ● performance - to ascertain the prototype meets basic performance requirements. Although all aspects should be kept in mind over the project lifetime, an attention should be given to different aspects at various project phases. For example, performance cannot be validated at the early project stages without a prototype. 4.3
Practice 3: Use Different Tools in a Flexible Way
As the aspects are quite different in nature several different tools need to be used along the validation process to check the system from different perspectives. In MARKOS case we decided to use five main types of validation activities: surveys, interviews (both individual and group discussions), usability workshops, field trials and comparative experiments (experiments comparing task execution with and without the validated system) as ways for collecting feedback. These tools are used in turns, on different project stages according to project needs e.g. interview and discussions are good at early stages while comparative experiments are more useful when prototypes are getting more stable, because in early stages it can struggle with many performance and usability issues, which should be solved in first place. 4.4
Practice 4: Conduct Validation in Different Environments
Evaluation carried out in different environments shall include: a lab/controlled assessment, where users are invited to laboratories or workshops organized under controlled conditions; a field environment assessment implying a real world context assessment where endusers use the prototype by themselves. Lab/controlled and field assessments are complementary. While field trials are more open and less formal, there is more space for completely new insights from external users. People using the system in their own environment behave differently compared to when assisted and under surveillance. In turn, lab/controlled workshops allow to perform more careful investigations on specific issues, aspects or components.
Copyright © 2014 The Authors
www.eChallenges.org
Page 4 of 9
One of the main challenges in validation is to identify and engage the relevant group of users, to provide feedback channels and to keep in contact with them. 4.5
Practice 5: Involve Small Groups of Representative Users
Proactive and pragmatic approaches for involving representatives from different user communities are also important. Validation should involve the main target user groups. The proposed practice is to consider small groups of users, but conduct several validation cycles. Such groups are easier to organize and their involvement is less expensive for a project budget. In a small group, it is also easier to keep good contact with all participants and therefore the feedback is more complete. Another challenge is the access to the relevant communities. The best approach is to have several partners having access to potential users of developed system. As all MARKOS partners employ developers, the project took the “dogfooding” approach involving all partners in validation. Thanks to this, validation is done in both industrial and academic communities and in several different cultural contexts. 4.6
Practice 6: Interact Closely with Dissemination
Dissemination and validation, in fact, interact with each other on the reciprocal exchange basis (see Figure 2).
Figure2: Interaction of validation and dissemination activities
The dissemination supports the full involvement of different stakeholders in the external validation process, addressing tailor-made messages. On the other hand, inputs collected from external validation help to refine the message of dissemination campaign. In this framework, the participation in dissemination events is an opportunity for both disseminate system value proposition and achievements and to gather useful feedbacks from external stakeholders. 4.7
Practice 7: Building and Validating the System Should Go Hand-in-Hand
The seamless integration within the development life cycle requires the exchange of validation findings in each cycle. Findings should be fed back into project immediately after the validation activity. Each finding reported to the development team should include Copyright © 2014 The Authors
www.eChallenges.org
Page 5 of 9
a good reasoning summarizing the comments from users and help understood their needs. It is also useful to identify and assign the role in a project that can implement the finding. It should be mentioned that the synthesizing of needs requires combining ideas from many points of view and a proper balancing between the original vision and the received feedback. Leveraging user feedback to validate the ideas, design and implementation does not mean that all comments raised by the users will be applied. The feedback needs to complement the scope of the project, adding value to an original vision of what the product should look like. Also the way to implement some findings is still under analyses.
5. Case Description, Initial Results and Lessons Learnt The approach has been defined and used in the MARKOS project. At the point of writing the paper, MARKOS project crossed the halfway mark and finished first validation phase, encompassing two cycles: concept validation and first prototype. The activities conducted so far included: a survey on the initial usage scenario, a focus group workshop on the conceptual stage, a pilot assessment and a usability workshop of the first MARKOS software prototype. Altogether almost 50 people, mostly with a technical background, have provided direct feedback to the MARKOS requirements and the first prototype. The group consisted of people from the MARKOS Partners’ organizations, but not personally involved in the MARKOS project, people from organizations in close relationship with MARKOS Partners, or completely external users involved thanks to dissemination activities. On the basis of the users’ comments a set of over forty actionable findings have been defined, influencing the development process of the MARKOS prototype. Most of the comments considered the functionality and usability of the MARKOS system. 5.1
Conceptual Phase – Validating the Demand and the Problem
The first validation activity was a survey based on four usage scenarios, namely “Software Search”, “Code reuse”, “License verification” and “User comments”. For each scenario, there were a few specific questions, asking technical details (different for each scenario) and eight general questions (common for all scenarios) related to the overall potential usefulness of such functionality. The specific questions were used to directly get users’ insights on particular design decisions. For example we asked users, which types of software entity they would like to search. The feedback was that all types are expected, but APIs, packages, interfaces, classes, and programs/applications ranked highest. The aim of general questions was to compare scenarios, their perceived usefulness to users and impact on the working routine and suggested improvements. The general questions included for example: “How long does it take to perform the described activity without MARKOS?”, “Please estimate how many times a year you would use a system providing the type of activity described in the scenario.”, “Which of the scenarios steps are the most important for you?” or “Is there any scenario step that you would like to remove or change?”. Answers to these questions helped the MARKOS team to identify the most innovative features and specify requirements priorities for implemented. Rankings of features included in each usage scenario have been created. Features ranked highest were assigned highest priority in the development plan. In particular, features which were initially scheduled to be released in the second version of the MARKOS system (e.g. ”Restrict the search to implementations that adopt non-reciprocal licenses” and “Find all implementations”) have been rescheduled to be implemented in the first year prototype. Users also provided some ideas for specific new features e.g.: a way to add some metrics or information related to the quality of the software implementation, or possibility to link some external documentation to the source code elements and aggregate information from other sources. Copyright © 2014 The Authors
www.eChallenges.org
Page 6 of 9
Another validation activity at the conceptual phase was a focus group workshop. To complement individual responses collected in the survey, the idea here was to get feedback in an interactive group setting where participants are free to talk with other. The workshop involved five developers external to project and consisted of two parts. The warm-up questions focused on current experiences of users on the area of searching and analysing open source software which MARKOS tries to consolidate and improve In the second part the MARKOS system main features and early mock-ups were presented to developers. Apart from confirming the need for an easy to use tool for developers for checking license compliance, the discussion resulted in two remarkable findings “great importance of social aspects” in analysing and selecting open source components for reuse and “the need for an aggregator of OSS related information and extensive authoring and dependency information in search/browsing”. As an outcome of the analysis of the focus group workshop, the MARKOS team has been considering and analysing the possibility to extend the functionalities related to the management of user content. Although the latter was considered outside the scope of the project, to introduce some social aspect to the system it was decided that MARKOS will build its “own” social asset i.e. an annotation framework in including comments database. In summary, users’ feedback at the concept validation phase is crucial to help validate the demand and the problem i.e. to identify what is the core functionality and make the important decisions on what to implement first. Moreover the findings help in requirements elaboration and design and implementation decisions on particular requirements. User can also provide many interesting ideas for new features. However, due to limited project resources, only crucial new features could be included in the development plan. To keep the proper balance between the original plan and the received feedback, in MARKOS, each new idea was discussed and analysed on project forum at technical meetings. In particular, general mechanism to provide user comments was considered sufficient for enabling user generated content and feedback on the software entities in MARKOS. 5.2
(First) Prototype Validation – Validating the Product
The first MARKOS prototype validation included a pilot field trial and a usability workshop. The pilot field trial was the first validation activity based on a running software prototype, involving eleven users with technical, research and business background from partner organisations. Users were asked to try the first prototype and provide feedback using the feedback channels created for the usual users, i.e. by email, an online survey or face-to-face. The most important finding obtained from this pilot field trial of the first MARKOS prototype was the need to improve and extend its content i.e. the knowledge base, by increasing the number of analysed projects added to the repository. It turned out that the very limited knowledge base did not allowed to recognize the potential of such service. The improvements also shall consider features of the user interface, especially, adding a help section and redesigning the prototype homepage. Moreover, the improvement of the integration of the license analysis tool and other MARKOS components was prioritized. Although users found the MARKOS license analysis tools to be one of its key features, majority of developers assessing those tools claimed they were too complex for regular use. The field trial outcomes updated the system features and development plan in the way that some of more advanced envisioned features of the license analysis tools were postponed or dropped. Instead, the remaining efforts of the project were decided to be put on simplifying and improving both the usability and look and feel of the License Analyser core features. Specific, design suggestions came from a usability workshop, which was carried out a few weeks after the pilot field trial with a goal to elaborate more on the usability issues the Copyright © 2014 The Authors
www.eChallenges.org
Page 7 of 9
users experience when using MARKOS. The suggested adjustments of MARKOS prototype portal included mainly a unification of the portal layout, adding short 3-step introduction, improving visualisation of the properties of software entities and unification of the look and feel of the license analyzer component. Changes have been accepted and added to the plan for the next development cycle. Also a major redesign of the License Analyser component was planned and usability testing workshops have been scheduled for the next validation cycle. As mentioned earlier in this paper, a validation is closely related to dissemination activities. During the pilot field trial, MARKOS attended two important events, which were beneficial for validation process in terms of involving new external users and collecting their feedback. At the Open World Forum 2013 [13], the project discussed possible community collaboration models and established initial contacts within OSS communities, such as OW2 community and Xen (Linux Foundation), as well as with other competitors on the market. MARKOS first prototype was also presented at the ICT 2013 Conference [14], where MARKOS met potential users from different areas such as SMEs, Industry, innovative organizations and developers, resulted in collecting good feedback on MARKOS features (e.g. GÉANT [15] project expressed an interest in MARKOS and License Analyser in particular). To summarize, prototype validation brings valuable and concrete insights for improving the solution. At this stage field trials and technical lab/controlled experiments (such as usability workshops) should go in turns in several cycles, and, building and validating the product should go hand-in-hand and be an iterative process, with some amount of validated learning [16] along the way. 5.3
Final Prototype Validation
Next prototype validation cycles will include more usability workshops focused on specific MARKOS components e.g. the License Analyser. While getting closer to the final validation, it is foreseen to use comparative experiments to evaluate the actual quality of use of MARKOS system. Such an experiment will compare results of different groups of people conducting the same task using MARKOS and other systems. It is also planned to have a case study of MARKOS usage in GÉANT community.
6. Conclusion This paper presents a pragmatic approach to software prototypes validation. The work presented in this paper was performed in the context of the MARKOS European research project aiming to support open source developers in community-based software development. The described approach was used to validate the prototype service of the MARKOS project, which provides an integrated view on the Open Source projects available the on web, focusing on functional, structural and license related aspects of software code. The presented approach can be used in research projects aiming at delivering prototype services and applications. The approach identified five important quality aspects for validating research software prototypes. It also highlights the need to start early and continue validation as a way to obtaining valuable findings for research, adopting the agile development philosophy to “release early, release often” to “release and validate early and often”. Early assessments combined with agile software development improve the delivery of software products that fulfil end-users expectations and reduce the risk of failure when achieving the user satisfaction. Even if these approaches are getting popular in the industrial software development, especially in start-up communities, they have not yet been widely adopted in research and innovation projects, as explained in the introduction. Copyright © 2014 The Authors
www.eChallenges.org
Page 8 of 9
The MARKOS example shows that adopting this philosophy allows to drive the development in the way to fulfil the real needs of users and enhance the overall quality of the system. In particular, early concept validation allows to set up the priorities for the development and extend or adjust the planned software features before the start of the actual development. Then validation activities help to set the priorities for the next development cycles. Several validation cycles targeted towards different groups of users also improve diversification of feedback sources and support community building process, which is an another important aspect of sustainability of the project results.
References [1] [2] [3] [4] A. [5] [6]
[7] [8] [9]
[10] [11]
[12] [13] [14] [15]
Horizon 2020 Work Programme 2014 – 2015, European Commission Decision C (2013) 8631 of 10 December 2013 M. Bolton, User Acceptance Testing – A Context-Driven Perspective, DevelopSense, 2007 The MARKet for Open Source - An Intelligent Virtual Open Source Marketplace. MARKOS project 2012-2015, http://www.markosproject.eu/ G. Bavota et all, The MARKet for Open Source: An Intelligent Virtual Open Source Marketplace Abran, et all, Guide to the Software Engineering, Body of Knowledge, ISBN:0769510000, IEEE Press Piscataway, NJ, USA, SWEBOK, 2004 Version M. R. Fine, Beta Testing for Better Software, ISBN 0-471-25037-6, Wiley Computer Publishing, 2002 Z. Yanga, S. Cai, Z. Zhou, N. Zhou, Development and validation of an instrument to measure user perceived service quality of information presenting Web portals , Information and Management, Elsevier, 2004 J. Rubin, D. Chisnell, Handbook of Usability Testing - How to Plan, Design, and Conduct Effective Tests, ISBN: 978-0-470-18548-3, Wiley Publishing, 2008 T. Brown, Design Thinking, Harvard Business Review, 2008 ISO/IEC 25010:2010(E), “Systems and Software Engineering - Systems and Software Product Quality Requirements and Evaluation (SQuaRE) - System and Software Quality Models,” International Organization for Standardization, Geneva, 2010. E. Almirall, M. Lee, J. Wareham, “Mapping Living Labs in the Landscape of Innovation Methodologies”, Technology Innovation Management Review, 2012 A.Ståhlbröst, B. Bergvall-Kåreborn, FormIT – an approach to user involvement. In European Living Labs - A new approach for human centric regional innovation, edited by Schumacher, J. and Niitamo, V.-P. Berlin: Wissenschaftlicher Verlag, 2008 The Open World Forum 2013, http://openworldforum.org ICT 2013, http://ec.europa.eu/digital-agenda/en/ict-2013 http://www.geant.net/Pages/default.aspx http://theleanstartup.com/principles
Copyright © 2014 The Authors
www.eChallenges.org
Page 9 of 9
