The quality of government websites in countries with the highest level of competitiveness Patricia Acosta-Vargas Facultad de Formación General Universidad de Las Américas - UDLA
Quito, Ecuador
[email protected]
Sergio Luján-Mora Department of Software and Computing Systems University of Alicante Alicante, Spain
[email protected]
Abstract—This article describes a study that assesses the accessibility of contents on websites of governmental institutions of 20 countries with the highest level of competitiveness. The accessibility assessment was carried out to verify compliance with the Web Content Accessibility Guidelines 2.0 (WCAG 2.0) published by the World Wide Web Consortium (W3C). The main goal of this study is to determine whether people with disabilities are able to access and use government websites. In addition, we considered the Website Accessibility Conformance Evaluation Methodology (WCAG-EM) to determine how easily a website can be accessed according to WCAG 2.0. The WCAG-EM provides guidance on how to use the methodology and considerations for specific situations. From the results, we conclude that the majority of the websites do not achieve an acceptable level of compliance. The websites of governmental institutions with the highest level of competitiveness do not show a greater level of web accessibility. Several non-compliance errors were found on the websites; by performing this evaluation, we have identified that there are major barriers for a large number of users. Therefore, there is significant work pending to make the websites examples of best practice in e-government sites. Keywords— Disability, e-government, usability, government websites, WCAG 2.0, W3C, Web content accessibility, web accessibility.
I.
INTRODUCTION
Nowadays, the World Wide Web is used in many different ways, from searches on a variety of topics to accessing online services such as those offered by governments. The number of e-government websites has increased greatly in recent years because authorities have recognized the many benefits that online services may provide. This success has been based on Internet availability and related technologies. Nowadays, 4.3 billion people have access to the Internet [1]. According to Freeman & Loo [2], there are three categories of benefits that governments can achieve by developing websites for e-government: efficiency, user convenience and citizen involvement. It is important to remember that the level of accessibility determines the ease of admission to websites for the public, especially for those with disabilities. Web accessibility determines how users perceive, understand, navigate and interact with the Web [3]. In the beginning, the Web was purely a text-based medium; however, when it also started to be used as a commercial platform, its design and
Luis Salvador-Ullauri Facultad de Ingeniería de Sistemas Escuela Politécnica Nacional - EPN Quito, Ecuador
[email protected]
visual appearance became important [4]. During that process, most web developers did not apply suitable designs for universal access; therefore, websites became less frequently accessed by people with disabilities. In recent years people have become aware of this problem and the term “Web accessibility” has become fairly common [5]. This paper presents an analysis of web accessibility following the Web Content Accessibility Guidelines 2.0 (WCAG 2.0) [6]. The analysis was applied to websites from 20 countries with the highest level of competitiveness. The accessibility assessment was carried out to verify compliance, considering that, if the home page is not accessible, users may have trouble accessing the other pages of the site. This preliminary study will alert government institutions to comply with accessibility guidelines and modify the design of their websites. It should be noted that there are some existing studies related to website accessibility. In the case of evaluating local e-government [7], sites developed by Greek local governments were analyzed to measure their efficiency, as well as their sophistication. This analysis revealed that most of the egovernment efforts are limited to information provision, without exploring the functions related to interactivity, transactions and more citizen-oriented services. A study on the accessibility, quality and performance of government portals and ministry websites [8] indicates that, although the number of government sites has increased rapidly in the past few years, their ultimate success will largely depend on their accessibility, quality and performance. The findings revealed that none of the sites evaluated met the recommended priority AA conformance level. A number of accessibility, quality and performance weaknesses were identified. The majority of the sites assessed failed to comply with the perceivable, operable, understandable and robust principles established by the WCAG 2.0. An article on quality evaluation of e-government websites in Turkey [9] found that developers have devoted their efforts to the quality of websites. The concept of quality is represented by criteria that cover service, user, content and use perspectives. The quality and reliability of an e-government
website is also an important factor for its successful implementation. The results indicate that there is an urgent need to improve the design features of e-government websites to make them more effective and user-centric. An investigation into how usable and clear the websites of European capitals are, from the point of view of German students adopting an e-government perspective [10], indicates that the main evaluation categories are easy to use: there is noise reduction and good quality is evident. This study was performed with an evaluation tool on websites of 17 European capitals. An analysis focused on measuring the quality of Malaysian government websites [11] highlights the overwhelming growth of information usage online. It is important to determine whether quality of information or website organization influences users to come back to a site. This study also assesses accessibility, usability, efficiency, security and other parameters. According to a case study in Kenya on e-government website user experience from a public value perspective [12], the success of e-government websites depends on how well they are perceived by the end users. User experience of egovernment websites is considered one of the most important factors generating success or failure on sites. The result of this research is that governments need to focus on all public value measures of public websites to increase usage and trust from users. A study of quality in government websites [13] also explores the webmasters perception and explanation of website quality. Despite the concept of advances in communication, there are surprisingly only a few studies on how webmasters perceive, experience and explain website quality or design issues. Studies on government websites, mobile websites of various countries and the experience of users while browsing suggest that the websites should be designed for all types of mobile devices and operative systems [14]. These sites must be accessible to all, regardless of their abilities, according to WCAG 2.0. This investigation confirms that there are also studies of web accessibility for higher education sites and studies of government sites in South America [15]. In both cases, the results indicate that the websites do not provide adequate levels of accessibility. The rest of this paper is structured as follows: section II explains some basic concepts about web accessibility and egovernments; section III presents the method and materials; section IV shows an analysis of the results; and lastly in
section V the conclusions and recommendations obtained in the analysis are presented. II.
E-GOVERNMENT AND WEB ACCESSIBILITY
A. E-government challenges Among the barriers to e-governance projects, there are bureaucratic processes, lack of accountability and transparency, lack of citizen participation [16], lack of trust, lack of resources, digital division, poor management and legal barriers [17]. According to the United Nations E-Government Survey [18], governments across the world have considered egovernments as a powerful means through which they can provide transparency and accountability, economic growth, social inclusion, knowledge sharing, skills development, new employment and better health and education services. Egovernments can stimulate the transference of innovative solutions, reduce extreme poverty and protect the environment [19]. E-governments provide faster access to information at lower costs to the citizen [20]. Website accessibility is the degree to which citizens and automatic tools can access web information [21]. The success of e-governance adoption by citizens depends on the quality of information presented on the sites and service, involving its design, security and ease of navigation. However, website quality from a user perspective is about usability, which means the user experience in terms of finding information and completing tasks [22], the ease with which a novice navigates a website and effective website usage and content-related issues [23]. B. Web accessibility Web accessibility refers to the inclusive practice of removing barriers that prevent interaction with people regardless of their disabilities. Websites can be accessible to all if they are designed according to certain principles. These principles were included incorporated by the World Wide Web Consortium (W3C) into the Web Content Accessibility Guidelines (WCAG). WCAG 2.0 is divided into four general principles [24]: Perceivable: the criteria allow the product to be perceivable by people, regardless of their disabilities. 2. Operable: the user interface components and navigation must be operable. 3. Understandable: the information and the operation of the user interface must be understandable. 4. Robust: the content must be robust enough that it can be interpreted reliably by a wide variety of users, including assistive technologies. 1.
These principals include 12 guidelines. The guidelines are composed of compliance criteria, each following an adequacy level of conformity that indicates the impact of accessibility. Three levels are identified within this structure: Level A: the minimum level. Level AA: the middle level; the website must meet all the criteria under levels A and AA. 3. Level AAA: the highest level. The website must meet all the criteria: A, AA and AAA. 1. 2.
Level AA is the generally accepted conformance level for web pages. Similarly, for each guideline and criterion of WCAG 2.0, there are three techniques to be developed to evaluate web content: sufficiency, recommended techniques and common failures. Website quality standards are based on how well web tools and languages are applied in designing and developing the site. Detection of flaws and misuse of codes can easily be detected by automatic software tools in conjunction with human judgement, even though it may be time consuming and tedious as website quality and performance may not always be directly evident to the evaluator. Automated software tools can also detect errors in HTML and CSS; they generate warnings, check spellings, determine browser compatibility and check bad links and site performance. III.
level of competitiveness [25]. The home page of each website was analysed, as that is the most important in terms of accessibility. If the home page is not accessible, users might have trouble reaching other pages of the same website. On 27 March 2012, the W3C published the first draft of Website Accessibility Conformance Evaluation Methodology (WCAG-EM) [26]. This method proposed the evaluation of all types of static and dynamic websites, mobile versions, etc. This method is based on WCAG-EM, which considers the knowledge provided by experts as essential. The evaluator must understand web technologies, accessibility barriers, techniques, tools and evaluation methods to identify such limitations. The method is flexible and can be applied to different situations and contexts, such as self-evaluation, evaluation by third parties, evaluation during development, periodic evaluations, etc. The diagram presented in Fig. 1 shows the five phases applied to the evaluation of websites in this work.
METHOD AND MATERIALS
For analysis of this preliminary work, 20 samples from websites of government institutions were studied, as displayed in Table I: TABLE I: GOVERNMENT WEBSITES Country
URL
Switzerland
https://www.admin.ch/gov/de/start.html
Japan Germany United States Finland
http://www.japan.go.jp/ https://www.bundesregierung.de/
Netherlands
https://www.government.nl/
Sweden Israel United Kingdom Denmark Singapore Qatar Norway Austria Belgium China Malaysia
http://www.government.se/ http://www.president.gov.il/
Luxembourg
http://www.luxembourg.public.lu/
Ireland France
http://www.gov.ie/ http://www.gouvernement.fr/
https://www.usa.gov/ http://valtioneuvosto.fi/
https://www.gov.uk/ http://denmark.dk/es https://www.gov.sg/ http://portal.www.gov.qa/wps/portal https://www.regjeringen.no/ https://www.bka.gv.at www.eutrio.be http://www.gov.cn/ https://www.malaysia.gov.my
Description Portal of the government of Switzerland. Website of the government of Japan. Portal of the government of Germany. Portal of the government of the United States. Website of the government of Finland. Portal of the government of the Netherlands. Portal of the government of Sweden. Portal of the government of Israel. Website of the government of the United Kingdom. Portal of the government of Denmark. Portal of the government of Singapore. Website of the government of Qatar. Portal of the government of Norway. Portal of the government of Austria. Portal of the government of Belgium. Portal of the government of China. Portal of the government of Malaysia. Website of the government of Luxembourg. Portal of the government of Ireland. Portal of the government of France.
The analysis was focused on countries with the highest
Fig. 1. Diagram of website evaluation
This study measured aspects of the site such as the type of web page, the size, complexity, the technologies used and the knowledge of experts who designed and developed the site. Phase 1: Define the evaluation scope In the first phase, it is necessary to define the scope of the site and the pages to which the evaluation will be applied. Aspects and developed services, versions of it and parts of the portal should be documented. Furthermore, the adequacy level
(A, AA, AAA) [27] to be evaluated should be defined. This study applied the level of conformity AA. A list of web browsers used and support products with accessibility features must also be defined. Phase 2: Explore the target website The purpose of Phase two is to determine the features that are essential for a website. Within the site, there may be many pages and states with different styles, designs, structures and functions. These pages have probably been generated using different templates or were written by different people. In this phase, the expert reviews the website, in order to check for broken links, detecting the most important pages of the website. In addition, the expert also examines and analyses the web pages for key information. Phase 3: Select a sample In Phase three, the main page of each institution is identified. Previously the states of the websites that were not part of the sample are reviewed, acting as indicators to verify results. Phase 4: Audit sample In Phase four, each sample site is verified to check whether it meets the compliance requirements and the adequacy level AA of WCAG 2.0. The assessment tests the functionality of the processes, the introduction of data, notifications and interactions. Phase 5: Record results Finally, in Phase five the results are documented. This documentation is generated in each process to justify results and ensure transparency and a complete analysis. At present, there are several tools available for assessment and verification, as well as for automating the process of evaluating the accessibility of the web pages and checking the level of compliance. The majority of automated assessment tools are based on WCAG 2.0. The use of web accessibility evaluation tools is a widespread practice. Evaluation tools are heavily employed as they help reduce the burden of identifying accessibility barriers. However, an over-reliance on automated tests often leads to setting aside further testing that entails expert evaluation and user tests [28]. A. Tools for automatic review of web accessibility These tools can never replace the revision made by an expert in web accessibility. They should be used as a first step, but not as the only one. There are several tools available for analyzing web accessibility, such as [29]:
AccessMonitor: Online tool that enables WCAG 1.0 and WCAG 2.0 to be evaluated. It allows a page posted on the Internet to be reviewed or HTML code to be directly uploaded or pasted. It reviews the checkpoints one by one and also offers a score of 1 to 10. AChecker: Online tool that allows BITV, Section 508, WCAG 1.0 and WCAG 2.0 to be reviewed at the same time. It also allows HTML and CSS code to be validated. It permits the review of a page posted on the Internet or HTML code to be directly uploaded or pasted. Deque Worldspace: Online tool that allows WCAG 1.0, WCAG 2.0 and Section 508 to be checked. eXaminator: Online tool that evaluates the implementation of accessibility guidelines in HTML and CSS of page content, using WCAG 2.0 as a reference and qualifying the final result on a scale of 1 to 10. TAW: Checks WCAG 1.0, 2.0 and mobile. Online version available for downloading and as a complement for Mozilla Firefox. Web Accessibility Checker: Online tool that checks the accessibility of a website. Verifies various accessibility guidelines, such as BITV, Section 508, WCAG 1.0 and WCAG 2.0. It permits the review of a page posted on the Internet or the page to be uploaded to the tool. The results are organized into three types of problems: known problems, likely problems and potential problems. IV. ANALYSIS OF RESULTS The pages were analyzed using the TAW 1 tool on 10 September 2016 and the eXaminator 2 tool on 15 September 2016. A. TAW The web accessibility evaluation was performed with the Test of Accessibility Web (TAW), an online tool developed by the Unit Web Accessibility of Foundation CTIC (Centre for Information and Communication Technology). This tool allows the accessibility of websites to be automatically analyzed. It is also possible to select the level of analysis: Level A, AA or AAA, according to the WCAG 2.0. TAW comprehensively revises all the items and pages that comprise the website. There are three parts to the report: Part 1: Header shows the logo and the version of the standard Web Accessibility Initiative (WAI). 1
http://www.tawdis.net/
2
http://eXaminator.ws/
Part 2: Analyzed Web, displays icons alert on accessibility issues found.
the percentage of variability explained by the model corresponds to 10.7%.
Part 3: Summary that includes problems found in accessibility, organized by priority, indicating the code of the verification point, description of the problem and line number of the page analyzed. The HTML tag generates the problem of accessibility. TAW automatically validates the accessibility of the site and it indicates the checkpoints to be tested manually. Table II contains the extracted numerical data analysis of the main problems, warnings and unverified points. The governmental institution website with the greatest number of problems is China, followed by Finland and Norway. The government websites with the fewest problems are the United Kingdom and the Netherlands, followed by Austria. The Finland website presents a high number of warnings, followed by Malaysia and Japan. The websites of governmental institutions that have the fewest warnings are Israel and Belgium, followed by Ireland.
Fig. 2. Pearson correlation, problems and warnings
Fig. 3 shows the Pearson correlation with the variables problems and unverified points. It can be observed that there is a weak negative correlation between these two variables. The coefficient of determination indicates that the percentage of variability explained by the model corresponds to 5.3%.
The United Kingdom website presents a high number of unverified points, followed by Finland, Switzerland, the Netherlands, Luxembourg, the United States, Belgium and Israel. The websites of government institutions that have the fewest unverified points are Singapore, Germany, Qatar and Austria. TABLE II: ANALYSIS OF PROBLEMS, WARNINGS AND UNVERIFIED POINTS USING TAW
Country Switzerland Japan Germany United States Finland Netherlands Sweden Israel United Kingdom Denmark Singapore Qatar Norway Austria Belgium China Malaysia Luxembourg Ireland France
Problems 20 27 13 5 71 0 22 3 0 12 57 10 64 1 2 108 11 8 9 50
Warnings 329 725 330 62 1496 189 410 1 362 150 433 238 337 123 16 195 1163 159 30 192
Unverified 18 17 15 18 18 18 16 18 19 17 14 15 17 15 18 16 17 18 16 17
Fig. 2 shows the Pearson correlation with two variables: problems and warnings that indicate web accessibility errors. These variables show that there is a moderate positive correlation between the problems and the warnings. The coefficient of determination, called R-squared, indicates that
Fig. 3. Pearson correlation, problems and unverified
Fig. 4 shows that there is a very weak positive correlation between the warnings and the unverified points. The coefficient of determination indicates that the percentage of variability explained by the model corresponds to 0.52%.
Fig. 4. Pearson correlation, unverified and warnings
The three cases show that there is no linear association, meaning that the linear model does not explain the relationship between quantitative variables. B. eXaminator This is an online service that automatically evaluates the accessibility of a web page, using as a reference the recommendations of WCAG 2.0. eXaminator produces a score between 1 and 10 as an indicator of the accessibility of the pages and provides a detailed report of the tests performed. A higher score towards ten indicates that the website is more accessible. Table III shows the websites with their scores obtained using the eXaminator tool. Luxembourg has the best score, with 8.0 points. Sweden follows with 7.2 and Ireland achieves 7.0. The institutions between 6.9 and 6.1 correspond to an average acceptable level: Denmark, Belgium, France, Japan and Qatar. The government institutions that score less than six are Malaysia, Israel, Finland and China. It was not possible to apply the eXaminator tool to Switzerland, Germany, the United States, the Netherlands, the United Kingdom, Singapore, Norway and Austria because the websites presented alert handshake failure. In some cases, the automatic review could not cover all the compliance criteria of WCAG 2.0; hence the manual evaluation requires an expert in web accessibility [30]. TABLE III: ANALYSIS WITH EXAMINATOR Country
Points
Elements
Luxembourg
8.0
348
Size (KB) 8.3
Sweden
7.2
738
13.4
Ireland
7.0
136
4.1
Denmark
6.9
310
6.3
Belgium
6.7
51
4.7
France
6.7
665
64.8
Japan
6.5
591
30.9
Qatar
6.1
616
19.6
Malaysia
5.2
774
11.6
Israel
5.0
7
1.9
Finland
4.9
1524
25.5
China
4.7
679
54
Switzerland Germany
Alert handshake failure Alert handshake failure
United States
Alert handshake failure
Netherlands
Alert handshake failure
United Kingdom
Alert handshake failure
Singapore
Alert handshake failure
Norway
Alert handshake failure
Austria
Alert handshake failure
Fig. 5 shows that 15% of the websites evaluated got either
7.0 points or higher: Luxembourg, Sweden and Ireland. Countries that obtained more than 6 and less than 7.0 points represented 25%: Denmark, Belgium, France, Japan and Qatar. Countries with less than 6 points represented 20%: Malaysia, Israel, Finland and China. Finally, eight countries were not analyzed with the eXaminator tool, because they presented alert handshake failure, corresponding to 40%. The result obtained in the analysis with the eXaminator tool shows the overall score for the websites. The parameters analyzed serve as a reference to correct the errors identified by an expert in web accessibility.
Fig. 5. Analysis using eXaminator
V. CONCLUSIONS AND RECOMMENDATIONS The results of this study show that all 20 government websites evaluated did not reach the acceptable level of accessibility with priority level AA. Therefore, there is a need to rectify the errors in order to be able to reach the accessibility level recommended by the W3C. It is necessary to include accessibility measures in the development of any website to correct all existing accessibility problems on every web page, which takes a long time. Web customization technology has been identified as a new and practical approach in ensuring return visits and adds efficiency and attractiveness to a website [31]. It is therefore suggested that policy makers should propose a legal framework to solve problems of web accessibility. The results can act as a benchmark to compare accessibility of government websites and portals. The findings can also provide insights into how to develop and improve governmental websites to make them more accessible for all people and particularly people with disabilities. The statement that university webs that apply accessibility policies is very subjective. This research may serve as an input for future projects with a greater number of evaluators. The assessment can be improved by giving weights to each
evaluation parameter. The identified barriers may hinder or prevent access to content for people who may have a physical or sensorial limitation. Web pages should offer several alternatives, different presentations to suit the needs of various users and should be replicable under different circumstances. This study has limitations and has taken risks in the planning stage, but can serve as lessons learned for future work. Future research may highlight better methods to implement the WCAG 2.0 guidelines into existing websites, in particular to develop more inclusive websites for all types of users. REFERENCES [1]
[2]
[3]
[4]
[5]
[6] [7]
[8]
[9]
[10]
[11]
[12]
[13]
International Telecommunications Union (ITU), “Measuring the Information Society 2014: Report 2014”, Retrieved from http://www.itu.int/en/ITU-D/Statistics/Pages/publications/mis2014.aspx, 2014. R.J. Freeman and P. Loo, “Web 2.0 and E-Government at the Municipal Level”, Proceedings of the 2009 World Congress on Privacy, Security, Trust and the Management of e-Business, pp.70-78, 2009. S. Luján-Mora, “Web Accessibility Among the Countries of the European Union: a Comparative Study”, Actual Problems of Computer Science, vol. 1, nº 3, pp. 18-27, 2013. C. Boldyreff, “Determination and evaluation of Web accessibility”, Enabling Technologies: Infrastructure for Collaborative Enterprises, Eleventh IEEE International Workshops, pp. 35-40, 2002. D. Ortner, L. U. A. Inst. Integriert Studieren and K. Miesenberger, “Improving Web accessibility by providing higher education facilities for Web designers and Web developers following the design for all approach”, 16th International Workshop on Database and Expert Systems Applications, pp. 866-870, 2005. W3C, “WCAG 2.0 Guidelines”, 12 11 2008. [Online]. Available: https://www.w3.org/TR/WCAG20/. [Last access: 02 02 2016]. P. Yannas and G. Lappas, “Evaluating local e-government: An analysis of Greek prefecture websites”, Digital Information Management, ICDIM '07. 2nd International Conference on, Lyon, pp. 254-259, 2007. W. Yaokumah, S. Brown and R. Amponsah, “Accessibility, Quality and Performance of Government Portals and Ministry Websites: A View Using Diagnostic Tools”, 2015 Annual Global Online Conference on Information and Computer Technolog (GOCICT), Louisville, KY, pp. 46-50, 2015. Y. Akgül, “Quality evaluation of E-government websites of Turkey”, 2016 11th Iberian Conference on Information Systems and Technologies (CISTI), Gran Canaria, pp. 1-7, 2016. M. Scholl, “How Usable and Clear are the Websites of European Capitals from the Point of View of German Students Adopting an EGovernment Perspective?”, 2016 49th Hawaii International Conference on System Sciences (HICSS), Koloa, HI, pp. 2893-2902, 2016. F. D. Saiful Bahry, M. S. Shahibi, Y. Kamis and M. N. Masrek, “Preffered information quality factors as a web content quality measures on malaysian government websites: A conceptual paper”, Technology Management and Emerging Technologies (ISTMET), Bandung, pp. 400405, 2014. G. Kamau, J. Njihia and A. Wausi, “E-government websites user experience from public value perspective: Case study of iTax website in Kenya”, 2016 IST-Africa Week Conference, Durban, pp. 1-8, 2016. Ølnes, S. “Small is beautiful or bigger is better? size of municipalities and quality of websites”, European Conference on e-Government: 389XV, Kidmore End: Kidmore End Academic Conferences International Limited. pp. 389-397, 2013.
[14] P. Giulianelli, C. Pons and C. Gonzalez, “Mobile government websites analysis”, 2012 7th Colombian, Computing Congress (CCC), Colombia, pp. 11-12, 2012. [15] S. Luján-Mora, R. Navarrete and M. Peñafiel, “eGovernment and Web Accessibility in South America”, First International Conference, Quito, pp. 77-82, 2014. [16] A. Rakan and A. Mohammad Yousef. “The Benefits Of Knowledge Management And E-Government in Raising Citizen Engagement Jordan Case Study”, Economics, Management & Financial Markets, pp. 213-220, 2014. [17] N. Rana, Y. Dwivedi and M. Williams. “Analysing challenges, barriers and CSF of egov adoption”, Transforming Government: People, Process and Policy, pp. 177-198, 2013. [18] United Nations E-Government Survey. “United Nations E-Government Survey 2014”, Retrieved from http://unpan3.un.org/egovkb/enus/ Reports/UN-E-Government-Survey-2014, 2014. [19] S. Mishra and R. Kant, “Social inclusion through Egovernance: A study of the beneficiaries of Uttar Pradesh (U.P.)”, Sumedha Journal of Management, pp. 128-136, 2013. [20] Asogwa, B., “Electronic government as a paradigm shift for efficient public services”, Library Hi Tech, pp. 141-159, 2013. [21] A. Abanumy, A. Al-Badi, and P. Mayhew, “E-government website accessibility: In-depth evaluation of Saudi Arabia and Oman”, The Electronic Journal of e-government, pp. 99-106, 2005. [22] J. Kim, and S. Lennon, “Effects of reputation and website quality on online consumers' emotion, perceived risk and purchase intention”, Journal of Research in Interactive Marketing, pp. 33-56, 2013. [23] D. Baker, “Advancing e-government performance in the United States through enhanced usability benchmarks”, Government Information Quarterly, pp. 82-88, 2009. [24] S. Luján-Mora, “Accesibilidad Web”,[Online], Available: http://accesibilidadweb.dlsi.ua.es/?menu=niveles-2.0, [Last access: 05 01 2016]. [25] K. Schwab, “The Global Competitiveness Report 2015–2016”, World Economic Forum, 2015. [26] Web Accessibility Initiative, “WCAG-EM Overview: Website Accessibility Conformance Evaluation Methodology”, Available: https://www.w3.org/WAI/eval/conformance, [Last access: 02 01 2016]. [27] Web Accessibility Initiative, “Selecting Web Accessibility Evaluation Tools”, Available: https://www.w3.org/WAI/eval/selectingtools, [Last access: 02 01 2016]. [28] M. Vigo, J. Brown and V. Conway, “Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests”, W4A '13 Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, New York, ACM, Article No. 1, 2013. [29] S. Luján-Mora, “Declaración de la accesibilidad”, http://accesibilidadweb.dlsi.ua.es/?menu=hr-revision-automaticas, [Last access: 23 10 2016]. [30] M. Vigo, J. Brown and V. Conway, “Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests”, W4A '13 Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, New York, ACM, Article No. 1, 2013. [31] M. Hashim, N. Noor and A. Hashim, “Star rating of web customization for Malaysia government portals/websites”, 2010 International Conference Information Society (i-Society), London, pp. 536-541, 2010.
