peer-review of reusable learning objects for ebooks in a peer-to-peer ...

3 downloads 3698 Views 743KB Size Report
goal of creating and delivering an online or offline course, ... These successes at creating a P2P ..... digital signature for the RLO and indicate if it is signed.
PEER-REVIEW OF REUSABLE LEARNING OBJECTS FOR EBOOKS IN A PEER-TO-PEER NETWORK ARCHITECTURE Rajendra G. Singh and Margaret Bernard Rajendra Singh Margaret Bernard Department of Computing and Information Technology The University of The West Indies Faculty of Science and Agriculture The University of the West Indies St. Augustine, Trinidad, W.I. [email protected] and [email protected] ABSTRACT It is a challenging task to take standards-based eLearning content from a client-server environment and place it on a Peer-To-Peer network to achieve similar goals. The client-server eLearning system architecture is designed for storing, sharing and delivering course content. In an initial pursuit of this challenge, a design for a Peer-ToPeer Reusable Learning Object Repository was proposed and prototyped. Now in this continuing work, the challenge of performing and managing peer-reviews for the content being shared on the Peer-To-Peer network is investigated, along with how to aggregate and sequence the content into a course for delivery. The solution introduces a scalable server design, mainly to maintain the integrity of peer-review data while leaving the eLearning content on network peers. From the prototype, the user can also now aggregate and sequence peer-reviewed Reusable Learning Objects into a course after which it is passed to a commercial compiler via scripting, along with preprogramed templates to produce eBooks. The result, is that this extended Peer-To-Peer prototype application demonstrates an end-to-end solution which can be used to manage and share peer-reviewed Reusable Learning Objects over a Peer-To-Peer network, and achieve the final goal of packaging and delivering shared content as online and offline eBooks. KEY WORDS EBOOKS, PEER-REVIEW, PEER-TO-PEER REUSABLE LEARNING OBJECTS.

and

1. INTRODUCTION A Teacher who may be an average Internet user and wants to produce or acquire eLearning content with the goal of creating and delivering an online or offline course, may consider that it is still a significant and challenging task, notwithstanding current pervasive client-server technologies which suggest that this should be easy to achieve. The Teacher usually has to select an eLearning platform in advance, then produce, and/or amend, and/or acquire from other users the eLearning content that is compatible with the platform and suitable for the course. While most client-server eLearning systems are promoting standard-based content creation, including the use of standards for assessment content, there are further challenges to produce, amend and acquire the assessment

content which would be relevant for the online or offline course that is to be built. This task is not trivial, even when the Teacher is one from a higher-level Institution where Information Technology (IT) resources are more readily available; including sometimes specialized eLearning IT support staff and repositories of content. We need to therefore consider making the task of producing and sharing eLearning content and the related assessment content easier throughout a global education system. In previous research, the idea of using Peer-To-Peer (P2P) network architectures instead of the traditional client-server architecture was considered, and the design of a P2P Reusable Learning Object Repository (RLOR) was presented by Singh and Bernard [1] to facilitate the sharing of content over a P2P network. In pursuing the P2P network architecture, the design needed to consider further a solution for rating the quality of the content, which exists in the form of Reusable Learning Objects (RLOs), without possible loss in integrity of the feedback data. It also needed to consider how selected RLOs from the Repository can be aggregated and sequenced to produce a course, and facilitate delivery to students. In this paper, we discuss these two additional issues and solutions to these problems. The solutions are implemented as an extension of the initial prototype application. The final outputs of the extended application are eBooks, which can be delivered to students both as offline and online content. Following Section 1 (INTRODUCTION), the paper gives a brief background summary in Section 2 (BACKGROUND) of the work that was previously completed with respect to the definition of a RLO suitable for a P2P network, and the design of a P2P RLOR, including some details of improvements made to the prototype. In Section 3 (PEER-REVIEWS IN A P2P NETWORK), the challenge and approach adopted for doing Peer-Reviews and maintaining Ratings for RLOs in a P2P network is presented and then discussed. Next, in Section 4 (PRODUCING EBOOKS), the extension to the P2P prototype application is presented which facilitates for the aggregation and sequencing of RLOs from the P2P network into courses and then discuss how it can be delivered to students as online and offline eBooks. A discussion relating to the eBook production then follows. The conclusions are then given in Section 5 (CONCLUSIONS).

2. BACKGROUND In first considering P2P architectures for managing eLearning content, Halm et al. [2] looked at the challenge faced by content authors and instructors when using client-server solutions. These ranged from managing different versions across different systems, offerings from one time period and learning situation to another, and a lack of comfort-level when placing content on Servers outside the control of the content authors and instructors. It was felt that if content authors and instructors had a facility on their own machine for storing their content, it will be easier to maintain one authoritative version of the content, and by introducing application features similar to traditional P2P file sharing network, it will allow for the content to be Internet accessible as in client-server models but without added overheads of hardware and complex network infrastructure. Overall, there should be an improvement in opportunities for content reuse. Halm et al. [2] on this proposal produced Lionshare, a P2P application for sharing eLearning content. Their requirement for the use of the authentication mechanism of Shibboleth [3] restricted adoption to only a few selected Tertiary level institutions that had or can access the required infrastructure. Hatala et al. [4] later developed an open communication layer for improving on the P2P content repository security and scalability to have a wider adoption. This however is a complex design by their own admission. Richards and Hatala [5] also worked on a P2P learning object system POOL, POND and SPLASH adopting use of the communication layer mentioned above. These successes at creating a P2P system for sharing eLearning content was the motivation for Singh and Bernard [1] to look at designing a similar P2P system but this time to consider any Internet-based user, not just technologically mature Tertiary level intuitions. The P2P repository design of Singh and Bernard [1], built on a definition of Reusable Learning Objects (RLOs) that seems to be a better fit for P2P sharing than just the traditional eLearning content, standard-based or otherwise. The RLO to be used on P2P networks according to this definition must have a required single stated Learning Objective, reference to the content that relate to the Learning Objective, and, reference to the formative and summative assessment content that relate to the learning content. In a later prototype enhancement of the repository, the assessment content must also be formatted using the Question and Test Interoperability (QTI) standard [6]. This simple RLO would make it easier to store, manage and share content and handle one of the major challenge of acquiring assessment content when the RLO is used in a course, something which none of the popular client-server systems can do without maintaining databases with questions and answers and providing search and discovery features to match content with possible assessment items. Singh and Bernard [1] also introduced the use of email certificates for signing the RLO so that changes can be

properly managed. The email certificates are also used to identify users, which acts as a method to discourage copyright violations common to the traditional P2P file sharing networks and applications. This email certificate identity also allows for users to optionally form groups and restrict sharing to amongst their group members. The P2P RLOR prototype is based on the open source file sharing Shareaza project [7] and restricted it to only using the more mature P2P communication protocol Gnutella2 [8]. The following work on peer-reviews for RLOs and eBook generation extends this P2P RLOR prototype to produce a more powerful and useful application for elevating a P2P eLearning system to a closer or better level than client-server eLearning systems. 3. PEER-REVIEWS IN A P2P NETWORK Rating files in the many traditional P2P file sharing applications [9] are done by users. Generally the users can assign a value between 1 (Poor) and 5 (Excellent), and they may also have the facility to enter a comment. This data is stored on the client where it is easy to manipulate and compromise the integrity of previously stored review data entered by other users, in effect making this approach an undesirable one for rating RLOs in the P2P RLOR. This seems to suggest that any peerreview data for RLOs need to be either digitally signed or kept on managed Database (D/B) servers. The latter appears to be especially necessary, if the review data is to be kept fully updated at all times, to all copies of a given RLO residing on different peers on the P2P network. Further, it will be more useful to have subject matter experts perform reviews on the RLOs and not just any random user on the P2P network, again suggesting some kind of account management for reviewer registration using managed servers. In further reviewing the approach used by the wellknown server-based repository, Merlot [10], and considering the above challenges along with some reviews of research done with respect to rating Learning Objects for quality and effectiveness [11, 12, 13, 14, 15, 16], it was decided that an adapted LORI (Learning Object Review Instrument) approach would be a good fit for reviewing RLOs in the P2P RLOR. The design would incorporate the use of a scalable D/B server architecture for the peer-review data, with account management features for managing reviewers, along with a registration system for users to identify RLOs that are to receive a review. The use of the servers will not include storing eLearning content on the D/B or other servers. Instead, the eLearning content will remain entirely on the P2P network where the P2P application will be extended so that a registered RLO that is assigned to a reviewer will be automatically downloaded to the reviewer’s P2P RLOR. The extension will facilitate the reviewer to enter all review related data using the P2P application, where it is then automatically and transparently uploaded via the reviewer’s account to the D/B on which the RLO is

registered. The process allows for any network peer with a copy of a given registered RLO to access and retrieve the peer-review data transparently from the server on which it is registered, thereby addressing the previously mentioned issue of having the most current review data across the entire P2P network. In adapting LORI 1.5 it was determined that only four of the nine dimensions needed to be captured explicitly, as some of the items were either not applicable because it was already inherent in the RLO design that the dimension is covered sufficiently, or, not applicable at the RLO level, again because of the design, but is to be given consideration when aggregating and sequencing for delivery to the learner. The following gives the adaption to LORI 1.5 which is used in the design, where the nine dimensions summary descriptions are taken from [17]: 1. Content Quality (veracity, accuracy, balanced presentation of ideas, and appropriate level of detail) – the reviewer is presented with a form to fill-in, built from the rubric given in Figure 1. 2. Learning Goal Alignment (alignment among learning goals, activities, assessments, and learner characteristics) – the RLO has a required learning objective by design, so the reviewer only needs to indicate if the stated objective is aligned with the reviewed content using a given scale. 3. Feedback and Adaptation (adaptive content or feedback driven by differential learner input or learner modeling) – with the goal to have RLO with only a single learning objective (as fine a granularity as possible), we propose transferring this dimension to a higher level where RLOs are aggregated and sequenced into a course. It is at the course level that we should be able to give more meaningful feedback and adaptation for learning styles, possibly leveraging information from a learner’s profile which can be maintained for structured course offerings over a given period of time. 4. Motivation (ability to motivate, and stimulate the interest or curiosity of, an identified population of learners) – as part of the RLO registration process, the user is required to fill out a form which captures the attributes Category, subCategory, Educational Level and pre-Requisites. The reviewer is asked if the RLO content matches against the user selections. If a “No” is given, the reviewer should enter a comment. These same attributes are also used to match against the reviewer’s profile to select possible subject matter experts to be assigned for the review. 5. Presentation Design (design of visual and auditory information for enhanced learning and efficient mental processing) – this is part of the reviewer’s form to fill-in, as given in the rubric of Figure 1.

6.

Interaction Usability (ease of navigation, predictability of the user interface (UI), and the quality of UI help features) – this is part of the reviewer’s form to fill-in, as given in the rubric of Figure 1. 7. Accessibility (support for learners with disabilities) – the belief is that this may be better determined at the course level and not at the RLO level. It is therefore omitted from the RLO review. 8. Reusability (ability to port between different courses or learning contexts without modification) – the RLO is designed to have the single learning objective, content and assessment contents so no additional information has to be explicitly gathered from the reviewer. 9. Standards Compliance (adherence to international standards and specification) – the main content may or may not be standardsbased. The reviewer indicates if it is in any standard-based format such as LMS or SCORM. It is only informational and does not impact on the review score. In the updated design of the RLO it is now necessary for the all assessments to be in QTI format. The reviewer checks this and indicates with a Yes/No. A No immediately causes the assessment score to be set to zero. The following summarized algorithm is used to arrive at a rating score for the dimensions of the LORI 1.5 implemented: Begin P2P peer-review algorithm { • The user request registration of a RLO on a given server, is assigned a GUID (Globally Unique Identifier) and digitally signs the RLO with the GUID inserted, the RLO state is set to “Request Review”, the user is presented with a data-form to capture the following: 1. Select Category from a predefined list; if item does not exist, allowed to add; 2. Select sub-Category from a predefined list; if item does not exist, allowed to add; 3. Select Educational Level (pre-school / primary / secondary / university); if item does not exist, allowed to add; 4. State any pre-requisites (restricted to fixed number of characters); 5. Comment (restricted to fixed number of characters); 6. Select the number of reviewers required for reviewing the RLO (minimum of 2); • Match registered subject matter experts (reviewers) with 1 to 3 of the data-form; (alert if no matches found); • Randomly select reviewers up to the amount assigned in 6 of the data-form; (weighted against selected attributes of reviewer’s history and current load when assigning); (alert if not enough reviewers);



• •

• •

If no alerts, set RLO state to “Reviewers Assigned” and create a temporary group with all selected reviewers and the user, to facilitate reviewers to access the RLO; else, inform user of alert(s); Set RLO state to “Review in Progress”; For each reviewer selected { • The reviewer’s P2P RLOR is automatically updated to include a download of the RLO from the user RLOR requesting the review. Each reviewer can confirm the download and when the RLO is fully retrieved they can proceed with the review; • Review RLO and assign a rating score for A. Learning Objective, B. Content, C. Assessment (Formative) and D. Assessment (Summative), using the P2P RLO rubric of Figure 1; • Average Score = (A+B+C+D)/4; • If A < 2 or B < 2 or data-form 1 to 4 does not match the content, then set Average Score to 0; • Calculate Average Rating Score for all reviews completed thus far and assign to RLO; • Indicate the number of reviews completed against the number remaining; • Flag RLO for removal from reviewer’s P2P RLOR; } Set RLO state to “Review Completed”; Delete the previously created temporary group with reviewers and user;

} End P2P peer-review algorithm.

All rating and review data is stored on the D/B where the RLO is registered and is retrieved transparently using the GUID by the P2P RLOR where any copy of the registered RLO may subsequently exist. The user who initially registered and digitally signed the RLO can also resubmit an improved version for a new review. Any copy of the previous version will receive notification that it has been updated when the rating and review data is accessed from a P2P RLOR. The P2P RLOR will be notified that a new version exists on the P2P network giving the new registered GUID of the RLO allowing for an automatic process for downloading the new version of the RLO if desired. 3.1 Peer-Review Discussion The peer-review approach for the RLOs in P2P networks cannot be the simple process similar to rating files in a traditional P2P file sharing application. It needs to be similar to strategies adopted in client-server systems, and in this case, the adaption of LORI seems to be a reasonable fit. The strategy of the user initiating a registration of RLOs for reviews, allows for the user to consider only when the RLO is matured. It may be possible to employ a peer-review process amongst users on the P2P network similar to [18] for maturing the RLOs prior to submission for peer-review by subject matter experts. The reviewer’s task is kept to a minimum as much as possible by having users supply some information similar to what is required for filling-in metadata tags for standards-based content such as [19], and then having the reviewers validate with a simple option selection, to facilitate some aspects of the rating. This reduces the work considerably and more focus can be placed on

Scale: 0 (None), 1 (Poor, needs work), 2 (Acceptable, but can be improved), 3 (Average), 4 (Above average), 5 (Excellent). Learning Objective: A. How well stated is the learning objective? (1 – 5) [Comments] B. Is it a single learning objective? (Yes/No) If B = Yes, Learning Objective Score = A else if A > 0 Learning Objective Score = 1 Content Quality: A. How accurate is the content material? (1 – 5) [Comments] B. How detail is the content material? (1 – 5) [Comments] C. How well related is the content material to the stated Learning Objective? (1 – 5) [Comments] D. How well is the presentation of the content material? (1 – 5) [Comments] E. How well can the user navigate the content material? (1 – 5) [Comments] F. (Informational only) Is the content in a standards-based format? (Yes/No) [List format(s)] Content Quality Average Score = (A + B + C + D + E) / 5 If the Assessments are not in QTI format, Assessment Content Quality Average Score = 0, otherwise proceed with rating the Assessments. Assessment Content Quality (separate reviews are done for each of Formative and Summative Assessment Content): A. How well do the questions match the content? (1 – 5) [Comments] B. How accurate are the answers to the questions? (1 – 5) [Comments] C. How is the variety and number of questions for the content? (1 – 5) [Comments] Assessment Content Quality Average Score = (A + B + C) / 3

Figure 1 – P2P RLO Review Rubric

reviewing the content and assessment content itself against the learning objective for the quality determination. While the algorithm and overall peer-review process explained seems reasonably sound it can be considered for further validation using quantitative studies. At this time however, the RLOs in the P2P network are too minimal to consider such studies, but one which would be possible later on, as more RLOs are developed and added on the network for sharing and reviews. 4. PRODUCING EBOOKS The goal of producing RLOs and sharing with other users is to have them adopted for delivery to learners in the form of a computer-based course. In initially designing the P2P RLOR, it was being considered that the RLOs from the Repository would be used in the traditional client-server eLearning systems, where appropriate plugins can be built to allow the RLOs to be used in such systems. A Moodle plugin similar to DOOR [20] was being considered. It was later noted that since the P2P

The general approach used to generate eBooks employs the use of an eBook compiler. It was discovered that the feature lists for eBook compliers vary widely and the review at [21] proved useful. The final choice however, was the compiler at [22] simply because of the maturity for programming features, licensing and eBook activation options. For the P2P RLOR prototype the commercial version of the compiler was acquired in order to have the full feature set. The Digital Rights Management (DRM) and activation management were two important features necessary to determine if the eBook would be used to deliver secure summative assessments. The combination of having JavaScript and imbedded programming facility also influenced the choice. The support for many audio and video formats apart from including HTML and PDF formats of content were also key considerations. Further, online webpages can be viewed within the eBook without the use of an external browser. The first step in generating an eBook using the P2P RLOR, is to select the RLOs for the course from peers on the P2P network, and create new ones as needed for the specific course which may not exist already on the

Figure 2 – Screenshot of an eBook created using RLOs from the P2P RLOR prototype RLOR was facilitating any Internet-based user to be a producer then it should allow similarly for any group of Internet-based users to be consumers. Further, having Internet-based course-creators would mean that only such users with access to a client-server eLearning system (and plugin for the P2P RLOR) would be able to create and present a course. This limitation was considered too restrictive for the average Internet-based user. As a result, the idea of using the RLO to produce eBooks was explored and determined to be a better fit into the P2P RLOR philosophy.

network. The RLOs are then sequenced into a course structure via a drag-and-drop Table of Content (TOC) Interface built into the extended P2P RLOR. The interface allows for including documents which are needed for the course but not RLOs, and web links to Internet sites which can be viewed from within the eBook. An example of a document which may be used that is not a RLO is one with the details on the course itself, and a web link example could be an Internet site linking to the Teacher’s information or supplementary course content and references. Once the TOC is built, the P2P RLOR generates an XML document that is passed to the

Figure 3 – Screenshot of eBook with Formative Assessment Tab selected compiler which has all the information required to access all RLOs, Scripts, documents and external web links. Prebuilt templates can also be selected and passed to the compiler which determines the look and behavior of the eBook, for example, a template can be passed with an expiry date set for the eBook. This allows users to register and activate the eBook before the content is accessible, present the modules in a given order (only when prerequisites modules are completed then the user is allowed access to the next module), and release the summative assessments on a specific date and time or when the user applies for an activation key. An important design in the TOC is the facility to aggregate and sequence the assessment content from several RLOs to a location such as at the end of a module, and to repeat some assessments at different review points. The XML TOC file, templates and other files and parameters are passed to the eBook compiler using a command line script generated by the P2P RLOR. The user does not have to interact directly with the compiler. Figure 2 shows a screenshot of an eBook for a first year undergraduate Database Management Course. The TOC on the left allows the user to navigate to different topics, and on the right, the documents, web site and RLO content is displayed. In the screenshot an RLO is selected and the content tab shows the main content of the RLO. In Figure 3, another screenshot is shown with the same RLO selected, but this time the assessment Tab and Formative Assessment Tab (Practice Test) is selected. The user can proceed to answer the questions and review the answers for reinforcing the learning content. An open source QTI player written in JavaScript by [23] is embedded into the eBook to facilitate the Assessment interactions.

Figure 4 shows the screenshot of the Rating tab. If the eBook is used while online it displays the real-time rating and review data details for the RLO retrieved from the server on which the RLO was registered for review and where the data is stored. The Signature tab will show the digital signature for the RLO and indicate if it is signed using a valid email certificate. 4.1 eBook Discussion At this stage producing standard-based eBooks was not considered simply because there was not a good fit which exists for managing the summative assessment. The DRM, activation of the eBook and activation of specific pages were important features for facilitating the summative assessment, with ongoing plans to now have the scores from the assessment securely uploaded to a server where the user profile from activation of the eBooks is maintained. Also, similar to how the rating is retrieved and displayed for a RLO the scores can remain secure and be displayed for the assessments. The production of an eBook as an EXE (executable) file limits the platform to Microsoft Windows clients; they cannot be used on MAC and Linux clients or mobile devices. This limitation was due to the security features already identified, however, an eBook player using Microsoft Silverlight [24] and HTML5 [25] is being prototyped by an undergraduate computer science student which will allow for reviewing two possible technologies that can be used to widen the platform for eBook consumption. Achieving the goal of delivering an eBook to any Internet-based learner, and having being produced from peer-reviewed RLOs shared on a P2P network comprising

Figure 4 – Screenshot of Rating tab showing rating and review data users from any location on the Internet, is significant. It may now be possible to exceed the reach of Teaching and Learning beyond what is common to client-server eLearning systems! 5. CONCLUSION Peer-review of RLOs in a P2P network is more challenging than a client-server environment because the integrity of the feedback from the reviews can be easily compromised if it resides on a peer. The decision to maintain this data integrity by using a D/B Server is a necessary one. In the prototype implementation, it is shown that the user can from the peer initiate a registration of an RLO on the server and reviewers can from their own peer complete the review without accessing the server directly. Apart from the registration process, all other server involvement to have the feedback stored and accessed on the D/B is transparent to all peers involved in the review process and subsequently the peers that will later access the rating information when users are viewing the feedback. The similar techniques of the eBooks accessing the rating for registered RLOs are also transparent. The server component is built using technologies that allows for a scalable implementation, where any number of servers with different owners can participate in the process. This modularity and scalability is important, since it lends supports for different groups of P2P users to have their own servers and management process, as well as for more public groups and individuals to use publicly available servers for their management of the reviews.

Aggregating and sequencing peer-reviewed RLOs from the P2P RLOR into eBooks introduces a sense of completion when creating and sharing RLOs with peers on a P2P network. The inherent design of the RLOs for the P2P network to have the Learning Objective, content and assessment content, all related and contained as the structure of the RLO, along with the digital signature of the author, inherently introduce a level of reusability which reduces effort required to implement LORI 1.5 for the peer-review, and the effort of retrieving assessment content when generating eBooks. Overall, the introduction of the peer-review and eBook generation has matured the initial prototype P2P RLOR into a possible first generation useful product, for any group of Internet-based users wanting to produce and share eLearning content and deliver (as eBooks) to students leveraging P2P network architecture. It is now with eager anticipation as the P2P RLOR is adopted and used by both users from the teaching and learning domains, that further investigations involving quantitative methods can be used to determine possible improvement for eLearning implementations in a P2P network architecture. ACKNOWLEDGEMENTS We would like to express our gratitude to two very enthusiastic students: Naresh Seegobin, a Post Graduate student and avid Software Developer, and Samuel Thompson, a final year Undergraduate Computer Science student for their skillful programming contributions in developing the group management and content rating ASP.NET DNN (DotNetNuke) modules. Also, it is not a

common occurrence to discover a first year Undergraduate who can program at the level of graduates. Jarrerd Hosien was such a discovery and we thank him immensely for his contributions in attempting to develop the first alpha Silverlight and HTML5 eBook reader/player for the P2P RLOR prototype. We wish them all the best in their future endeavors. REFERENCES [1] R. G. Singh & M. Bernard, The design of a Peer-To-Peer Reusable Learning Object Repository, Proc. 6th International Conference on Education, Samos Island, Greece, 2010, 412-418. [2] M. J. Halm, R. Shuey, M. Hatala, G. Richards & M. Dooris, LionShare: Connecting and Extending Peer-to-Peer Networks, A Penn State Proposal to the Andrew W. Mellon Foundation, accessed April 30, 2010, http://www.cob.sjsu.edu/jerrell_l/ lionshare_ mellon_pdf.pdf. [3] Shibboleth, accessed March 20, 2012, http://shibboleth.internet2.edu/. [4] M. Hatala, G. Richards, T. Eap & A. Shah, Secure Communication Layer for Scalable Network of Learning Object Repositories, In S. Pierre, E-Learning Networked Environments and Architectures: A Knowledge Processing Perspective, New York: Springer, 2006, 276-305. [5] G. Richards & M. Hatala, POOL, POND and SPLASH, In R. McGreal, Online Education using Learning Objects, London: RoutledgeFalmer, 2004, 236-243. [6] IMS Question and Test Interoperability Integration Guide, Version 2.0 Final Specification, accessed April 30, 2010, http://www.imsglobal.org/question/qti_v2p0/imsqti_intgv2p0.ht ml. [7] Shareaza, accessed April 30, 2010, http://shareaza.sourceforge.net. [8] Gneutella2, accessed April 30, 2010, http://g2.trillinux.org. [9] Top 10 Most Popular P2P File Sharing Programs in 2009, accessed May 10, 2010, http://www.p2pon.com/top-10-mostpopular-p2p-file-sharing-programs-in-2009-at-your-request. [10] MERLOT: Multimedia Educational Resources for Learning and Online Teaching, accessed February 18, 2012, http://merlot.org/. [11] J. Nesbit, B. Belfer & J. Vargo, A Convergent Participation Model for Evaluation of Learning Objects, Canadian Journal of Learning and Technology, 28 (3), 2002, accessed March 20, 2012, http://www.cjlt.ca/index.php/cjlt/article/view/110/103. [12] I. Box, Submission and Peer Review of Learning Objects Using a Community-Based Repository, Proc. of the 2004 Informing Science and IT Education Joint Conference, Rockhampton, Australia, 2004, 113-121. [13] Y. Akpinar & H. Simsek, Should K-12 Teachers Develop Learning Objects? Evidence from the Field with K-12 Students, International Journal of Instructional Technology and Distance Learning 4(3), 2007, 31-44, accessed March 1, 2012, http://www.itdl.org/Journal/Mar_07/Mar_07.pdf. [14] J. McKenzie, L. Pelliccione & N. Parker, Developing peer review of teaching in blended learning environments: Frameworks and Challenges, Proc. Ascilite, Melbourne, 2008, 622-627. [15] Y. Akpinar, Validation of a Learning Object Review Instrument: Relationship between Ratings of Learning Objects and Actual Outcomes, Interdisciplinary Journal of E-Learning and Learning Objects 4, Ed. Heinz Dreher, 2008, 291-302. [16] A. Auvinen, The challenge of quality in peer-produced eLearning content, eLearning Papers, No. 17, 2009, accessed

February 18, 2012, http://www.elearningeuropa.info/files/media/ media21212.pdf. [17] Learning Object Review Instrument accessed February 18, 2012, http://edutechwiki.unige.ch/en/Learning_Object_Review_ Instrument. [18] E. Gehringer, et al., Reusable Learning Objects Through Peer Review: The Expertiza approach, Innovate 3 (5), accessed March 3, 2012, http://www.innovateonline.info/pdf/vol3_issue5/ Reusable_Learning_Objects_Through_Peer_Review__The_Exp ertiza_Approach.pdf. [19] IMS Global Learning Consortium Specifications, accessed April 30, 2010, http://www.imsglobal.org/specifications.html. [20] DOOR - Digital Open Object Repository, accessed April 30, 2010, http://door.sourceforge.net/. [21] eBook Complier Review, accessed February 18, 2012, http://www.ebookfreeway.com/p-compiler-review.html. [22] HTML Executable eBook Compiler, accessed February 18, 2012, http://www.htmlexe.com/index.php. [23] qti–player, accesses February 18, 2012, http://code.google.com/p/qti-player/. [24] Sliverlight, accessed February 18, 2012, http://www.silverlight.net/. [25] Building Apps with HTML5: What You Need to Know, accessed February 18, 2012, http://msdn.microsoft.com/enus/magazine/hh335062.aspx.