2014 XVI Symposium on Virtual and Augmented Reality
Glassist: Using Augmented Reality on Google Glass as an Aid to Classroom Management Manoela Silva, Daniel Freitas, Edvar Neto, Caio Lins, Veronica Teichrieb Jo˜ao Marcelo Teixeira Voxar Labs Departamento de Estatstica e Informtica Informatics Center - UFPE UFRPE Recife, Brazil
[email protected] {mmos, dqf, excvn, csnrl, vt}@cin.ufpe.br
themselves, and subject their learning to peer review and critique [2]. Teachers have many errands as they enter in a classroom. They are responsible for students learning and as a consequence they need to set learning goals, to elaborate lesson plans and to manage and discipline the class. Teachers need to be innovative, to share and collaborate with colleagues. In the hands of a skilled teacher, technology tools can be used to cultivate lively debate and exchange of ideas, support the social construction of knowledge, and democratize the classroom. Technology is more than a tool; it is a mean to support deep and engaged learning. Augmented Reality (AR) is a new piece of technology that has a huge potential in education [3]. [4] point out that despite all fields of knowledge can benefit from AR, teaching, learning and training will be greatly transformed with new ways of interaction between students, teachers and information due to the mixture of the virtual and the real content. Aiming to support teachers errands in the classroom, our work discusses the development of an application for Google Glass that uses AR. The application, named Glassist, was designed to help teachers create students portfolio, manage information, add new content to students folders and share that content with their colleagues, coordination and parents. The paper is organized as follows: in section 2, we show related works regarding the use of Google Glass and AR in education; section 3 discusses some of Google Glass technical features and some basic concepts of AR. In the following section, we explain the Glassist application concept; section 5 describes the development process; and section 6 presents the preliminary results. Finally, we conclude the paper and point out some future works.
Abstract—Google glass is a hands-free, head-mounted intelligent device that can be worn as a wearable computing eyewear. This work discusses its potential in education by proposing an application, the Glassist, aiming at helping teachers management tasks. The application allows teachers to create individual portfolios for students, manage their information and share it with peers. The tool uses Augmented Reality to recognize students face and display relevant information about them. We also discuss some preliminary results regarding its evaluation. To conclude, we point out some future works. Keywords-augmented reality; google glass, education
I. I NTRODUCTION The insertion of technology is a topic widely discussed in the field of education. Researchers have shown that when used properly, technology has the potential to improve learning. Mastering how to use technology became as important as many different abilities, such as reading, writing and counting. In a globalized, interconnected world where information and knowledge are just a click away, it is impossible to think about schools disconnected from technology since it can and should be used as a strategy to improve the quality of classes as well as to call students attention and enhance their creativity. Theories of knowledge, learning, teaching, and technology are shifting. Knowledge is more than just memorization and recall; it is an active, situated, and engaged process of making meaning, interpretation, and developing deep understanding [1]. According to [1], it is important to remember that the quality of the teacher is essential. Teaching is more than information delivery; engaged teaching involves design and support of rich learning experiences. Teachers as well as students learn by engaging in meaningful design work; by studying, doing, experimenting, and reflecting; by collaborating and conversing with peers; by sharing what they see, do, and create with others. In order to understand how to effectively design learning that uses technology to increase student engagement, teachers need opportunities to engage and learn in similar ways 978-1-4799-4261-9/14 $31.00 © 2014 IEEE DOI 10.1109/SVR.2014.41
II. R ELATED W ORKS Google Glass has been heralded to be a revolutionary tool to class. Although it was not created with this purpose, there are many speculations on how to use this tool in class. Currently, there are lots of apps available and the 37
power of the technology in schools is limited to mostly video and image capturing purposes. Students can now turn into documentarians. Likewise, teachers can record their own lesson for training purposes [5]. Many websites, especially technology blogs and some researchers have pointed out some uses for Google Glass in the classroom as can be seen in the list as follows [6] [7] [8]: •
•
•
•
•
•
•
•
Google promoted a contest before launching Google Glass. Winners received the equipment to test it and give feedback. Some teachers have got the device and started to use it in class. Science teacher Vanden Heuvel is using Google Glass as an online teaching tool. The teacher records videos for his popular science education channel STEMbite [10]. He does interesting lessons and showcases the potential of Glass in education. Google has also given Google Glasses to college students in order to explore it in filmmaking. The participating schools are American Film Institute, California Institute of the Arts, Rhode Island School of Design, University of California, Los Angeles and University of Southern California. According to [11], the company says the schools will explore how to use Glass for documentary filmmaking, character development, location-based storytelling and other projects people have not considered yet. Another pair of glasses was given to Assistant Professor Meghan Corbin. She works at Mercyhurst University and intends to use it in High Education. Her experience will be shared at the weblog Glass in Class [12]. Technology coordinator at a pre-K-2nd grade school, Margaret A Powers, also won Googles competition “If I Had a Glass” and earned a pair to use in her school. She created a blog [13] in order to share her experience using the device in school. Professor Ralph P. Hall from Virginia Tech [14] also got his Glass and with his team is testing it in his University courses as well as developing applications for it. He claims some initial success in recording assignmentfeedback videos for students. Another educational use for Google Glass is to help doctors explain a surgery or treatment method to other doctors and students, and also receive data of patients from distant places [15]. Although it is early to have a good evaluation of the tool, we can notice that teachers are willing to be guinea pigs and test potential technology in their classrooms. Professor Margareth A. Powers reviewed five big takeaways for Google Glass in the classroom. According to [16], there is a lot of potential to growth. She believes as more apps are developed the possibilities for Glass will continue to increase. She also highlights that people can take almost instant, high quality pictures. In her opinion, it can be a great tool to use in field trips and any situation one needs to be hands-free. In addition to that, she explains you can grow home-school connections via Tumblr. Professor Koci Hernandez [17] is another professional in the field willing to try Google Glass in high education. He has already suggested one change that would benefit in teaching facial recognition. One tool that has a lot of potential in education and can be efficiently combined with Google Glass is AR. [4] believe
Learn new languages: One of Google Glass features is to present text based translation in real time. This can be particularly helpful in language learning courses as students would be able to decode and interact with the new language in real time. Help creating presentations and research materials: Google Glass allows users to shoot quick videos and images. These features may be helpful for both teachers and learners when they need to prepare presentations or document research. Quick on-the-go research: With Glass, students and teachers will have a web browser available at all times; this will enable them to search information, make notes, bookmarking important pages for later use and much more. Flip the classroom with TED-Ed: Teachers can record videos and display them on TED-Ed and then add some additional resources and guides to them. Build portfolios: It is important for students to build portfolios during their academic tenure as they prepare to seek and land jobs. The features of Google Glass will help learners to store and retrieve information. Distance learning: With Google Glass, courses are constantly accessible. People can watch videos, read discussion posts and see what is happening in class in real time. Students will have the opportunity of being in class without being physically at the same location. Personalized studying: Learners will have the opportunity to learn in the style they prefer. Whether it is through listening to an audio lecture, watching a video, reading a paper or combining different activities. Google Glass allows learners to learn and explore in a personalized way. New courses, skills and careers: Many authors believe that it is a pretty safe assumption that Google Glass will lead to new applications and technology innovations that can result in new career possibilities and new instructional training needs, as happened with mobile applications, such as the tablet app explosion.
According to [9], education is one of the nine industries that will be revolutionized by the introduction of Google Glass. They predict that students will be able to live-stream the lecture to classmates who are sick that day. On the other hand, students will have access to all the answers needed for a test. This implies that schools will need to determine the best uses for Glass in the classroom environment.
38
A. Google Glass
that the field of education and training will be transformed by new types of interaction promoted by the use of AR. Jeff Orr, senior practice director at ABI Research, states that Google Glass coupled with AR content can potentially “give students an experience they would not normally encounter in their classroom”[18]. [19], in a meta-review of the literature, identifies some positive and negative aspects of using AR in the classroom. Among the positive aspects, there are: increased content understanding, long-term memory retention, improved physical task performance, improved collaboration and increased student motivation. Teachers are always looking for new ways to effectively teach and assess students learning. Although we can see a tendency to develop applications for students needs, teachers have many errands when conducting a class, such as classroom management and students assessment, so, they might benefit of using technology as well. In order to meet those needs, there are innumerous applications, especially mobile, to help teachers better assess their learners and organize their lessons as can be seen in the following list: • •
•
•
•
Googles Glass works as a 50g Android-based wearable computer [25]. As shown in Figure 1, it is similar to an eyewear, but with the advantages of having embedded processing power and also input and output peripherals, all in the same package.
Figure 1.
Google Glass components (http://www.brille-kaufen.org).
Tests performed by the authors show that the Glass battery has a maximum lifetime of one hour, when used in full load situations (recording a 720p resolution video or running computer vision algorithms). Therefore, Google Glass is not meant to be an active device. It was originally conceived to give punctual information (notifications) to the user. Working that way, its battery can last a day or more. The touchpad (located on the external part of the CPU case, on the lateral side of the device) supports multitouch interaction. It is possible to identify single or double tap gestures, as well as swipes in a specific direction performed by the user with his/her fingers. All content on Glass is exhibited on its 640x360 display as a 64cm screen and 2.4m away. Since the prism is not placed right in front of users view, it does not interfere significantly with his/her vision. Due to the fact that the prism is located slightly above users eye of sight, the creation of see-through AR applications is compromised. In order to implement them, developers must first solve the alignment problem between virtual screen (projected by the prism) and real world content, which means that a small portion of the image captured by the Glass camera matches the virtual display area. This mapping may vary according to the distance between Glass and real object being viewed.
NearPod: This application allows teachers to manage content on their students’ devices [20]. A+ Spelling Test: This application is designed for elementary kids. It enables students to test their spelling knowledge and allows teachers and students to check the results [21]. GoClass: It enables teachers to create lessons and share them with students. It also allows them to formulate questions as well as draw diagrams and share materials with students. In addition, teachers can also keep track of students interactions with the lesson [22]. Teacher Clicker Socrative: This application enables teachers to create different types of quizzes and grade students [23]. My class talk: This application allows teachers to score points and rank students participation in the classroom. Teachers can even upload photographs to help identify the students [24].
B. Augmented Reality III. BACKGROUND
AR has been very promising by its ability to make user interaction more natural and improve the information visualization quality. So, due to its promising applicability, it has been extensively studied. An important step in AR applications, since the real world scene has already been tracked, is the virtual elements visualization by the users [26]. The most common way today is through monitors, where the person places a marker in front of the camera and observes through the computer screen the virtual object inserted. Another way to display the virtual information is to place these data directly into
The application designed by the authors is based on the use of AR as a tool to help teachers management tasks. As the Glassist aims to exhibit contextualized information about the students for the teacher at runtime, an essential part of the system is a display technology that is simple to use and light, as well as allows hands-free usage. In our work we used the Google Glass wearable platform together with AR, and major information about these technologies are given in sequence.
39
analyze a typical day of this professional. We executed an interview based on the Day in the Life method, in which the designer follows the subject through a typical day, observing and recording events to collect data [29]. Then, the interviewee answered questions and was encouraged to tell us about her day. The main goal of this process is to collect data about the teachers activities, their difficulties and challenges, and other daily aspects. After the data collection process was completed, we noted that teachers have to collect various information about each child all the time, as well as access this information frequently. We also noted that there is a high potential to wearable devices, such as Google Glass, since these devices allow hands-free operation. Additionally, using Google Glass will allow monitoring of the children without disturbing the class. In this way, a quick brainstorming process was performed, in which some alternatives were generated and then selected. In the next topic we detail the alternative chosen.
the environment, not only in the viewpoint of the observer, and is called spatial AR. This insertion can be done through simple screens attached to the environment or in a more complex way, through the use of semi-translucent screens to create the effect of holograms. The most common form of exhibition in spatial AR is held directly by projections on a surface, called the projective AR. Hands-on displays like smartphone and tablet displays are very popular as well, since these devices had their computational resources increased lately allowing AR applications execution. Other famous category of visualization devices is the Head-Mounted Display (HMD), that can be optical or video see-through. Figure 2 illustrates an example of a see-through screen. It can be said that the Google Glass is an HMD, but it doesn’t natively correctly align virtual and real worlds, as mentioned above. In this work we propose the Glassist AR application working integrated with the Google Glass, used as an optical see-through HMD for exhibiting the augmented scene.
B. Solution Description
Figure 2. screen.
Our product, named Glassist, consists of a teacher assistant on Google Glass that allows the creation of profiles for each child, entering and viewing information about them. By using Google Glass with AR features, the application aims to help day-to-day main activities of teachers, in a seamless, non intrusive way. Glass does not natively correctly align virtual and real worlds, a necessary step to turn it into a complete AR experience device. Figure 3 (left) illustrates how Glass works, exhibiting on its screen the image captured by the camera overlaid with the virtual objects. On the right, it can be observed how our Glassist application works, using the Google Glass as a see-through display for the AR application. One of the main features of the system is the possibility to recognize each student using a face recognition technique. Knowing which student is the one the camera is pointing to, the system can allow access and edition of his portfolio: a database where all the schools staff (teachers, psychologists and coordinators) can organize and track the students information. This information can be shown in real time, using AR, so it can be easily understood and manipulated by the teacher. Photos, videos, audio, text notes and status icons can be added to the portfolios using gesture and speech commands. The application works on modules based on the different activities children have in a day inside the school. Their necessities and behavior at the playground time are very different from the ones inside the classroom. Therefore, the modules to be selected at the initial screen of the system are Classroom, Meal time and Outside (for playground activities, excursions, etc). Classroom mode: in this module, the system shows to the teacher the students names, last humor status (in case the
Microsoft’s see-through 3D display using Samsung OLED
IV. T HE G LASSIST C ONCEPT In the following section we show the design process for the Glassist product conception as well as describe the features of the application. A. Product Design Our proposal was to generate ideas for children’s education focused on the teachers activities using wearable devices. In order to reach a solution to the problem at hand, we conducted a quick design process based on the traditional design method [27] [28]. The design process included: data collection, generation and selection of ideas, and alternative detailing. As part of a user-centered design, an interview with a 25 years old preschool teacher was performed in order to
40
Figure 3. User view while experimenting Google Glass. The small screen on the top-right represents content shown on Glass prism. The left image shows how Google Glass natively works. On the right, Google Glass is shown as a see-through device for Augmented Reality.
previous teacher noticed an abrupt change of humor), recent notes and warnings, as remedies schedules. Meal time mode: when the children are at meal time, the teacher must remember several details about student’s food restrictions. At this module, it will be possible to see each learner’s food allergies, eating schedules and other significant details regarding their alimentation health as shown in Figure 4.
Figure 5.
Figure 4.
User view of the outside mode screen.
the face recognition algorithm has recognized him/her. When the teacher taps the touchpad, the system exhibits the profile screen of the student that is in the center of the camera view. In this level, the teacher can swipe between the students profiles or tap to view more information about one of them and add new notes, photos, videos, or other information to it. The teacher can also add data to the portfolios by centering a student on the camera view and clicking on the shutter button, adding a photo or video directly to his/her portfolio.
User view of the meal time mode screen.
Outside mode: the teacher will see information related to each students health status, previous bad behaviors and other relevant data. The system can also help handling extra class activities by counting the students and, thus, helping the teacher to keep the group together. In addition, it detects face expressions and puts this data into pupils’ portfolio so, a report showing their general mood variation over the month can be generated as can be seen in Figure 5. The system is connected and accessible from different schools devices, such as computers, tablets, smartphones and Google Glasses, thus, facilitating access to the data.
V. A PPLICATION D EVELOPMENT Glassists first implemented prototype contains two of its main features, which are face recognition of the students and access to their respective portfolio. Having the applications concept described earlier we will focus more on the Glassists code-level architecture. Basically we have three classes, two Android activity classes with one of them being the main one, and a class that represents an object that is instantiated as an attribute on another class. A few modules were used to motivate the idea of the possibilities that the use of Google Glass can bring in combination with AR. Each module will be described as follows. FaceDetectionActivity: The FaceDetectionActivity is the core class of the application, being responsible for managing
C. Application Flow The system starts with a fullscreen visualization of the camera image. When a student is detected, his/her name and status icons appear right above him/her, indicating that
41
the camera activity, image processing and face recognition methods. We used the open source computer vision and machine learning software library OpenCV for the image processing techniques [30]. Although the library has many image handling examples it lacked a face recognition algorithm. To overcome this obstacle we used its face detection algorithm, which uses an adapted idea of Haar wavelets [31] and a simple yet sufficient comparison algorithm. The OpenCV library gives us the position and boundary of the faces on screen coordinates, next we extract the face image from the current frame onto an OpenCV object (Mat), and then fondly compare it to a set of face images stored in the applications database. The implementation of the comparison algorithm was done by comparing byte by byte and checking if the sum of the squared differences of each byte pair is small enough to be a valid candidate. JView: JView is a class that has the sole purpose of being an attribute of FaceDetectionActivity. It extends JavaCameraView, an object from the OpenCV library which holds a listener for controlling the camera states. ProfileActivity: This module holds the methods for visualizing each students portfolio. On the contrary to the previous modules it doesn’t use external libraries, only native Java and Android components such as fragments. The integration of these modules gives us Glassist’s first prototype.
Figure 7.
It is noteworthy that the face recognition is incomplete in terms of precision. This means that sometimes we end up with more than one template for a single face. We plan to approach this problem with improvements on the comparison done on the face recognition algorithm. In general, the obtained results show that Glassist is a viable tool for using computer vision techniques to recognize multiple faces on the Android platform aiming the Google Glass device. VII. C ONCLUSION Although technology can have a great impact in education, sometimes, this purpose is not achieved because there is no efficient integration of new tools into the educational system. Regarding the use of Google Glass, it is important to note that its introduction can be scary for many teachers since they may not be comfortable in a classroom with students wearing glasses that will tell and show them anything they want to know. It can also be frightening for school administrators since this piece of technology allows to live broadcast anything that is going on in a school building. Finally, everything will be open to the public and there must be a change in the educational process. Hence, there is a need to think about teaching in a learner-centered way. We believe our application, the Glassist, may benefit education in a variety of ways; some of them are described as follows. Differentiated instruction: teachers understand that all children are different and that every child brings his/her own set of strengths and weaknesses. Therefore, instruction needs to be adapted to individual needs. By using Glassist as a way to collect information about students, teachers can use that data input in order to diversify their instruction and reach all learners in class. Identify new competencies: by cataloguing information about students development and difficulties, teachers can identify learner’s competencies and concentrate their effort on helping them overcome their problems. Teachers can also use this information when forming groups and, thus, help the exchange of knowledge in class. Google Glass features enable the teacher to acknowledge and respect each student
VI. P RELIMINARY R ESULTS With the previously described architecture, Glassist was able to recognize multiple faces and display information above them, in this case the person’s name. Figure 6 shows the result of this functionality from the point of view of the teacher wearing the Glass and using the AR application.
Figure 6. Glassist.
Example of a student’s portfolio at runtime.
Example of face recognition and information overlay by the
Next the teacher can access the students portfolio by tapping the Google Glasss touchpad once. The portfolio shows significant information about the student like for example if he/she has a specific allergy. Figure 7 shows an example of a portfolio at runtime.
42
as an individual with unique interests, talents, and learning strategies. Exchange of ideas: the use of Glassist may promote the exchange of ideas between teachers, coordinators and parents. This should facilitate the decision-making process regarding student’s needs as well as to provide a better picture of learner’s development. As future works, two main paths are being traced. The first one corresponds to the optimization of the already implemented features (face detection and recognition) in order to fasten the processing of image frames and also diminish the false positive rate of face recognition. Our second objective is to work with the recognition of face expressions, so that this information can be added up to the student’s portfolio automatically. According to [32], the use of Local Binary Patterns (LBP) has proven to be an effective method to perform such task.
[12] G. in Class. (2014, fev) Glass in class. http://glassinclass. blogspot.com.br.
VIII. ACKNOWLEDGMENTS
[18] S. Gitlten. (2014, fev) Will google glass usher augmented reality into the classroom? http://educationismylife.com/ googles-glass-the-implications-for-education/.
[13] M. A. Powers. (2014, fev) 365 days of glass. http:// 365daysofglass.com. [14] R. P. Hall. (2014, fev) Ralphphall. http://ralphphall.wordpress. com. [15] T. Hindu. (2014, fev) Google glass, a useful education tool for teachers. http://www.thehindu. com/todays-paper/tp-national/tp-andhrapradesh/ google-glass-a-useful-education-tool-for-doctors/ article5488640.ece. [16] M. A. Powers. (2014, fev) Reviewing google glass for the classroom - 5 big takeaways. http://bit.ly/1nWebiJ. [17] S. Grossman. (2014, fev) Professors envision using google glass in the classroom. http://bit.ly/1cB310P.
The authors would like to thank Brazilian National Research Council - CNPq (Processes numbers: 134114/2013-2, 381714/2014-4, 381424/2014-6, 13015479, 381712/2014-1) for partially funding this research.
[19] I. Radu, “Why should my students use ar? a comparative review of the educational impacts of augmented-reality,” in ISMAR, 2012, pp. 313–314. [Online]. Available: http: //dx.doi.org/10.1109/ISMAR.2012.6402590
R EFERENCES [1] L. M. Allan, J. Ribeiro, M. Bettine, M. Mandaji, R. S. Salom, and R. Laurato, “Crescer em rede,” Salvador, Brasil, Tech. Rep., 2013.
[20] N. P. Inc. (2014, fev) Nearpod. http://www.nearpod.com. [21] A. apps. (2014, fev) A+ spelling test. https://itunes.apple.com/ us/app/id501211008\?mt=8.
[2] M. Jacobsen. (2014, fev) Teaching in a participatory digital world. http://bit.ly/1bW59CD.
[22] G. Class. (2014, fev) goclass. https://www.goclass.com/ guestapp/index.aspx.
[3] T. N. M. Consortium, “Horizon report,” California, USA, Tech. Rep., 2011.
[23] socrative. (2014, fev) teacherclicker. https://itunes.apple.com/ us/app/id477620120\?mt=8.
[4] R. Tori, C. Kirner, and R. Siscoutto, Fundamentos e tecnologia de realidade virtual e aumentada. SBC, 2006.
[24] M. C. Talk. (2014, fev) My class talk. http://www.myclasstalk. com.
[5] C. Deamicis. (2014, fev) How one teacher is using google glass to change her classroom. http://pando.com/2013/12/13/ [25] E. Ackerman, “Google gets in your face [2013 tech to how-one-teacher-is-using-google-glass-to-change-her-classroom/. watch],” Spectrum, IEEE, vol. 50, no. 1, pp. 26–29, 2013. [6] J. Dunn. (2014, fev) The teachers guide to google glass. http: //www.edudemic.com/the-teachers-guide-to-google-glass/.
[26] E. Kruijff, J. E. Swan II, and S. Feiner, “Perceptual issues in augmented reality revisited.” in ISMAR, vol. 9, 2010, pp. 3–12.
[7] M. D. Vivo. (2014, fev) Google glasses: The future of educational tech. http://www.avatargeneration.com/2013/04/ google-glasses-the-future-of-educational-tech/.
[27] P. Jordan, Designing Pleasurable Products: An Introduction to the New Human Factors, ser. Contemporary Trends Institute series. Taylor & Francis, 2002. [Online]. Available: http://books.google.com.br/books?id=0s3el8sDjHsC
[8] M. Burke. (2014, fev) 5 ways google glass can be used in education. http://bit.ly/1egloVl.
[28] B. L¨obach, Design industrial: bases para a configurac¸a˜ o dos produtos industriais. Edgard Bl¨ucher, 2001. [Online]. Available: http://books.google.com.br/books?id= hm0hAAAACAAJ
[9] M. R. Dickey. (2014, fev) How google glass will revolutionize 9 industries. http://read.bi/1h9mEOB. [10] A. V. Heuvel. (2014, fev) Stembite. http://agl-initiatives.org/ stembite/.
[29] C. Kraft, User Experience Innovation: User Centered Design that Works, ser. Apressus Series. Apress, 2012. [Online]. Available: http://books.google.com.br/books?id= IKcUiLUHqsUC
[11] HuffPostCollege. (2014, fev) Google glass given to college students to explore filmmaking. http://huff.to/1dfhMTX.
43
[30] G. Bradski, “The opencv library,” Dr. Dobb’s Journal of Software Tools, 2000. [31] P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, vol. 1, 2001, pp. I– 511–I–518 vol.1. [32] S. Jain, S. Bagga, R. Hablani, N. Chaudhari, and S. Tanwani, “Facial expression recognition using local binary patterns with different distance measures,” in Intelligent Computing, Networking, and Informatics. Springer, 2014, pp. 853–862.
44
