A System for Memorizing Chinese Characters using a Song based on Strokes and Structures of the Character Yuma Ito
Tsutomu Terada
Masahiko Tsukamoto
Grad. Sch. of Engineering, Kobe University 1-1 Rokkodaicho Nadaku, Kobe, 657-8501, JAPAN
Grad. Sch. of Engineering, Kobe University, PRESTO JST 1-1 Rokkodaicho Nadaku, Kobe, 657-8501, JAPAN
Grad. Sch. of Engineering, Kobe University 1-1 Rokkodaicho Nadaku, Kobe, 657-8501, JAPAN
[email protected]
[email protected]
ABSTRACT Hundreds of thousands of Chinese characters are used in the Japanese and Chinese writing systems. Memorizing all the Chinese characters requires great effort because learners have to memorize not only the shape of the character but also its meaning and usage. Research to find a method to present the shape of the figures using sound has been conducted. Consequently, it is possible to generate a song for learning Chinese characters comprehensively using sounds that represent the shapes of the characters and lyrics that represent the meaning and usage. We propose a method to generate a song based on the structures, strokes, and usage of Chinese characters and a method to memorize Chinese characters using the generated song.
Categories and Subject Descriptors K.3.1 [Computers and Education]: Computer Users in Education—Computer-assisted instruction (CAI)
Keywords Memorizing Chinese characters, Generate song, Learning support
1. INTRODUCTION It is important to learn the characters when learning languages. In particular, the Japanese and Chinese writing systems have the most characters of all. The number of characters is far more than a hundred thousand. Even just considering the Chinese characters defined by Unicode [1], which is one of the character codes, the number of characters is approximately 70,000. Learning all the Chinese characters is difficult because it requires memorizing not only the shape of the characters but also the stroke orders and meanings. Although various systems and researches have been developed for learning the shapes and strokes of a single Chinese Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected].
iiWAS ’15, December 10-12, 2015, Brussels, Belgium c 2015 ACM. ISBN 978-1-4503-3491-4/15/12. . . $15.00 ⃝ DOI: http://dx.doi.org/10.1145/2837185.2837235
[email protected]
character [2, 3], there is no method for learning the meanings, usage examples, and idioms of Chinese characters at the same time. On the other hand, Oxford et al. [4] argues that the efficient way to learn the vocabulary of a foreign language is correlating the words with a sense such as visual sense, auditory sense, and tactile sense. Moreover, it is important to read aloud the meaning and an example sentence that the word is actually used in. Abboud et al. [5] researched a system that presents the shape and color of the basic figure and simple characters to a person using sound. In this way, it is possible to construct a system for learning Chinese characters using sounds and other auditory stimuli. Therefore, we propose a system for memorizing Chinese characters using a song based on strokes and structures of the character. According to Yiu et al. [6], there are 36 kinds of strokes. We allocate each stroke to a melody and describe the shape of the Chinese character by the melodies. In addition, a Chinese character has a fixed structure, such as a left component and right component. We define the component part of the Chinese character as component and the combination of components as structure. Figure 1 shows examples of components and structures of Chinese characters. There is a Chinese character in each square. The red area and blue area show the components of the Chinese character in each square. Moreover, the name of the component is written under the character. Because of these structures and components, we can recognize the rough shape of Chinese characters. We allocate the structure and components to a chord that represents the mood of the song. Accordingly, the sequence of the sound generated in accordance with the strokes becomes a melody that follows musical theory. This enables learners to perceive and memorize the melody easily. Finally, we allocate the meaning and example sentence of the Chinese character to the lyrics of the melody. Learners can learn the shape of a Chinese character by melody and chord progression and the meaning and the usage by the lyrics.
2.
RELATED WORK
There are several researches relating to supporting learning Chinese characters [2, 3]. These researches propose a method for learning the shape and strokes of Chinese characters for elementary school students and non-native Chinese speakers who have never learned Chinese characters. Teo et al. [2] proposed a system to learn the shape and strokes of
Proposed system Input Chinese characters
Left component Right component
Surround from lower left component
Database of Chinese character information Composed of * Stroke data * Component data
Surrounded component
Generate chord & melody
(a)
(b) Learner
Figure 1: Examples of component and structure Chinese characters by haptic interface. Additionally, Wen et al. [3] argued the system needs to learn the structure of Chinese characters. They focus on the strokes and structures of Chinese characters. Thereby, we regard the strokes and structures as an important element. However, Chinese characters are generally used in idioms, which are composed of two or more Chinese characters and have a special meaning. Learners cannot learn the idioms of Chinese characters by these methods, because they focus on learning one character at a time. Furthermore, because there are many homonyms, where the pronunciation is the same but the meaning is different, it is necessary to learn the difference between them. Nevertheless, these methods do not support the learning of homonyms. Accordingly, we constructed a system where learners can learn all of the strokes, stroke orders, meanings, and usage examples for Chinese characters including idioms. Abboud et al. [5] proposed a system that presents the shape and color of a figure and simple characters using sound. This system allocates a note of different pitch to each pixel in a vertical direction and scans the figure from the left. It represents the shape of the figure by sounding notes corresponding to the pixels on the scanning line. Abboud et al. state that the user can recognize the basic figure, such as a rectangle, triangle, and some lines. Although people can recognize a figure by sound as mentioned in this research, this method is not appropriate for Chinese characters due to their complexity of shape. Thus, we propose a new method to present the shape of Chinese characters.
3. SYSTEM DESIGN Methods of presenting characters such as Chinese characters include the following: • Presenting absolute position This method includes Abboud’s method [5] mentioned in section 2 and the method that allocates the sounds to each coordinate of an image of a character. However, it is difficult to memorize all the sounds that are allocated to each coordinate because of the complexity of shape. • Presenting stroke direction A Chinese character can be described as an assembly of strokes. Therefore, this method includes the method that presents stroke direction. • Presenting structure Chinese characters have certain structures as mentioned in section 1. Because these structures give the rough positions of strokes and meanings, it is appropriate to present the structures to learn Chinese characters.
Database of example sentences
Output song
Composed of example sentences Allocate lyrics
Figure 2: Processing flow of proposed system
In our research, we apply a method of presenting structure to our proposed system because we focus on learning not only the shapes of Chinese characters but also the meanings and usage examples.
3.1 Outline of proposed system Figure 2 illustrates the processing flow of the proposed system. The proposed system has two databases; one is composed of stroke data and component data of Chinese characters, and the other is composed of example sentences. As shown in Figure 2, the proposed system searches for the component data and stroke data for each Chinese character input by the learner. The system generates a chord and melody based on the component data and the stroke data. Moreover, after searching for a suitable sentence in the example sentence database, the system allocates it to the generated melody. Finally, the proposed system outputs the song. Next, we explain the usage example. Learners firstly recall the song that is used at learning time when he/she wants to recall the Chinese character when learning or taking an examination. At the learning time, such as when preparing for an examination, learners train to recall the melodies and Chinese characters by listening to the song. In contrast, at the time of actual practice, such as taking an examination, learners recall the character information without listening to the song. After recalling the melody, learners write down Chinese characters by recalling the shape of the characters. In our research, the proposed system outputs the song in accordance with the following three phases: converting components to chords, converting strokes to melodies, and allocating lyrics. We explain the phases in the following sections.
3.2
Converting components to chords
As mentioned above, the structure gives the rough shape and meaning. Therefore, we allocate the structures to the chord progression that describes the mood of the song. Unicode [1], which is a character code, defines 12 kinds of ideographic description sequences (IDS). By using IDS, it is possible to describe 13 kinds of structures including structures that cannot be divided any further, such as “一” and “ 木”. In accordance with IDS, because there are at most three
Table 1: Allocation between structures and chord progressions Structure
Chinese character
Chord progression
Single
一, 木
C
(a)
Left to right
好, 校
C, G
(b)
Above to below
字, 分
Am, G
(d)
Left to middle and right
班, 街
C, F, G
(c)
Above to middle and below
享, 学
Am, G, E
(e)
Fully surrounded
回, 国
Bm, A
(d)
Surrounded from above
問, 風
Cm, B♭
(d)
Surrounded from below
凶
Dm, C
(d)
Surrounded from left
匠, 医
D, A
(b)
Surrounded from upper left
右, 広
Em, D
(d)
Surrounded from upper right
句, 式
E, B
(b)
Surrounded from lower left
進, 起
F, C
(b)
Overlaid
刃, 斥
C, Cm
(f)
components that compose one Chinese character, the chord progressions should be composed of one to three chords. We propose 6 kinds of chord progressions that are often used in popular music and expand them into 13 kinds of chord progressions by changing the root note. Firstly, we present the 6 kinds of basic chord progressions in the following list. Note that the chords in the following list are indicated by Roman numerals, and the descriptions in the brackets are the examples when the root note is C. (a) I (C) (b) I, V (C, G) (c) I, IV, V (C, F, G) (d) Im, VII (Cm, B♭) (e) Im, VII, V (Cm, B♭, G) (f) I, Im (C, Cm) Table 1 shows the allocation between structures and chord progressions that are actually used in this research. The left column shows the name of the structure and its figure, the middle column shows the corresponding Chinese characters, and the right column shows the chord progression and the symbol of the chord progression mentioned above that underlies the chord progression in this table. Furthermore, each chord of the chord progressions corresponds to each component shown in white, gray, and black. Note that for the overlaid structure, we allocate the C chord to the component written at first on the stroke order and the C minor chord to that written at second.
3.3
Converting strokes to melodies
According to Yiu et al. [6], Chinese character strokes are classified into 36 kinds of strokes. Moreover, they can be subdivided into 10 substrokes. Figure 3 shows examples of components, strokes, and substrokes of a Chinese character “句”. From the top rows describe Chinese character, components, strokes, and substrokes. The corresponding component is colored red in the components row in the figure. Additionally, the names of the substrokes are described in the substrokes row. As shown in Figure 3, the Chinese character “句” is classified as having the structure of surrounded from upper right. The component of upper right has two strokes, and that of lower left has three strokes. In other words, this Chinese character has five strokes. In addition, these five strokes are subdivided into eight substrokes. In our research, we allocate the substrokes to a one-beat melody. The proposed system firstly divides all the strokes belonging to the Chinese character into substrokes. After that, the system converts the substrokes into melodies and combines the melodies into a song. The reason the length of the melody generated from the substrokes is one beat is that it is easy for a person to recognize a melody of this length. A melody following musical theory enables learners to recognize and memorize it easily. We use Orpheus [7], which is an automatic composition system, to generate melodies. Orpheus generates melodies from chord progression, rhythm, and pitch transition probability defined by several parameters such as lyrics and tune using dynamic programming [8]. In our research, we define rhythm and pitch transition probability from substrokes. Additionally, the proposed system generates melodies using dynamic programming in the same
Chinese character
Components
Strokes
Substrokes Falling leftward
Rightward stroke
Downward stroke
Change in stroke direction
Downward stroke
Rightward stroke
Downward stroke
Rightward stroke
Figure 3: Example of components, strokes, and substrokes
way as Orpheus. The melody generation method has three sections: definition of pitch transition probability, allocation between substrokes and transition probabilities, and connection of the melodies.
3.3.1 Definition of pitch transition probability Figure 4 shows the allocation between substrokes and melodies. The substrokes and their names are described on the left, and melody patterns are presented on the right. The horizontal lines in the figure represent the length of one beat. Furthermore, vertical lines are drawn at every quarter beat. The red arrows in Figure 4 represent the degree of pitch transition. The proposed system calculates the pitch transition probability based on this degree of pitch transition. We propose five kinds of degree of pitch transition: stable, falling, rapidly falling, rising, and free. Note that four of them are illustrated in Figure 4. We define the number of notes belonging to a melody as N and the pitch of the i-th note as the MIDI note number ni (1 ≤ i ≤ N ), where the range of pitch is from C3 to C5 (60 ≤ ni ≤ 84). At this time, we denote the pitch transition probability between ni and ni+1 by p(ni , ni+1 ). The pitch transition probability p(ni , ni+1 ) in each degree of pitch transition are defined by Equation (1)–(5). In the following equations, n and c(ni+1 ) are described by the following two equations. n = ni+1 − ni , { 1 ni+1 belongs to the chord, c(ni+1 ) = 0.2 ni+1 does not belong to the chord. Stable As shown in Figure 4, because there is no transition in pitch, p(ni , ni+1 ) is described by Equation (1). { 1 n = 0, p(ni , ni+1 ) = (1) 0 otherwise. Falling The pitch falls by at most five semitones as shown in Figure 4. Therefore p(ni , ni+1 ) is denoted by Equation (2). { c(ni+1 ) −5 ≤ n ≤ 0, p(ni , ni+1 ) = (2) 0 otherwise. Rapidly falling As shown in Figure 4, the pitch falls by more than the parameter of falling. p(ni , ni+1 ) is de-
Rightward stroke Downward stroke Falling rightward Falling leftward Right curve (Horizontal) Right curve (Vertical) Left curve
Tiny dash Change in stroke direction Flick up and rightward
Stable No transition in pitch
Falling
Rapidly falling
Rising
Falling at most Falling at least Rising at most 5 semitones 5 semitones 5 semitones
Figure 4: Allocation between substrokes and melodies
fined by Equation (3). c(ni+1 ) p(ni , ni+1 ) = |n|/10 0
n < −5, −5 ≤ n ≤ 0, otherwise.
(3)
Rising Because the pitch rises slightly as shown in Figure 4, p(ni , ni+1 ) is described by Equation (4) { c(ni+1 ) 0 ≤ n ≤ 5, (4) p(ni , ni+1 ) = 0 otherwise. Free This parameter allows a note to jump to a nearby pitch. Therefore, p(ni , ni+1 ) is denoted by Equation (5). { c(ni+1 ) |n| ≤ 5, p(ni , ni+1 ) = (5) c(ni+1 )/4 otherwise.
3.3.2 Allocation between substrokes and transition probabilities We explain the allocations between substrokes and melodies in Figure 4 and the following items in this section. Note that the titles of the following items correspond to the names of the substrokes presented in Figure 4. Rightward stroke This stroke is represented by one quarter note because its line does not move in a vertical direction. Downward stroke Because the line of this stroke falls vertically, it is denoted by two eighth notes with rapidly falling. Falling rightward This stroke is used, for example, for the second stroke of “人” and the third stroke of “大”. Therefore, we describe it as four sixteenth notes connected to falling. Falling leftward This stroke is represented by four sixteenth notes as well as falling rightward. However, the degrees of pitch transition of each note consist of stable, falling, and stable. Right curve (horizontal) This stroke is used only at the second stroke of “心” and the third stroke of “必”. Additionally, change in stroke direction always follows it. This stroke is described by two sixteenth notes and one eighth note connected to falling and stable. Right curve (vertical) This stroke is used, for example, for the fifth stroke of “式”. Change in stroke direction as well as right curve (horizontal) always follow it. We describe it by the sequence of notes that is composed of two sixteenth notes and one eighth note connected to rapidly falling and stable to express the stroke direction as more vertical than that of right curve (horizontal). Left curve This stroke is used, for example, for the sixth stroke of “家”. Moreover, change in stroke direction always follows it. This stroke is represented by one eighth note and two sixteenth notes connected to stable and falling. Tiny dash Tiny dash in both the right direction and left direction are denoted as tiny dash. For example, the first stroke to the fourth stroke of “学” correspond to tiny dash. The stroke is described by one eighth note and one eighth rest to express this short stroke. Change in stroke direction This stroke is not an independent stroke and always follows another stroke. For example, this stroke is used at a stroke such as the end of the fifth stroke of “学” and the end of the second stroke of “丁”. Although this is a short stroke like tiny dash, this stroke is connected to another stroke with rising. Flick up and rightward This stroke is used, for example, for the fifth stroke of “良” and the fifth stroke of “虫”. This stroke, like tiny dash and change in stroke direction, is often short. Therefore, a rest is put at the latter half of the beat and two sixteenth notes connected to rising are put at the first half of the beat.
Chinese character
Chinese character
Component Stroke
Stroke
Stable, Rising No rest
Free No rest
Component
Component
Stroke
Stroke
1 beat rest
2 or more beat rest
Figure 5: Connection method
3.3.3
Connection of melodies
The proposed system successively generates a note from the first substroke based on the method mentioned in the previous sections. In this section, we explain the connection method between the note at the end of a stroke and the note at the start of the following stroke. Figure 5 shows the connection method. The sequences of notes in Figure 5 represent the rhythm of a melody generated from substrokes. The rectangles labeled stroke, component, and Chinese character surround the melodies generated from substrokes that are composed of each stroke, component, and Chinese character. Additionally, the connection method is shown at the bottom of the figure. The proposed system connects the melodies by stable or rising (if the following substroke is change in stroke direction) if the two melodies belong to the same stroke and connects by free if the melodies belong to a different stroke. Moreover, the system does not put a rest in these connection cases. Next, when the two melodies belong to a different component, the system puts a 1 beat rest between the melodies. This enables the melody to have a block and learners to memorize Chinese characters by memorizing every component. Finally, the system puts a 2 or more beat rest between the melodies belonging to the different Chinese characters. This prevents learners from confusing this connection with the connection between the components. Note that the proposed system calculates the appropriate length of the rest so that the first note of the Chinese character always begins at the first beat of the measure. This enables learners to recognize the beginning of Chinese characters easily. Furthermore, the notes circled in red in Figure 5 represent the first notes of the components. The system allocates them to the root note of the chord that is allocated to the component.
3.4
Allocation of lyrics
Because the lyrics are the most important key for learners to recall the melody, it is necessary to have lyrics that can be recalled easily. Thus, we propose a method to use an example sentence where the Chinese character is actually used in the lyrics. Let Y be a text string of an example sentence including a Chinese character or an idiom, and X be a text string of the pronunciation of the Chinese character or one Chinese character belonging to the idiom. The system connects Y and X by the Japanese postpositional particle no, which corresponds to the preposition “of” in English, and let the connected text string be the lyrics. Note that if there are not enough notes to allocate the example sentence, only the pronunciation will be represented in the lyrics. In addition, we propose a method to allocate the lyrics to the melody equally. Figure 6 shows an example of equal
/fu-ka n/ overlook
Chinese charcters: Example sentences: for
/ta ka da i ni no bo t te shi ga i wo fu ka n su ru/ overlooking the town from the hill
for
/ta ka da i ni no bo t te shi ga i wo fu ka n su ru/ overlooking the town from the hill
Figure 7: Example of generated melody
kt = arg max Mk ,
(6)
1≤k≤K Mk ≤N
Figure 6: An example of the equal allocation where Mk = m(Yk ) + m(X) + 1. allocation. The sequence of the notes at the top of the figure represents the generated melody, and the squares at the bottom of the figure represent the mora of the lyrics. The mora is a unit in phonology, which is often used in Japanese. Furthermore, most moras are composed of only one consonant and only one vowel in Japanese. For example, the word cho-ko-re-e-to, which means “chocolate”, has five moras. In the example shown in Figure 6, the lyrics of four mora are allocated to eight notes of a melody. For the notes that have no allocation, the system allocates the mora vowel of the note just before. Accordingly, it is important to reduce the number of notes that have no allocation to create a song that is easy to listen to. Because the number of notes depends on the number of strokes of the Chinese character, the system generate lyrics for each Chinese character. We define Yk (1 ≤ k ≤ K) as all example sentences with regard to a Chinese character or an idiom, X as the text string of pronunciation of one Chinese character, m(X) as the number of mora for the text string X, and N as the number of notes that compose the melody. The lyrics are provided by Equation (6).
If the system cannot find an example sentence that satisfies Equation (6) due to lack of notes, the system uses X itself, which is the pronunciation of the Chinese character, as the lyrics.
3.5
Example of generated melody
We present an example of the generated melody. Figure 7 shows the musical score of the melodies*1 . Firstly, the table at the top of Figure 7 shows the information of Chinese characters and example sentences. The text strings enclosed by slashes represent the actual pronunciation in Japanese. Additionally, white space and hyphens are inserted between moras and Chinese characters, respectively. The underlines show the text that corresponds to the Chinese character. Secondly, the lower part of the figure shows the musical score of the melodies that are generated by the proposed system. The two rows above the score denote components, which are colored red, and strokes from the top. The strokes and the *1
The sound of the example is uploaded here. https://youtu.be/QmhsNsYKfsc
overlooking the town from the hill
/fu-ka n/ overlook nettle rash spread over the body
/ji n-ma-shi n/ nettle rash
Figure 8: Screenshot of prototype system Figure 9: Example of question sheet rectangles that surround the strokes are positioned above the corresponding melodies. In the example of Figure 7, the melody is generated from the idiom of “俯瞰”; the pronunciation of which is /fu-kan/, and the meaning of which is overlook. Since both of the structures of each Chinese character are left to right, the proposed system allocates the chords C and G to the chord progression. The number of notes of each Chinese character are 25 and 32. Accordingly, both of the example sentences chosen by the system are /ta ka da i ni no bo t te shi ga i wo fu ka n su ru/ in the pronunciation description, which literally means “overlooking the town from the hill”. As shown in the lyrics of Figure 7, the final lyrics are the text strings of the example sentences (/ta ka da i ni no bo t te shi ga i wo fu ka n su ru/) and the pronunciations of Chinese characters (/fu/ and /ka n/) connected by the postpositional particle no.
4. IMPLEMENTATION We implemented the prototype system mentioned in section 3. Figure 8 shows the screenshot of the prototype system. The prototype system has text boxes for inputting a Chinese character and its pronunciation at the upper left of the figure and for inputting an example sentence at the upper right of the figure. When the button on the upper right of Figure 8 is pushed, the prototype system outputs the lyrics and melody in the format of musicXML [9]. Simultaneously, the system presents the Chinese character, its structure, and its strokes as shown in Figure 8. The prototype system has a dataset of Chinese characters, their structures, and their strokes in XML format and search data from input Chinese characters. We used the data of 20,903 Chinese characters from the CHISE [10] (CHaracter Information Service Environment) as the structure data. With regard to stroke data, we input data of 2,136 Chinese characters manually. We developed the prototype system using Microsoft Visual C# 2013.
5. EVALUATION In our research, we evaluated the proposed system using the prototype system.
5.1 Evaluation procedure We evaluated the proposed method compared to the method of memorizing the Chinese characters without any learning
support. The participants memorized 15 idioms of Chinese characters in both methods. We measured the number of recalled Chinese characters after three days. The details of each method are as follows: Conventional method Memorizing Chinese characters by only looking at a list that consists of Chinese characters and their example sentences. Proposed method Memorizing Chinese characters by listening to the generated sound and using the list that is the same as that used in the conventional method. Note that the participants were permitted to speak and move their mouths, but not permitted to write down the characters during the experiment. The evaluation procedure is as follows: (i) The participants received a question sheet and answered the dictation questions of Chinese characters. We call the answers answered at this time before learning. (ii) The participants memorized the Chinese characters using each method until they had memorized them completely. (iii) Three days after finishing memorizing the characters, the participants answered the same dictation questions as used in phase (i). We call the answers answered at this time after learning. Additionally, the participants who used the proposed method were permitted to listen to the song used in phase (ii) if they are completely unable to recall the answer. They were required to write down whether they had to listen to the song to answer the question. Figure 9 shows an example of the question sheet used in phase (i) and phase (iii). The question sheet has example sentences. Note that the question sheet that is actually used in the evaluation is written only in Japanese. There are no annotations like the blue and orange rectangles shown in Figure 9. Chinese characters for sentence questions are shown with other Japanese characters (called katakana), which are underlined, that represent the pronunciation. The text in the blue rectangle in Figure 9 is the pronunciation text of the Chinese character for a question using the alphabet and
the text that is translated into English. Hyphens and white spaces written in the pronunciation text are inserted between Chinese characters and moras, respectively. In other words, the example at the top is three moras and is described by two Chinese characters, and the example at the bottom is five moras and is described by three Chinese characters. Moreover, the text in the orange rectangle in the figure is the text of the example sentence translated into English. The participants are required to answer 15 questions in five minutes. Note that the order of the example sentences for questions used in phase (i) and phase (iii) is different. In addition, the sentences used in phase (ii) are the same as those written on the question sheet.
Table 2: Number of correctly answered Chinese characters
5.2 Participants
assumed that this was caused by the shortness of the memorizing time and being unfamiliar with the melodies generated by the proposed system. Accordingly, we should improve the evaluation method by, for example, setting the training term so that the participants are familiar with the melodies of the proposed method. Secondly, there is the opinion that it is necessary to connect visual information and auditory information. Therefore, we should propose a visual presentation method such as highlighting the stroke corresponding to the sound that is currently played. Finally, a participant stated that while he was not aware of the lyrics just after beginning memorizing, he become aware of them gradually. Furthermore, another participant mentioned that because it is good for learning Chinese characters to recall them by an example sentence, it is necessary to use example sentences as lyrics. This supports the necessity of using example sentences as lyrics.
There were six participants, all of whom are native speakers of Japanese. Furthermore, they are all graduate school students who major in information engineering. Three of them memorized the characters by the conventional method, and the others memorized them by the proposed method.
5.3 Questions and songs We took the 15 dictation questions for this evaluation from level 1 of the Japan Kanji Aptitude Test held by the Japan Kanji Aptitude Testing Public Interest Foundation [11]. The reason we chose level 1 is that it is difficult for a native speaker of Japanese to guess the correct answer. Note that the 15 questions we chose for evaluation include 25 Chinese characters. The song used in this evaluation is generated as follows: • Convert the musicXML file generated by the prototype system into a voice file (WAV format) using CeVIO [12], which is voice synthesis software. • Generate the song file (MP3 format) by giving the piano accompaniment as chord information to the voice file using GarageBand [13], which is software for creating music.
5.4 Result and discussion Table 2 shows the number of Chinese characters that the participant answered correct. The values outside the brackets are the sum of the correct answers without listening to the song, and the values in the brackets are the sum of all the correct answers. Additionally, the values on the bottom row represent the mean and the standard deviation in each method. With regard to the correct answers without listening, according to the result of analysis of variance, there is no significant difference between the methods in before learning (F(1,4) = 0.08), and there is a significant tendency between methods in after learning (F(1,4) = 6.32, p < .10). Similarly, with regard to the sum of all correct answers, according to the result of analysis of variance, there is no significant difference between methods in before learning (F(1,4) = 0.08), and there is a significant tendency between methods in after learning (F(1,4) = 6.04, p < .10). As a result, there is no advantage when using the proposed method. We discuss what causes this result. Firstly, we discuss whether the participants were able to associate the melodies with strokes. All the participants who memorized the characters by the proposed method stated that although they could have associated the melodies, they could not recall the melodies and the characters in the after learning. It is
Method
Conventional method
Proposed method
6.
Participant P1 P2 P3 Mean S.D. P4 P5 P6 Mean S.D.
Before learning 2 2 2 2 0.00 4 3 0 2.3 1.70
After learning 19 24 23 22 2.16 16(17) 17(17) 6(7) 12.7(13.3) 4.78(4.50)
APPLICATION EXAMPLES
Finally, we explain the application examples of the proposed method. The main feature of our research is representing stroke information, such as the strokes of Chinese characters, and enabling learners to associate the strokes and the meanings of the characters with music. There are several situations when we have to memorize stroke information without memorizing characters. We explain the application examples with Figure 10 and the following comments. • Memorization of ideographic commands and motions There is some application software that requires stroke motion, such as Graffiti of Palm [14] and Glyph of Ingress [15]. Figure 10(a) shows an example of ideographic command. Another examples is memorizing the motion of a magic wand as shown in Figure 10(b). This motion often appears in fantasy stories, movies, and video games. To make the game more enjoyable, it is important to memorize these motions. • Memorization of route Figure 10(c) shows an example of the route within a building. The proposed method can be applied to memorize a short cut in a building or a route from the exit to the parking location as shown in the figure.
7.
CONCLUSION AND FUTURE WORK
We proposed a system for memorizing Chinese characters using a song based on the strokes and structures of the characters. The proposed system supports memorizing Chinese
Magic of xxx
Command A
(a) Ideographic command
(b) Motion of magic wand
Destination
You are here
(c) Memorization of route
Figure 10: Examples of application
characters by the song, which is composed of a melody generated based on the structures and strokes, and the lyrics using the example sentence. We compared the proposed method with the conventional method. As a result, although there were no advantages when using the proposed method, we obtained many useful ideas for improvements. Future work includes creating a visual presentation method, such as highlighting the stroke corresponding to the sound. We should also improve the experiment procedure by, for example, setting the training term so that the participants can be familiar with the generated melody. Additionally, we intend to develop an appropriate allocation method for lyrics and a melody generation method to make memorizing characters easier.
8. ACKNOWLEDGMENTS This research was supported in part by a Grant-in-Aid for Scientific Research (A)(23240010) and by Super Cluster Program (Kyoto Area, Construction of Highly-Efficient Energy to Achieve a Clean and Low Environmental Impact Society) from the Japan Science and Technology Agency.
9. REFERENCES [1] The Unicode Consortium, The Unicode Standard, http://www.unicode.org/versions/latest/. [2] C. Teo, E. Burdet, and H. Lim: A robotic teacher of Chinese handwriting, Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2002. HAPTICS 2002. Proceedings. 10th Symposium on, pp. 335–341 (2002). [3] Y. Wen, L. P. Prieto, and P. Dillenbourg: Paper-Based Tabletop Application for Collaborative Chinese Character Learning, Proc. of the 11th International Conference on Computer Supported Collaborative Learning (CSCL2015), pp. 665–666 (2015).
[4] R. Oxford and D. Crookall: Vocabulary learning: A critical analysis of techniques, TESL Canada Journal, Vol. 7, No. 2, pp. 09–30 (1990). [5] S. Abboud, S. Hanassy, S. Levy-Tzedek, S. Maidenbaum, and A. Amedi: EyeMusic: Introducing a “Visual” Colorful Experience for the Blind Using Auditory Sensory Substitution, Restorative Neurology and Neuroscience, Vol. 32, No. 2, pp. 247–257 (2014). [6] C. L. Yiu and W. Wong: Chinese character synthesis using METAPOST, Proc. of TUG, pp. 85–93 (2003). [7] S. Fukayama, D. Saito, and S. Sagayama: Assistance for Novice Users on Creating Songs from Japanese Lyrics, The 2012 International Computer Music Conference (ICMC2012), pp. 441–446 (2012). [8] R. E. Bellman: Dynamic Programming, Princeton University Press (1957). [9] MusicXML, http://www.musicxml.com/. [10] CHISE: Character Information Service Environment, http://kanji.zinbun.kyoto-u.ac.jp/projects/chise/ids/. [11] Japan Kanji Aptitude Testing Public Interest Foundation, http://www.kanken.or.jp/. [12] CEVIO Official Site, http://cevio.jp/. [13] GarageBand, http://www.apple.com/jp/mac/garageband/. [14] A. Butter and D. Pogue: Piloting Palm: The inside story of Palm, Handspring, and the birth of the billion-dollar handheld industry, John Wiley & Sons (2002). [15] Ingress, https://www.ingress.com/intel.
