Abstract. This paper proposes a fast and effective question-answer system for encyclopedia domain using a new answer indexing method. We define about.
A LF Based Answer Indexing Method for Encyclopedia Question-Answering System Hyeon-Jin Kim, Ji-Hyun Wang, Chang-Ki Lee, Chung-Hee Lee, and Myung-Gil Jang 161 Gajeong-dong, Yuseong-gu, Daejeon, 305-350, Korea {jini, jhwang, leeck, forever, mgjang}@etri.re.kr http://km.etri.re.kr/index.php
Abstract. This paper proposes a fast and effective question-answer system for encyclopedia domain using a new answer indexing method. We define about 160 answer types. The indexer generates AIU(Answer Index Unit) structures between answer candidates and content words within LF(Logical Form) and sentence boundary. We select essential terms among question terms using syntactic information for ranking the answer candidates. Experiments show our new method is good for the encyclopedia question-answering system.
1 Introduction Question-Answering(QA) processing has been attracting a great deal of attention recently[1]. Many researches on question answering have been carried out to make up for the weak points in IR systems.[2,3]. We have implemented AnyQuestion2.0(http://anyQ.etri.re.kr) which is a encyclopedia question-answering system following AnyQuestion1.0 in 2003[4]. AnyQuestion1.0 is a question-answering system that extracts a short-answer for user’s question only in the person category of encyclopedia domain. The method for finding answers is 3-step answer process(which is the combination of IE-supported QA technique and passage extraction method). However, it takes long time to extract an answer in AnyQuestion1.0 on account of passage extraction method. In AnyQuestion2.0, we adopt answer indexing method instead of passage extraction method to solve this problem and improve IE-supported QA method. Moreover, we extend category from only person to entire domain of encyclopedia and add up techniques to be able to process descriptive answer(for questions that ask for definition, reason, method, objective etc.) as well as short answer. In this paper, we focus on the question answering technique using answer indexing method among techniques of AnyQuestion2.0. This paper is structured as follows: In the second section, we review the related works. Section 3 describes answer indexing technique and section 4 explains our answer processing method. Results of our evaluation test are presented in section 5. Finally, we draw our conclusions. G.G. Lee et al. (Eds.): AIRS 2005, LNCS 3689, pp. 679 – 684, 2005. © Springer-Verlag Berlin Heidelberg 2005
680
H.-J. Kim et al.
2 Related Works In current techniques for QA, there are IE(Information Extraction)-supported QA technique[5], passage extraction method based on IR system[6], the technique of answer indexing[7,8], and so on. [5] is a typical IE-supported QA system. They define templates about each entity(for example: person entity) and then fill up template values using IE techniques. In a closed domain such as encyclopedia texts or newspapers, IE-supported method is successful[6,9] but is an impractical solution in opendomain due to the dependency of IE systems on domain knowledge. Therefore, passage extraction methods have been the most commonly used ones by many QA systems. In the passage extraction methods, sentences or passages which are the most relevant to the question are extracted and then answers are retrieved by using lexicosyntactic information or NLP techniques[6]. However, it takes a long time to extract an answer in these QA systems because rules should be applied to each sentence including answer candidates on the retrieval time[7]. To overcome this problem, [7,8] uses a method for indexing answer candidates in advance. In [7,8], they define answer types(they called QA token or semantic category) and identify answer candidates in a text and then index them. Especially, [7] uses a predictive answer indexer based on 2pass scoring method. However, [7,8] uses not high-level information(grammatical role or dependency structure etc.) but low-level information like the term frequencies and the distances.
3 Answer Index For the indexing module, we rely on natural language processing techniques including morphological process, word sense disambiguation, answer type tagging (similar to the extended named entity recognizer) and syntactic analysis. We define about 160 answer types in consideration of user’s asking points for finding answer candidates. They have 15 top levels and each top node consists of 2 or 4 layers. The base set of such types is; PERSON, STUDY_FIELD, THEORY, ARTIFACTS, ORGANIZATION, LOCATION, CIVILIZATION, DATE, TIME, QUANTITY, EVENT, ANIMAL, PLANT, MATERIAL, and TERM. The AT-tagging engine annotates with answer types(AT) for each sentence and then the indexer generates AIU(Answer Index Unit) structures using the answer candidates(the AT annotated words) and the content words which can be founded within the same context boundary. We adopt LF(Logical Form) and sentence as the context boundary. First, the LF-based method extracts the AIU structures within the same LF relations. We defined LF(logical form) as the syntactic relation between a verb and other predicate arguments in the same dependency structure[4]. (e.g. verb(x, y, z,…) ). We classify these predicate arguments and verbs into three sets; answer candidate word set(the AT annotated words), verb set and content word set(noun, compound noun, adverb, genitive phrase etc). We construct AIU structures based on the following formula and then add up LF information(S:subject, O:object, V:verb, A:adverb) for each AIU structures.
A LF Based Answer Indexing Method for Encyclopedia Question-Answering System
C (content words set) = {c1 , c 2 ,c3 ...... cl } V (verb set) = {v1 , v2 ,v3 ...... v m } A (answer candidate words set) = {a1 , a 2 , a3 ...... a n } AnswerIndexUnit = ( ci , a j ) ci ∈ C , a j ∈ A AnswerIndexUnit _ forVerb = ( ci , v j , a k ) ci
∈ C,
vj
681
(1) ∈V ,
ak
∈A
Second, the sentence-based method constructs the AIU structures within a same sentence. In other words, we adopt AIU structures between answer candidate word set and content word set within a sentence boundary with the exception of same AIU structures which is extracted from the first method. In addition, we exclude verb set in this method because verbs produce side effects even if they appear with the answer candidates within the same sentence, so verb set is restricted in LF-based method. We also prevent effectively over-generation of AIU structures. We append distance information between answer candidates and content words to AIU structures. Fig.1 contains as example of the data structure passed from the indexing module. [example sentence] The Nightingale Award, the top honor in international nursing, was established at the International Red Cross in 1912 and is presented every two years (title: Nightingale) [AT-tagging process] , the top honor in international nursing, was established at in and is presented every (title: ) [answer candidates] The Nightingale Award:CV_PRIZE the International Red Cross:OGG_SOCIETY 1912:DT_YEAR two years:DT_DURATION Nightingale:PS_NAME [LF structures] •establish (, , ) •present (, ) AnswerIndexUnit
AnswerIndexUnit_Verb
(the International Red Cross, The Nightingale Award):Obj ect-Subject (1912, The Nightingale Award):Adverb-Subject (two year, The Nightingale Award)Adverb-Subject etc. (the International Red Cross, establish, The Nightingale A ward):Verb-Subject (1912, establish, The Nightingale Award):Verb-Subject etc .
AnswerIndexUnit
(1912, two year):distance info. (two year, the International Red Cross):distance info etc.
Fig. 1. Example of index processing
682
H.-J. Kim et al.
4 Answer Processing The answer processing module searches the relative answer candidates from index DB using question analysis and calculates the similarities and then extracts answers. This module is composed of question term- weighting part and answer-ranking part. First, the question term weight module assigns weight to each question terms to get potential answers. We choose essential terms among question terms according to the following equation. ScoreE ( Sei ) = ∑ wi * sf i i
sf1 = Title point ( title of encyclopedia)
(2)
sf 2 = LF point ∈ {subject point, object point, adverb point} sf3 = AT point ∈ {PLO point, notPLO point} wi = each feature weight
Equation 2 shows three features; title point(if question term is title of encyclopedia), LF point(if term is LF arguments(Subject, Object, Adverb)) and AT point (if term is answer type). Especially, in case of AT point, if the term is PLO type(Person, Location, Organization), it will be added to more points than not PLO type. Essential term is a question term which has the highest scores. After selection of essential term, we assign term weight as in the following equation. After assigning term weight, we calculate the similarities between query terms and answer candidates. To compute similarities, we use AND operation of a p-Norm mode. if essential terms Q(Weight) = extra_each_weight + extra_plus_weight * (
1
)
max_score_count
else Q(Weight) = extra_each_weight − extra_plus_weight max_ score _ count : count of essential terms total _ query _ count : total count of question terms total _ verb _ query _ count : total count of verbs verb _ weight = (total_verb_query_count / total_query_count)* Wv Wv : constant value for verb weight
(3)
extra _ total _ weight : 1 - verb_weight extra _ each _ weight = extra_total_weight / total_query_count extra_plus_weight = max_score_count / total_query_count
Second, the input to the answer ranking part is results of relative answer candidates searched from index DB. The answer candidates are ranked accordingly to the following equation. In this system the final score is combination of first score(Wi)and second score(Score(Ri)). First score is calculated from sum of Q(weight) described in previous section. Second score(Score(Ri)) has following tree types of weight. • Document matching weight : existence weight of title information recommended from question analysis. Ex) where was Gauss born? (recommended title is ‘Gauss’. If answer candidate exists in ‘Gauss’ document, this weight is added.)
A LF Based Answer Indexing Method for Encyclopedia Question-Answering System
683
• Distance weight : distance weight between query term and answer candidate. (LF boundary > within 1 or 3 words boundary > 3 words over boundary) • Occurrence weight : number of appearing counts of the same answer candidate in high rank set (In this system, we assign scores to candidate answer according to their frequency in 30 ranked answers.) Score( Ai ) = α * wi + (1 − α ) Score( Ri ) Score( Ri ) = swi * sf i
∑
wi =
∑ Q ( wi )
i
(4)
i sf1 = Document matching weight sf 2 = Distance weight sf 3 = Occurrence weight swi = feature weight
5 Experiments To experiment on our system, we use ETRI QA Test Set[5] which consists of 402 pairs of question and answer in encyclopedia. Our encyclopedia currently consists of 163,535 entries, 13 main categories, and 41 sub categories in Korean. For each question, the performance score is computed as the reciprocal answer rank(RAR) of the first correct answer. To compute the overall performance of AnyQuestion2.0, we use the Mean Reciprocal Answer Rank(MRAR). We consider 5 answers in the highest ranks as the answer candidates. For this experiment, we used 402 pairs of the evaluation set. Table1 shows the result of AnyQuestion1.0(using passage retrieval system). The performance of the proposed method in this paper is shown in Table2. Table3 summarizes the result of another AnyQuestion2.0 except for LF based method. Table 1. Result of AnyQuestion1.0 Number of correct answer MRAR
1
2
3
4
5
188
25
13
3
2
0.51
Table 2. Result of AnyQuestion2.0 (using LF) Number of correct answer MRAR
1
2
3
4
5
185
25
10
3
1
0.50
Table 3. Result of AnyQuestion2.0 (not using LF) Number of correct answer MRAR
1
2
3
4
5
149
38
15
4
5
0.43
684
H.-J. Kim et al.
As shown in Table1 and Table2, the performance of AnyQuestion2.0(using LF, Table3) is similar to that of AnyQuestion1.0(using passage retrieval system, Table1). The problem of AnyQustion1.0 is that the average response time of this system exceeds 5 second, while AnyQuestion2.0 takes less than the maximum 0.5 second. This result means AnyQuestion2.0 is more useful. Moreover we tested performance of LFbased indexing method. From Table2 and 3, we see that the accuracy of AnyQuestion2.0 using LF method is higher than that of AnyQuestion2.0 excluding LF method. The fact shows that the proposed method in this paper is good for the encyclopedia question-answering system.
6 Conclusion We presented a fast and effective question-answer system for encyclopedia domain. We focus on answer indexing method based on syntactic relation in this paper. For answer indexing method, we classified user’s asking points into 160 answer types. We explained how our system generated AIU(Answer Index Unit) structures within LF and sentence boundary in indexing process. For ranking the answer candidates, we chose essential terms using syntactic information. We have shown that our proposed method is more useful with some experiments. While these methods have improved our previous QA system, we note that more improvements may be pursed in future work. We have to construct more practical QA system not only for the encyclopedia domain but also for other domain. Further work includes the fine-tuning of current system and we plan to expand our AnyQuestion system to be able to process list type questions.
References 1. Toru Takaki : NTT DATA TREC-9 Question-Answering Track Report, TREC 2000 2. Ellen M, Voorhees: Overview of TREC 2003 Question Answering Track. The Proceedings of the twelfth Text REtreival Conference(TREC-12), November 2003. 3. Harabagiu S., Moldovan D., Pasca M., et al.: FALCON: Boosting Knowledge for Answer Engines TREC-9, 2000. 4. H. J. Kim, H. J. Oh, C. H. Lee., et al.: The 3-step Answer Processing Method for Encyclopedia Question-Answering System: AnyQuestion 1.0. The Proceedings of Asia Information Retrieval Symposium (AIRS) (2004) 309-312 5. Wei Li, Rohini K. Srihari : Extracting Extract Answers to Questions Based Structural Links, Coling-2002 6. Sanda M. Harabagiu, Steven J. Maiorano: Finding Answers in Large Collections of Texts: Paragraph Indexing + Abductive Inference, AAAI-1999. 7. Harksoo Kim, Jungyun Seo: A Reliable Indexing Method for a Practical QA System, Coling-2002. 8. Prager J., Brown E. and Coden A.: Question-Answering by Predictive Annotation, The proceedings of SIGIR 2000. 9. Julian Kupiec: MURAX: A Robust Linguistic Approach for Question Answering Using Online Encyclopedia, SIGIR 93
