Chapter. 1 Introduction
Yüklə 1,52 Mb.
|
- Bu səhifədəki naviqasiya:
- Label Variants f1
- Total 1817 457
- Inter-annotator Agreement
- Kappa Statistics Strength of Agreement
- Observed Agreement Expected Agreement Kappa Value
- Label Attachment Score (LAS) Agreement on Labels (LA) Agreement on Attachments (UAS)
- S. NO Labels Confusions
Statistical Results As mentioned in the chapter five, the three datasets that have been used for the current task consist of 682 POS annotated sentences of varying lengths, taken from three different text domains, i.e. newspaper editorials (ASL = 25 Ws or 15 Cs), short-stories (ASL = 11 Ws or 8 Cs) and critical discourse (ASL = 16 Ws or 10Cs), are partially parsed into 8125 chunks. The task with which this chapter is concerned is to deduce, annotate and find score for each inter-chunk GR, holding among 8125 chunks in 682 structures. In aggregate, 4287 GRs have been found holding under 682 dependency structures among 8125 chunks. The 4287 GRs are further classified under 25 labels, each with its frequency count in three different domains and also in aggregate, as shown in the Table.2. However, the score for each attachment label is given in underspecified manner, i.e. no separate frequency score of the variants is given.
Table.2. Showing Frequency Distribution of GRs The empirical facts given in pie chart in the Fig.24 reveal that ccof is the most frequent GR which covers 20% of the total GRs. Therefore, co-ordination and the sub-ordination form the bulk of grammatical operations occurring Kashmiri text. Fragof constitutes 8% of the total GRs found in Kashmiri, indicating the strength of V2-phenomenon. Pof constitutes 7% of total GRs showing the significant occurrence of complex predicates in Kashmiri. Similarly, k1 constitutes 14% and k2 constitutes 12% of the relational bulk of Kashmiri text, indicating that SUBs and OBJs constitute 26% GRs in aggregate which is quite significant. It is interesting to see that quantitatively, k1, k2, ccof and fragof together cover more than half of the total relational bulk. These facts further reveal that 39-40% GRs in Kashmiri are karakas and rest, about 60%, are non karakas and 65% of GRs are dependency relations while as 35% of the relations are non-dependencies in which 6% are non-rooted dependencies, i.e. the attachments are made with non-root heads (in genitive, participial and relative clause modifiers). 16% of GRs are adverbial modifiers and only 1% of GRs are relative clause modifiers. Finally, it is important to point out that only 30% GRs belong to sub-categorization frame, thus, represent the arguments relations while as the 61% of GRs fall outside the sub-categorization frame, thus, represent adjunct relations. 
Figure.24 Showing Proportion of Each GR
One of the biggest challenges to a treebank project is maintaining consistency in annotations. It includes both, achieving significant inter-annotator and intra-annotator agreement. To check the inter-annotator agreement, two independent annotators need to annotate the same data with while as intra-annotator agreement can be achieved if an annotator encounters the same constructions or phenomenon many times during the course of annotation, the annotator annotates them consistently by sticking to the previous decisions regarding. Since, consistency increases the usefulness of the data for training or testing automatic methods for linguistic investigations. The understanding of various linguistic phenomena and the annotation guidelines is also often reflected in inter-annotator agreement studies. In order to check the consistency in the annotations of the current treebank, a dataset of 200 sentences was annotated by two annotators who had proper understanding of various issues and the guidelines for Kashmiri treebank. When the two annotated datasets were compared, a confusion matrix was formulated as shown in the Table.6. The matrix shows for which label and for how many times there is confusion. For example: in the first row of the table, there is confusion of adv with rt one times, with vmod two times, k7p two times, nmod one times, k2 one times, k7 one times, k7t one times and pof one times. Inter-annotator agreement was measured using Cohen’s kappa (Cohen, at al., 1960) which is the mostly used agreement coefficient for annotation tasks with categorical data. Kappa was introduced to the field of computational linguistics by (Carletta et al., 1997) and since then many linguistics resources have been evaluated using the matrix such as (Uria et al., 2009; Bond et al., 2008; Yong and Foo, 1999). The kappa statistics show the agreement between the annotators and the reproducibility of their annotated datasets. However, a good inter-annotator agreement does not necessarily ensure accuracy of attachment labels as the annotators can make similar kind of mistakes and errors. The kappa coefficient k is calculated as:
Pr (a) is the observed agreement between the annotators and Pr (e) is the expected agreement, i.e. the probability that the annotators agree by chance. Based on the interpretation matrix of kappa value proposed by Landis and Koch (Landis and Koch, 1977) as shown in Table.3, the agreement between two annotators on the data set used for the evaluation is reliable as given in the Table.4. There is a substantial amount of inter-annotator agreement which implies that there is similar understanding of the annotation guidelines and of the linguistic phenomenon found in the data. The label attachment score, agreement on only labels and agreement on only attachments are given in Table.5.
Table.3. Coefficients for the Agreement Rate
Table.4. Kappa Statistics
Table.5. Kappa Statistics
Table.6. Confusion Matrix Showing Disagreement Labels
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
