Archiving and Analysing Techniques of the Ultra-Large-Scale Web-Based Corpus Project of NINJAL,Japan

Abstract

In 2011, the National Institute for Japanese Language and Linguistics (NINJAL) launched a corpus compilation project to construct a web corpus for linguistic research comprising ten billion words by 2016. The project is divided into four categories: Page Collection, Linguistic Annotation, Release and Preservation. For Page Collection, web crawlers are employed to collect web text by crawling 100 million pages every three months and retaining several versions of the text for three-month periods. For Linguistic Annotation, the linguistic studies web corpus contains annotated linguistic information. To improve the usability of these linguistic resources, normalization tasks such as tag removal, word segmentation, dependency parsing, and register estimation are performed. For Release, word lists and n-gram data are published based on the crawled and annotated text corpus. In addition, applications are being developed to enable searching for morphosyntax patterns in the ten-billion-word corpus. For Preservation, crawled web pages are preserved in chronological order as web archives primarily to support the survey of ongoing linguistic changes. In this paper, we present the basic design of the four categories. Additionally, we report the current status of the corpus using basic statistics of the crawled data and discuss the importance of deduplicating sentences.

Keywords

web archive web corpus Japanese language resources crawling linguistic annotation

Get full access to this article

View all access options for this article.

References

Baroni Marco Ueyama Motoko (2006) ‘Building general- and special-purpose corpora by web crawling’. Proceedings of the 13th NIJL International Symposium, Language Corpora: Their Compilation and Application. Tokyo, Japan, March, 31–40.

Kawahara Daisuke Kurohashi Sadao (2006) ‘Case frame compilation from the web using high-performance computing’. Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC-2006), Genoa, Italy, May, 1344–1347.

Kilgarriff Adam Reddy Siva Pomikálek Jan Pvs Avinesh (2010) ‘A corpus factory for many languages’, Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC-2010), Malta, May, 904–910.

Maekawa Kikuo Yamazaki Makoto Ogiso Toshinobu Maruyama Takehiko Ogura Hideki Kashino Wakako Koiso Hanae Yamaguchi Masaya Tanaka Makiro Den Yasuharu (2013) ‘Balanced corpus of contemporary written Japanese’. Language Resources and Evaluation. Berlin: Springer.

Mochihashi Daichi Yamada Takeshi Ueda Naonori (2009) ‘Bayesian unsupervised word segmentation with nested Pitman-Yor language modeling’. Proceedings of Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing Volume 1 (ACL-IJCNLP-2009), Singapore, August, pp. 100–108.

Pomikálek Jan Suchomel Vít (2012) ‘Efficient web crawling for large text corpora’. Proceedings of the Seventh Web as Corpus Workshop (WAC7), pp. 39–43.

Shinzato Keiji Shibata Tomohide Kawahara Daisuke Hashimoto Chikara Kurohashi Sadao (2008) ‘TSUBAKI: An open search engine infrastructure for developing new information access’. Proceedings of Third International Joint Conference on Natural Language Processing (IJCNLP2008), Hyderabad, India, January

Srdanović Irena

Tomaž Erjavec Kilgarriff Adam (2008) ‘A web corpus and word-sketches for Japanese’. Shizen gengo shori (Journal of Natural Language Processing), 15(2), pp. 137–159.

Ueyama Motoko Baroni Marco (2005) ‘Automated construction and evaluation of Japanese web-based reference corpora’. Proceedings of Corpus Linguistics 2005. Birmingham, UK, July.