Manually Annotated Sub-Corpus
MASC is a balanced subset of 500K words of written texts and transcribed speech drawn primarily from the Open American National Corpus (OANC). The OANC is a 15 million word (and growing) corpus of American English produced since 1990, all of which is in the public domain or otherwise free of usage and redistribution restrictions.
All of MASC includes manually validated annotations for logical structure (headings, sections, paragraphs, etc.), sentence boundaries, three different tokenizations with associated part of speech tags, shallow parse (noun and verb chunks), named entities (person, location, organization, date and time), and Penn Treebank syntax. Additional manually produced or validated annotations have been produced by the MASC project for portions of the sub-corpus, including full-text annotation for FrameNet frame elements and a 100K+ sentence corpus with WordNet 3.1 sense tags, of which one-tenth are also annotated for FrameNet frame elements. Annotations of all or portions of the sub-corpus for a wide variety of other linguistic phenomena have been contributed by other projects, including PropBank, TimeBank, MPQA opinion, and several others. Co-reference annotations and clause boundaries of the entire MASC corpus are scheduled to be released by the end of 2016.
WordNet sense annotations for all occurrences of 114 words are also included in the MASC distribution, as well as FrameNet annotations for 50-100 occurrences of each of the 114 words. The sentences with WordNet and FrameNet annotations are also distributed as a part of the MASC Sentence Corpus.
Genres
Unlike most freely available corpora including a wide variety of linguistic annotations, MASC contains a balanced selection of texts from a broad range of genres:
| Genre | No. files | No. words | Pct corpus |
|---|---|---|---|
| Court transcript | 2 | 30052 | 6% |
| Debate transcript | 2 | 32325 | 6% |
| 78 | 27642 | 6% | |
| Essay | 7 | 25590 | 5% |
| Fiction | 5 | 31518 | 6% |
| Gov’t documents | 5 | 24578 | 5% |
| Journal | 10 | 25635 | 5% |
| Letters | 40 | 23325 | 5% |
| Newspaper | 41 | 23545 | 5% |
| Non-fiction | 4 | 25182 | 5% |
| Spoken | 11 | 25783 | 5% |
| Technical | 8 | 27895 | 6% |
| Travel guides | 7 | 26708 | 5% |
| 2 | 24180 | 5% | |
| Blog | 21 | 28199 | 6% |
| Ficlets | 5 | 26299 | 5% |
| Movie script | 2 | 28240 | 6% |
| Spam | 110 | 23490 | 5% |
| Jokes | 16 | 26582 | 5% |
| TOTAL | 376 | 506768 |
Annotations
At present, MASC includes seventeen different types of linguistic annotation (* = in production; ** currently available in original format only):
| Annotation type | No. words |
|---|---|
| Logical | 506768 |
| Token | 506768 |
| Sentence | 506768 |
| POS/lemma (GATE) | 506768 |
| POS (Penn Treebank) | 506768 |
| POS (FrameNet) | 506768 |
| Noun chunks | 506768 |
| Verb chunks | 506768 |
| Named Entities (person, org, loc, date) | 506768 |
| Penn Treebank syntax | 506768 |
| Coreference | *506768 |
| Clause boundaries, nucleus/satellite distinctions, discourse markers | *506768 |
| FrameNet frames/frame elements | 39160 |
| PropBank | **88530 |
| Opinion | 51243 |
| TimeBank | *55599 |
| Committed Belief | 4614 |
| Event | 4614 |
| Dependency treebank | **5434 |
| Lexical substitution | **35,547 |
All MASC annotations, whether contributed or produced in-house, are transduced to the Graph Annotation Format (GrAF) defined by ISO TC37 SC4’s Linguistic Annotation Framework (LAF). The online tool ANC2Go can transduce annotations over all or parts of MASC to any of several other formats, including CONLL IOB format and formats for use in UIMA and General Architecture for Text Engineering.
Distribution
MASC is an open data resource that can be used by anyone for any purpose. At the same time, it is a collaborative community resource that is sustained by community contributions of annotations and derived data. It is freely downloadable from the MASC download page or through the Linguistic Data Consortium.
MASC is also distributed in part-of-speech-tagged form with the Natural Language Toolkit.
See also
References
- Ide, N., Baker, C., Fellbaum, C., Passonneau, R. (2010). The Manually Annotated Sub-Corpus: A Community Resource For and By the People. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden.
- Passonneau, R., Baker, C., Fellbaum, C., Ide, N. (2012). The MASC Word Sense Sentence Corpus. Proceedings of the Eighth Language Resources and Evaluation Conference, Istanbul.
- Ide, N., Suderman, K., Simms, B. (2010). ANC2Go: A Web Application for Customized Corpus Creation. Proceedings of the Seventh Language Resources and Evaluation Conference (LREC 2010), Valletta, Malta.