akaza-corpus-stats — Generated Data Notice
===========================================

This archive contains statistical language data (n-gram wordcount tries
and vocabulary files) generated by the akaza-corpus-stats pipeline.
These files are derived from the following data sources:


1. Japanese Wikipedia
---------------------
Source: https://dumps.wikimedia.org/
License: Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)
         https://creativecommons.org/licenses/by-sa/4.0/
Copyright: Wikimedia Foundation and contributors

The text content of Japanese Wikipedia articles was used to compute
word frequency statistics. The generated files contain aggregated
statistical data (word counts and n-gram frequencies), not article text.

As a derivative work of Wikipedia content, redistribution of the
generated data is subject to the terms of CC BY-SA 4.0.


2. Aozora Bunko (青空文庫)
--------------------------
Source: https://www.aozora.gr.jp/
License: Public Domain (著作権の消滅した作品)

Texts from Aozora Bunko, a digital archive of Japanese literary works
whose copyrights have expired, were used to compute word frequency
statistics.


3. CC-100 Japanese
-------------------
Source: https://data.statmt.org/cc-100/
Paper: Conneau et al., "Unsupervised Cross-lingual Representation Learning
       at Scale", ACL 2020. https://aclanthology.org/2020.lrec-1.494/
License: The CC-100 corpus was extracted from Common Crawl data.

The Japanese portion of the CC-100 corpus was used to compute word
frequency statistics. The generated files contain aggregated statistical
data (word counts and n-gram frequencies), not the original text.


4. IPADIC Dictionary
--------------------
Source: https://github.com/daac-tools/vibrato
License: BSD-3-Clause (IPADIC), Apache-2.0 (Vibrato)

The IPADIC dictionary was used by the Vibrato tokenizer for
morphological analysis during the generation process. The dictionary
data is not included in the generated files.


Generated by
-------------
akaza-corpus-stats: https://github.com/akaza-im/akaza-corpus-stats
Pipeline scripts are licensed under MIT License.
