https://github.com/tensorflow/models/tree/master/lm_1b
Introduction
“In this release, we open source a model trained on the One Billion Word Benchmark (http://arxiv.org/abs/1312.3005), a large language corpus in English which was released in 2013. This dataset contains about one billion words, and has a vocabulary size of about 800K words. It contains mostly news data. Since sentences in the training set are shuffled, models can ignore the context and focus on sentence level language modeling.
In the original release and subsequent work, people have used the same test set to train models on this dataset as a standard benchmark for language modeling. Recently, we wrote an article (http://arxiv.org/abs/1602.02410) describing a model hybrid between character CNN, a large and deep LSTM, and a specific Softmax architecture which allowed us to train the best model on this dataset thus far, almost halving the best perplexity previously obtained by others.”
@article{jozefowicz2016exploring, title={Exploring the Limits of Language Modeling}, author={Jozefowicz, Rafal and Vinyals, Oriol and Schuster, Mike and Shazeer, Noam and Wu, Yonghui}, journal={arXiv preprint arXiv:1602.02410}, year={2016} }