INTRODUCTION

As a non-native speaker in America, language was the first problem I met. So during the first quarter at UCLA, I am thinking about how machine learning can help language learners. Fascinated by the effectiveness of Recurrent Neural Networks, I decided to use it to implement a smart prompter Android App.

Purpose of the Project

The language model should be trained to predict the next word according to a sequence of text history.
The Android App should be able to do inference in real time so trade-off between the model size and predicting precision is necessary.
The input of the whole App should be human voice. However, it was not easy to do speech recognition as well in just one project. I used Google voice recognition service to do this for me.

Prior Work

Recurrent Neural Networks (RNNs)

In a traditional neural network like CNN, inputs are usually independent of each other (it's OK since most of the tasks for CNN is image recognition). However, by using such kind of neural networks, it would be impossible for me to predict a word without knowing the word history before it. Basically, Recurrent Neural Networks calculate the output for each time step based on the current input and previous cell states. In this case, RNNs are able to remember the input history and were proven to have good performance on sequential problems[1].

The problem of Long-Term Dependencies

In theory, RNNs can capture information about what happened in all the previous time steps, but in practice they are limited to looking back only a few steps.

Long Short Term Memory networks (LSTMs)

LSTMs, introduced by Hochreiter & Schmidhuber (1997), are a special kind of RNN, capable of learning long-term dependencies.

The repeating module in a standard RNN contains a single layer.

The repeating module in an LSTM contains four interacting layers.

In an LSTM, it uses three "gate layers" to decide which part of input and previous state will be used to calculate the new cell state and generate the output.

Forget Gate:

Input Gate:

Candidate State:

Cell State:

Output Gate:

Output:

To learn more about LSTMs ,read this great article[2].

Gated Recurrent Unit (GRU)

A slightly more dramatic variation on the LSTM is the Gated Recurrent Unit, or GRU, introduced by Cho, et al. (2014). It combines the forget and input gates into a single “update gate.” It also merges the cell state and hidden state, and makes some other changes. The resulting model is simpler than standard LSTM models, and has been growing increasingly popular[2].