Learning Time Sequences

---

There are many tasks that require learning a temporal sequence of events. These problems can be broken into 3 distinct types of tasks:

• Sequence Recognition: Produce a particular output pattern when a specific input sequence is seen. Applications: speech recognition
• Sequence Reproduction: Generate the rest of a sequence when the network sees only part of the sequence. Applications: Time series prediction (stock market, sun spots, etc)
• Temporal Association: Produce a particular output sequence in response to a specific input sequence. Applications: speech generation

Some of the methods that are used include

• Tapped Delay Lines (time delay networks)
• Context Units (e.g. Elman Nets, Jordan Nets)
• Back propagation through time (BPTT)
• Real Time Recurrent Learning (RTRL)

Tapped Delay Lines / Time Delay Neural Networks

---

One of the simplest ways of performing sequence recognition because conventional backpropagation algorithms can be used.

Downsides: Memory is limited by length of tapped delay line. If a large number of input units are needed then computation can be slow and many examples are needed.

A simple extension to this is to allow non-uniform sampling:

where w_i is the integer delay assoicated with component i. Thus if there are n input units, the memory is not limited simply the previous n timesteps.

Another extension that deals is for each "input" to really be a convolution of the original input sequence.

In the case of the delay line memories:

Other variations for c are shown graphically below:

This figure is taken from "Neural Net Architectures for Temporal Sequence Processing", by Mike Moser.

[Top] [Next: Recurrent Networks I] [Back to the first page]