Simple Competitive Learning

---

In competitive networks, output units compete for the right to respond.

Goal: method of clustering - divide the data into a number of clusters such that the inputs in the same cluster are in some sense similar.

A basic competitive learning network has one layer of input nodes and one layer of output nodes. Binary valued outputs are often (but not always) used. There are as many output nodes as there are classes.

Often (but not always) there are lateral inhibitory connections between the output nodes.(in simulations, the function of the lateral connections can be replaced with a different algorithm)

The output units are also often called grandmother cells. The term grandmother cell comes from discussions as to whether your brain might contain cells that fire only when you encounter your maternal grandmother, or whether such higher level concepts are more distributed.

Vector Quantization (VQ)

Vector quantization is one example of competitive learning.

The goal here is to have the network "discover" structure in the data by finding how the data is clustered. The results can be used for data encoding and compression. One such method for doing this is called vector quantization.

In vector quantization, we assume there is a codebook which is defined by a set of M prototype vectors. (M is chosen by the user and the initial prototype vectors are chosen arbitrarily).

An input belongs to cluster i if i is the index of the closest prototype (closest in the sense of the normal euclidean distance). This has the effect of dividing up the input space into a Voronoi tesselation

.

Implementing Vector Quantization with a Network

Algorithm:

• Choose the number of clusters M
• Initialize the prototypes w_*1,... w_*m (one simple method for doing this is to randomly choose M vectors from the input data)
• Repeat until stopping criterion is satisfied:
  • Randomly pick an input x
  • Determine the "winning" node k by finding the prototype vector that satisfies
    
    | w_*k - x | <= | w_*i - x | ( for all i )
    
    note: if the prototypes are normalized, this is equivalent to maximizing w_*i x
  • Update only the winning prototype weights according to
    
    w_*k(new) = w_*k(old) + m( x - w_*k(old) )
    
    This is called the standard competitive learning rule

    See Maple Example.

VQ and Data Compression

Vector quantization can be used for (lossy) data compression. If we are sending information over a phone line, we

• initially send the codebook vectors
• for each input, we send the index of the class that the input belongs

For a large amount of data, this can be a significant reduction. If M=64, then it takes only 6 bits to encode the index. If the data itself consists of floating point numbers (4 bytes) there is an 80% reduction ( 100*(1 - 6/32) ).