Re: Neural nets and spelling

[Ian Clarke]

> If we were to use a neural net to accomplish the above task in one go, we 
> would need a huge neural net.  Assuming a feed-forward net (the most common
> and best investigated), encoding each letter will require at least
> 1 input or at best as many inputs as there are letters.
> We need such an encoding for every letter in the word, and then we need the 
> same for the output, or arguably more because nobody says that the 
> replacement word has the same amount of letters as the input word.

This is not nescessarily true.  If we were to use a neural network to
translate full words into soundalikes in a manner similar to "Wordnet" we
would require 29 X 7 input nodes, say about 7 hidden units (not
nescessarily with full connectivity between the input nodes and the hidden
nodes) and about 26 output nodes.  This works out at roughly 2000
connections, not very many on a modern computer.
 
> In between, we need a hidden layer of nodes.  The amount of nodes needed is 
> difficult to determine in advance, and it depends on the number of input 
> nodes used, and the problem at hand.

True, I am using the "Wordnet" system as a starting place (about 7 hidden
nodes).  This system converted normal english into phonetically spelled
english and I am assuming that the soundalike conversion that we want to
do will be of similar difficulty.

> Ignoring all of these unknowns, let's speculate and have a look at how many 
> connections, we'll need in an optimistic setting:
> 
> Assuming that each letter is encoded using 5 inputs, and we have 26 letters.
> We want to restrict outselves to words of up to 10 letters.  Let's be opti-
> mistic and assume that we can settle for 10 hidden nodes.
> 
> 26*5*10 inputs connected to 10 hidden nodes that are connected 
> to 26*5*10 outputs.
> 
> That gives us 1300*10+10*1300 connections: A total of 26000 connections
> in an optimistic scenario.

The mistake you are making here is in how many output nodes we would need.
It would be silly to think that the neural net would deal with entire
words in one chunk, rather we would scan through the words with several
context units at either side of the letter we are looking at (above I
suggest 3 on each side).  For each letter in the source word the neural
net decides whether to use that letter, or a different letter, or no
letter at all in the output soundalike.  This method only requires about
27 output nodes (one for each letter of the alphabet, and one to indicate
no letter).

> In order to train the network, we have to collect a body of examples from
> which it can learn.  Given that we have such a huge space (it's a function
> from an 1300 dimensional space to another 1300 dimensional space), we need
> a huge body if we want to get predictable behaviour for just a minor fraction
> of the input space.

The dimensionality of the input and output space is not particularly
relevant to a neural net's ability to generalise, I think that a few
thousand examples should be plenty.

> Judging from this, I doubt a neural net is useful in this situation, 
> with the setup described above.

Probably true, which is why I had a different setup in mind.

> One can arguably design a different neural-net that solves a smaller task
> in the bigger task of suggesting spell-corrections.  That would probably
> be more feasible.  For instance, instead of feeding the entire word, one
> would feed a part of the word, and instead of getting a complete replacement
> word, we would only get one letter at the time.
> In this way, the neural net would reduce to something that we can handle,
> but the cost is that the net isn't as general.

This idea is close to what I have suggested, it is a shame you didn't
persue this idea instead.  Also, I don't see why you say that the net
isn't as general.

Anyway, I agree that this is worth trying out, I am somewhat busy at
present but I will try to thrash out a test program in C to test the idea
as soon as I get myself organised (Nobody hold their breath!).

Ian.

__________________________________________________________________________ 
Ian Clarke                                    http://www.dcs.ed.ac.uk/~iic 
      `An intellectual is someone who has been educated beyond their 
                              intelligence'