Re: Neural nets and spelling

[Asger K. Alstrup Nielsen]

> The mistake you are making here is in how many output nodes we would need.
> It would be silly to think that the neural net would deal with entire
> words in one chunk, rather we would scan through the words with several
> context units at either side of the letter we are looking at (above I
> suggest 3 on each side).  For each letter in the source word the neural
> net decides whether to use that letter, or a different letter, or no
> letter at all in the output soundalike.  This method only requires about
> 27 output nodes (one for each letter of the alphabet, and one to indicate
> no letter).

The discussion I presented had the aim of reaching this conclusion:
We would not be able to handle the entire word in one go with a neural
net.  So we agree on this.

> The dimensionality of the input and output space is not particularly
> relevant to a neural net's ability to generalise, I think that a few
> thousand examples should be plenty.

The ability to generalise in a sane way.  Consider the trivial example
of having two input nodes.  We train the network with lots of variation
on the first input node, but keep the second input node fixed.
Then we put the system into production, and some day somebody changes
the second input node.  The net will behave in an unpredictable way.

But I agree with you that we can get a useful subset of generalization
from a few thousand examples, provided we have a sane network topology.  
However, that's still a lot.

> This idea is close to what I have suggested, it is a shame you didn't
> persue this idea instead.  Also, I don't see why you say that the net
> isn't as general.

The net is trivially not as general, because with a window, we will
not be able to generalize beyond the window size.

Consider the example that I write "enternational" instead of "international".
If the window size is four letters to each side, the neural net will look
at "....enter", and won't notice that something is wrong.

> Anyway, I agree that this is worth trying out, I am somewhat busy at
> present but I will try to thrash out a test program in C to test the idea
> as soon as I get myself organised (Nobody hold their breath!).

First step is to collect a test body.  Then it's easier to see what obvious
limitations any given net configuration will have.

Greets,

Asger Alstrup