Re: [gnubg] Temporal difference learning. Lambda parameter.

[Timothy Y. Chow]

Philippe Michel wrote:

> The engine doesn't "plan ahead", does it ? It approximates the 
> probabilities of the game outcomes from the current position (or we can 
> say its equity for simplification).
>
> My understanding is that its potential accuracy depends on the neural 
> network (architecture + input features) and the training method 
> (including the training database in the case of supervised learning) has 
> influence on how close to this potential one can go, and how fast.

I haven't done any actual training of backgammon nets, but I think what 
Oysetein was saying is that TD learning is a method of trying to figure 
out (crudely speaking) "where you made your mistake when you lost," and it 
works well when you don't have to "backtrack too far" when you're 
readjusting your weights.  But for positions where there's "long-term 
planning" (e.g., rolling the prime around the board), one intuitively 
expects TD learning not to work so well.

It's true that once you have a reasonably good network, you can 
"fine-tune" it using other methods.  For example, for a perfect bot, 
0-ply, 1-ply, 2-ply, etc., should all give the same answer, but an actual 
bot won't, so you can get some improvement just by forcing the bot to iron 
out these inconsistencies.  This can be done using various supervised 
training methods and not necessarily TD learning.  But my understanding 
(which could be flawed) is that TD learning still enters the picture at 
the very first step, when you're starting from scratch (with only the 
rules and no heuristics).

If there's some area of the game where your network is still doing very 
poorly, then you may need to do more "from scratch" training, rather than 
just bootstrapping off what you already have.  I think this is why Oystein 
is suggesting revisiting TD learning.

Tim