CDR

Papers and Discussions

I thought I'd use this page to put pen to paper and share the ideas for others to try or even just to clear up my own thoughts on subjects before I tackle the problem.

For those that are interested in going into further detail I've added some links to papers and reports on this page. Some of it is technical but I've still found that they have opened my mind to some of things that can be done with the norn brain structure.

Motivation

Note 8/10/1999: Some of the ideas in this discussion have been implemented in the Canny norns for Creatures 2.

The first modification idea came from a paper I came across titled Motivation. In it you'll find an interesting overview of the Drive Reduction Theory. An excerpt:

According to Hull, events that threaten survival give rise to internal drive states, and behaviors that act to reduce drive are thus rewarding. For instance, lack of food causes an increase in the hunger drive, and the consumption of food is rewarding because it leads to a reduction in the hunger drive.

[...]

According to Hull, at any given time the behavior with the greatest potential to reduce a given drive is released. If the drive persists, that behavior is inhibited, and the second strongest response in the drive hierarchy will be released, and so on.

Sound a little familiar? The document is available (along with a lot of others) at the Boston University Neurobotics Lab. The original appears to be an except from the book Handbook of Brain Theory and Neural Networks. The original document in postscript format is available in the publications page of the Neurobotics lab. I've converted this paper titled Motivation into Adobe PDF format which you can read if you have Adobe Acrobat.

Attention Lobe

Reading this paper gave me an idea for a brain modification to the standard norn that I thought someone might like to try. First some background on how the attention seeking mechanism in a norn works.

Currently the attention seeking aspect of a norn is controlled by the Attention lobe. Whatever cell is firing the strongest in this lobe indicates the object that the norn is currently paying attention to. It is this object that a norns actions will be directed against.

The Attention lobe has 40 cells. One for each noun object available in Creatures. The activation levels of each cell in the lobe comes from the dendrite connections from two source lobes. The first input source is the Stim Source lobe (connected via the class 0 dendrites). There is a one-to-one mapping between each cell in the Attention lobe and each cell in the Stim Source lobe. The second input source is the Noun lobe (connected via the class 1 dendrites). There is also a one-to-one mapping between each cell in the Attention lobe and each cell in the Noun lobe.

The Stim Source lobe cells get activated when the norn notices some object within the norns sensory range (line of vision, touch, etc). The higher the output value the more urgent the stimulation. So a moving toy right next to the norn may activate at a higher level than a piece of food sitting further away not doing anything. The Stim Source lobe is a winner-takes-all lobe. This means that the highest firing cell is the only firing cell. In this example the toy will be the output of the lobe. This fires along the dendrite connection to the same cell in the Attention lobe. In this example the Attention lobe 'toy' cell will be firing at the same level as the Stim Source 'toy' cell.

The Noun lobe cells get activated when the norn hears a noun spoken. I'm not sure if other norns speaking nouns activate it but definitely nouns typed at the keyboard. So if I were to type 'plant' and the norn knew the word 'plant' to be related to object 'plant' then the 'plant' cell of the Noun lobe would fire. Once again this travels along the dendrite connection to the Attention lobe and fires the 'plant' cell of that lobe.

The Attention lobe is also a winner-takes-all lobe. It will see the firing of the 'toy' and the 'plant' cells. The cell firing with the highest value (say the 'toy') will be the object that the norn pays attention to. If the norn then decides to 'Push' it will be pushing the toy. Note that the two input source lobes are added together so if the user types 'food' it will be added to the 'food' object Stim Source value and the total result used to work out the winner of the Attention lobe.

Given that nouns aren't sent to norns all that much the main input to the Attention lobe is the Stim Source lobe. Remember that the level of activation (ie. the value of the output on each cell) of the Stim Source cells depends upon how active that object is or how close the norn is to it. The norn could be super hungry and about to die of starvation but if it all it can see is that moving toy or that sexy female norn nearby it will never be able to push that piece of food lying just a bit further away. Can we change this in some manner?

I think so. It would be good if we could have the drive levels of the norn somehow adjust the activation levels of the Attention lobe based upon experience that the norn gains from performing actions on objects. For example, a norn learns that food lowers hunger drive. Hunger drive is very high, there is food nearby but a sexy female is closer and moving around.

Usually the norn will pay attention to the female. But our 'adjusted' norn will think 'a sexy female is vying for my attention - but my sex drive is low so the overall activation effect of that is low. Ohh, look, food is nearby but not very interesting - but I'm starving therefore the food is much more interesting because I know it lowers hunger drive. I'll pay attention to the food'.

Effectively we want to modulate the output of each cell of the Attention lobe based on the drive levels of the norn where those drive levels can be affected by the object referenced by that cell. And we want the mapping between object and drive to be learned by the norn - not hard coded. So the norn learns that food lowers hunger therefore food will be more interesting when I'm hungry. a Norn learns that Norns can lower their sex drive therefore norns are more interesting when their sex drive is high. Does that make sense? I think this may even solve the problem where norns are so interesting to other norns that they forget to eat.

Given the genetic brain structure of the norn is the above possible to do? I believe it is. As part of the tutorials I hope to demonstrate how this could be done. But in the meantime I leave the following ramblings for those that would like to plow ahead and give it a go...

Possible Implementation

Remember that the following is speculation so may not work. Any input on ideas or people who have done/tried similar things would be greatly appreciated.

We want to modulate the Attention lobe cell values by values in the Drive lobe. There are 40 cells in the Attention lobe and 16 possible drive values. We want the relationship between what drive (to keep the discussion simple we will only allow for one drive) modulates a given Attention lobe cell to be learnt by the norn. So to start with the norn may think that if it is hungry a toy will be more interesting. As a norn learns that a toy does not lower hunger it will try some other object. Eventually it learns that food lowers hunger so food becomes more interesting. The mechanism to do this is dendrite migration.

In essence it is similar to the relationship between the decision lobe and the concept lobe. There a particular concept (or combination of perceptions) can result in up to sixteen decisions to be made. Each decision is given a weighting that is adjusted according to whether reward or punishment is given.

There are 40 Attention lobe cells and 16 drive lobes cells. We want each attention lobe cell to map to one drive lobe cell. This would be done with dendrites on the Attention lobe cell with the source lobe being the Drive lobe. Allowing only one drive to have an effect would mean using one dendrite. The wiring at birth would be random so we need a learning mechanism to control dendrite migration. Let's ignore that for now. We need to modulate the output of the attention lobe based upon the value of the input dendrite. If you followed Tutorial Two you know that we can probably do this through the Neurone state rule. Modulation is done with the TIMES opcode. We want to modulate by the value of the input dendrite. This is the type0 or type1 opcode depending on which dendrite class we used.

Unfortunately a lobe can only have two source lobes and the Attention lobe already has two! We can either get rid of one or create a brand new lobe. This new lobe would have 40 cells and connect to the Drive lobe as described above using class 0 dendrites. The class 1 dendrites would connect to the Stim Source lobe. The result of cells firing would be the Stim Source value modulated by the drive lobe value. This new lobe would replace the Stim Source lobe as an input in the Attention lobe. I would see the Neurone State Rule of this new lobe as being something like: 'state:PLUS:type1:TIMES:type0'. Does that make sense?

Now the only problem is the learning mechanism to enable the dendrites to migrate if the norn learns that the current drive mapped to the object is not affected by that object in any way. How to do this? Probably by copying the mechanism used in the Decision and Concept lobes. We know that drive reduction produces reward. If a norn's drive is high, it uses an object which lowers a drive, it will be rewarded. This will be detected by the dendrite link between our new lobe and the drive lobe. This will strengthen the connection. If a norn's drive is high, it uses an object which increases the drive further it will be punished, This will be detected by the dendrite link between our new lobe and the drive lobe. This will weaken the connection. When the connection strength hits zero the dendrite will migrate. This is my weakest point of understanding so a bit of experimentation will clean it up. If we can get the first part working and then concentrate on learning reinforcement later I will be happy.

Home