What neuroscience can teach us about neural networks?

Neural networks have the word "Neural" attached to it. And of course neural refers to Neurons we have on our brain. But how exactly does a neuron in Neural networks correspond to a neuron in our brain.

Why?

Yeah, why should I care what is the relation between both types of neurons? There's an important reason to it. Our brains have about 100 billion neurons and it takes about 20Watt of power to keep it running well. And we can do some seriously complex things using our brains.

Now, compare that to GPT-3, it has about 175 Billion parameters and takes 504000000 Watt to run! GPT3 is quite good, but it still fades when compared to the things that jumble of neurons in our skulls can do. This is why we need to learn from the way our brains work.

What to expect?

The way we will proceed is by comparing eye and visual cortex with neural networks that can recognize images. This way, we are comparing bananas with bananas.

How do they compare?

We'll have to start by understanding how brain understands visual inputs from our eyes. The part of the brain we are taking about is called Visual Cortex and it is responsible to understand light stimulation in our eyes.

Our eyes have 2 types of photo receptor cells, cones and rods. Rods are mostly light sensitive and cones color sensitive. There's a bit more to it, but this will do the job for us. So, each cell in eye (cones and rods) have one-to-one correspondence to a neuron in our brain.

if you stimulate 3019th rod on left eye, a neuron in your visual cortex will light up. If you stimulate 3020th rod, another neuron next to the previous one will light up. This is why, there is a one to one correspondence.

But, now what about something more complex, something like a line. if you stimulate 10 rods in a row in one direction, another part of the brain called Visual MT will light up. Visual MT is the part of the brain that is responsible for directional understanding. But can we keep continuing? Can we have one neuron for each and every aspect of visual perception.

Do we have a neuron in brain which will light up when you see a square at 30deg angle 2 feet away from you with side length of 10cm? Doesn't sound right? Yeah. we don't. There is no chance that we can encode for every single thing we see. Our brains simply don't have enough neurons.

But, never the less, this did not stop scientists to go our there looking for a Grandmother neuron. Basically, a neuron that encodes to see your grandmother when her face is tilted to her left. We did not quite find anything like that, but still there was a subtlety. Go and read about Jennifer Aniston neuron which is a neuron that fired when a chimp was shown a photo is Jennifer Aniston 🤯.

So, how does the brain work, actually? How does it know if you are looking at your grandmother or your mother? That's where layers come in. Layers of neuron that abstracts out further.

For example, if 20 neuron fires when they see a ring, then because of these 20, 5 others are simulated if they are equidistant. And so on, this will correspond to a circle in brain. So, there are no neurons encoding for a circle and a circle only, but a few that are layered.

How does a neural network work?

Exactly same in spirit, a neural network uses layred structure to find patterns and not by remembering each sample simply because there isnt enough memory to remember everything. Somethings need to be logically deduced to arrive at the answer.

If you want to see the complete inner workings of a neuron, head over to this video by 3B1B on the topic.