In the short story Funes the Memorious, author Jorge Luis Borges creates a unique character who has a perfect memory. He can tell time accurate to the exact minute without the help of a watch. He can recall the exact shape of each cloud in the sky and the ripples in the pond. He can recite any paragraph from any page of any book he has read. But as Borges describes, he is "hopelessly limited in his possiilities", his absurd projects reveal "a certain stammering greatness" and he is incapable of Platonic ideas, of generalities, of abstractions; his world is one of intolerably uncountable details.
This story was published in June 1942. But the protagoninst resembles a modern-day computer in more ways than one.
Deep Learning has been used to do things which one could never expect from a computer half a century ago. Everyday, deep learning achieves a new feat. But what is deep learning? If I were allowed to commit the sin of trivialising a concept instead than simplifying it, this is how I would describe it: It's just a huge algorithm which takes in a lot of numbers, churns the numbers through its 'layers' and gives some output while optimizing a function that we (humans) provide it. Seen this way, it's not intelligent, is it? Add to this, the fact that there's no guarantee that its output is always correct.
Don't get me wrong here! This article isn't intended to bash and lash deep learning. In fact, I appreciate the achievements and contributions of the deep learning community and ocassionally get fascinated by the results they generate. I also consider myself fortunate to understand, in depth, some of the work done in deep learning. The point I want to hammer home is that this is not 'intelligence'. Deep Learning uses thousands of annotated images to learn the difference between apples and oranges, whereas a 3-year old kid learns that by seeing probably just one or two of each fruit. I think there is something very basic that we (deep learning community) are not doing right.
Let's first take a look at how Deep Learning originated. It started with the introduction of a 'perceptron', a.k.a. the artificial neuron, the building block of a neural network. If you, like me, have attended any introductory deep learning courses, the instructors usually make it a point to compare the perceptron to the biological neuron. They also point out that the biological neuron is so much more complex than the artificial one. (I hope this doesn't inspire any students to make the neural networks any more complex!!) After listening to this comparison, people start thinking that with some modifications neural networks might behaving like our brains, at least for some tasks. But that's not going to happen unless we fix an important issue. This issue has nothing to do with how fast the perceptron processes data or even what it does with it. It has to do with how data is stored, in general, in computers.
There's very little research on how our brain stores information, and unfortunately none of that has been applied to artificial intelligence or computers in general (AFAIK). Computers have perfect memory. When a computer sees an image of an “apple”, it stores a 2D representation accurate to each pixel. We never store pixel-level details in our brains. We simply store an abstract shape and colour. This is why we can learn that something is an “apple” just after seeing 2-3 images of apples, but deep learning algorithms need thousands of images.
I don't know if Borges, in 1942, intended to draw a parallel between his character and computers, but his story definitely motivated me to think about how computer's perfect memory is the reason why it computers can create abstractions, and hence can't think. Hopefully this article motivated you to think about this topic too. I don't know what researchers can do to incorporate abstract memory in computers, but this surely is food for thought!
By the way, I know that Variational AutoEncoders and Principal Component Analysis are few techniques that come close to abstractions, but they sort of store a compact representation of data. It's debatable if that can be called an abstraction.