Astounding talk by Sutskever

A 30 minute talk given yesterday by Ilya Sutzkever already being seen as seminal.

We have gone through a decade of astonishing advancements in neural network learning. We were right to equate an artificial neuron with a human neuron. Biologically inspired AI has been hugely successful. It led to huge success through scaling. We forget that we were using crude string matching, tables and statistical techniques. We worked in this world and from 2014 were successfully producing real product. But we knew its limitations.

PEAK DATA
Then came GTP 2.0 and LLMs. These have been astonishingly successful. On speed alone they do what takes humans ages to complete, they are also often better on quality. But Sutskever’s big message was that pre-training will end due to data limitation, peak data, the fossil fuel of AI.

What do we do after pre-training? Agents, synthetic data and OpenAI o1 are examples of attempt to move forward. He draws from biology showing the relationship between body size of mammals and their brains. Hominids have an accelerated brain to body ratio. We are different. But so is AI, which can also be different. This is evolution on steroids. What took billions of years to achieve through evolution to hominids, is being done in just a few years.

SUPERINTELLIGENCE
Progress in AI has been remarkable but he thinks we are heading towards ‘superintelligence’. At the moment we have amazing systems which in many areas (evaluated) exceed human capabilities. They still get a little confused at times, who doesn’t. Hallucinations happen but models will eventually autocorrect. This may already be happening.

We have lots to gain from current AI, as it is so incredibly useful. Even if we stopped right now there is so much that can be done with the current technology to keep us busy for a decade. It is a productivity boost in may areas of human endeavour as well as being a discovery tool.

Systems are only very slightly agentic but agency, reasoning and self-awareness will happen – why? They are useful. The human brain, stopped growing in size but we got stronger cognitively, so humanity kept advancing. Similarly, the agents and tools on top of LLMs will fuel the progress through agentic behaviour. This future AI will possess unpredictable capabilities and self-awareness, transforming their capabilities.

FUTURE
He left us hanging and is very careful not to become too flippant and certain of himself, like some silly, keynote futurist. He has worked on the leading edge of this technology for decades and has contributed hugely to the field. While careful about predicting the exact nature of future development, which he thinks we cannot predict with certainty, he nonetheless, paints a picture of rapid progress and achievement.