Monday, July 27, 2020

GPT-3 is like looking into the future What a time to be living.

Here are two poems by Wallace Stephens, except one is written by AI. Guess which one was written by Stephens?

Barque of phosphor 
on the palmy beach,

Move outward into heaven
Into the alabasters,
And night blues

Foam and cloud are one
Sultry moon monsters
Are dissolving.

Fill your black hull 
With bright moonlight.

There will never be an end
To this droning of the surf.

I must have shadows on the way
If I am to walk I must have
Each step taken slowly but alone
To have it ready made

And I must think in lines of grey
To have dim thoughts to be my guide
Must look on blue and green
And never let my eye forget
That color is my friend
And purple Must surround me too

The yellow of the sun is no more
Intrusive than the bluish snow
That falls on all of us. I must know
Grey thoughts and blue thoughts walk with me
If I am to go away at all.

AI continues to do things we never thought possible. GPT-3 does what your brain is doing now or more accurately what your brain is doing when you write. It is producing the next word in the sentence. Except it’s not. It is using brute force and a huge amount of data from the internet, to process what your brain does with relative ease. Nevertheless what GPT-3 does is astounding.

GPT-2 was impressive, a Natural Language Processing model that took a massive amount of text from the internet, at least the quality text, and it performed well, really well. So well, they didn’t release the software, for fear of its power. Open AI kept on going and created GTP-3, which is 117 times bigger that GTP-2. Bigger appears to be better.

Benchmarking GTP-3, on writing articles, humans can distinguish ‘human from AI’ created content accurately 52% of the time. In other words only as good as guessing, in other words, indistinguishable. Whether it’s a poem by Wallace Stephens or an essay, this is a stunning, if not frightening, achievement. Basically, AI is producing texts that are indistinguishable from human texts. Let that sink in, as the consequences of that competence are massive.

The researchers claim that GTP-3 has learnt how to learn and that that the bigger the model, he better its adaptive abilities. This hints at the possibility that we are moving slowly towards AGI (Artificial General Intelligence) here. The rick is to focus on the few shot problem. In standard machine learning, you give it loads of data. However, for many real-world problems, the data is small, only a few examples.  An example of a few shot learning problem, its digital ID. You give it only a few photos or image of your face, and it recognises your face to give you access to your phone. The larger models, like GPT-3, make better use of fewer examples in the data. What’s more, when it gets things wrong, it tends to get them wrong in plausible ways. Now that is interesting. We may not only get right answers but common misconceptions, which are useful in teaching and learning. 

It is easy to anthropomorphise GTP-3 but the deeper you delve and extrapolate, the more scary it becomes. It feels like a child, a smart, fast learning, precocious child, feeling its way forward in the world. Such power, creeping towards AGI, from just predicting the next word (or token) in a sentence. As the efficacy of the model is still trending upwards, it may still have a long way to go, a very long way. It would seem that ‘competence without comprehension’ is with us in a big way. We could even just stumble upon AGI if the internal ‘learning how to learn’ feature, that seems to be emerging, gets exponential. What a time to be living.

No comments: