This article helped me breathe a bit easier, especially taking a step back and thinking how long diffusion will likely take. I have this (dramatic) schema in my mind that it'll happen overnight and we will all be found with our pants down (metaphorically).
Thanks again for reorienting my understanding of AGI to something more realistic (you’ve done this in previous essays too).
One thing you don’t mention is the cost of training and running AI. If we reach some arbitrary metric of AGI, but the cost to interact with it is prohibitively high, does it really matter that we’ve gotten it?
Another thought I had based on your article is that even if progress on LLM base models stall (due to price or theoretical limitations), we will continue to see their use (and utility) increase for a long time as companies and people retool and retrain to learn how to use them properly.
"For example, it will be superhuman at playing chess, despite the fact that large language models themselves are at best mediocre at chess. Remember that the model can use tools, search the internet, and download and run code. If the task is to play chess, it will download and run a chess engine."
Would we not consider this cheating? A human can use a chess computer, too, but we don't consider the person smarter for it. Shouldn't we apply the same mental model to AI?
That's exactly our view! (In retrospect we should have been more explicit about it.)
In that section, we are taking AGI definitions at face value in order to critique those definitions, as reflected in the title of the section "It isn’t crazy to think that o3 is AGI, but this says more about AGI than o3".
In a later section, we point out that economic impact will take a while because humans are a moving target, and this is one reason why.
And in AI as Normal Technology, one of our central points is that "superhuman" and "superintelligence" don't mean what they're usually implied to mean because the right comparison is never human vs AI but rather human + AI vs AI.
I don't think you're right about the labs being incentivized to declare AGI achieved. They're incentivized to keep insisting it's coming any day now because that keeps the VC cash flowing in. But declaring it here means that all dries up. Any lab could claim they've released it whenever. They could do it today. But they won't because the investments pour in for the promises.
* If a startup makes this claim to get attention, the big players might be forced to follow suit to avoid looking like they've fallen behind (which we mention in the essay).
* If they declare AGI achieved, they could arguably raise even more cash than they're doing now if they claim that they are going to automate the whole economy and all they need to do is to build out the infrastructure for inference fast enough.
* Another way it could help them raise even more money is if they say that having achieved AGI puts them in the perfect place in the race to ASI.
I wonder what happens if a large group of otherwise independent AI/ML researchers band together and inject some "chaos" into the system: Write an op-ed or open letter or whatever with a large bunch of "aligned" scholars (see what I did there) and preemptively declare that AGI is here (say that o3-high or whatever) is it. That would be an interesting shock to the system! How would that change the dynamic between MS and ClosedAI? Of course it would potentially harm the reputations of said scholars ... but of course one can always just point and say "well would you like to prove us wrong?" Probably unethical, but I wonder if any of the tech executives would know any empathy or ethics if it came up behind them ...
Great read ! Too much hype on AGI because it’s driving investment; but with no clear definition AND purpose it feels more like a narrative than a true goal. Importantly I am quite convinced LLMs alone in their current forms CANNOT help achieve “human like intelligence” because they don’t have multi sensorial cognitive capabilities. LLMs are amazing at synthesising knowledge at hyper scale. But that’s all they are.
Such good stuff: Love the substack! This whole AGI vs. ASI thing is exhausting! For the "rationalists" and AI enthusiasts that claim that Superintelligence (TM)(R)(C) is just "out performing all humans at all tasks". To me this seems like total nonsense! A comparison is alwaysbetween humans and say other primates (as though evolution and intelligence is some neat continuum) ... but the issue is that if we are by definition more intelligent than other life then, and the metric is "cognitive tasks" that we outperform other animals. What metrics are those exactly?
A single spider can navigate more "varied" and perform more complex tasks than we can on a given day. It wouldn't make sense to give a squid a verbal analogies test. What determines which one of those tasks is more difficult than the other? Is there a handy scale for the "Algorithmic Complexity" of any arbitrary task (apparently it is famously difficult to prove that an arbitrary task is more complicated than another in a "universal way")?
And besides: would we truly be even able to communicate with a truly "super intelligence"!? what use would it have of language? Or abstractions/concepts we use to "hide" complexity of mental objects to make sense of the world that is vastly more complex than the artificial environments of our RL trained "agents" with limited "symbolic" processing capabilities! Would we even be able to differentiate their actions from randomness!? I think Cosma Shalizi's and Alison Gopnik's idea about LLMs being "cultural" technologies is more coherent than what the AI doomer cult's ideas!
As an example about "rate limitations/choke points" to diffusion of tech. I work for a large company people have heard of (just a regular boring ass non tech company) that is very profitable and has a presence in every continent.The company JUST MOVED from a paper based "approval" system ... to a "digital" one a few years ago! But the digitization was just Pdf versions of forms, "coversheets" and "drawings" that are scanned in ... so at best I use an approved LLM (a lobotomized o3-mini and claude 3.5) to translate and write emails etc etc. But the whole thing could have been designed with the technology of the 2000s to obviate the need for me to manually sign documents in the first place! Our "General Purpose Technology" is Excel: And I don't mean for number crunching or pivot tables ... but as a drawing tool, layout tool, presentation tool, communication tool ...
It's not like I need to solve canned Algebraic Topology or AIME/Frontier Math problems in my day to day! I bet few people do! In the real world, you won't be given a well crafted prompt of a math problem ... valuable knowledge is "diffuse" and "tacit" and most of the time it's about how you can use that knowledge to figure out what even needs to be solved.
Thanks for making the argument in plain & simple language - I'm convinced. The one place you did not cover (& I know you said you would get back to this later) is military uses of AI. Both diffusion & potential harmful use (including in the cyber domain) appear to me to be riskier than all the other domains where your arguments make sense.
One other quibble I have is on the speed of diffusion in China - from what anyone can observe with payment systems, e commerce, EV's..the Chinese are much quicker to innovate & adopt than anyone else. Your argument here does not bake all this evidence.
And more to the point- the models are still bad. Just so bad. I don't know what sorts of experiences other people are having that are enabling these evangelical awakenings, but every time I check back in with an LLM to see if there's some actual utility there, something dumb happens. I don't think I have ever had an LLM do a single task that was not sufficiently gnarled up in some way that it was an open question whether I actually saved any time or effort. Which is not to say it is not all a very impressive magic trick! But how are the 'the AI god is nigh' people not noticing that these models are worse at algebra than ten dollar calculators? They can seem very smart and answer last year's bar exam questions but brutally fail this year's- gosh, it's almost like this is mostly search. Which isn't nothing- but it's also not transformative, especially when the value of search is at least gesturing towards provenance, attribution, and alternative perspectives.
Kurzweil goes on to discuss the notion of a second threshold, after the Turing test, that will need to be crossed in order for machines to gain the ability to self-improve:
"Edward Feigenbaum proposes a variation of the Turing test, which assesses not a machine's ability to pass for human in casual, everyday dialogue but its ability to pass for a scientific expert in a specific field. The Feigenbaum test (FT) may be more significant than the Turing test because FT-capable machines, being technically proficient, will be capable of improving their own designs."
But then also notes:
"The entire history of AI reveals that machines started with the skills of professionals and only gradually moved toward the language skills of a child."
and, even better:
"Reasoning in many technical fields is not necessarily more difficult than the common sense reasoning engaged in by most human adults." (<--- THIS)
Finally:
"I would expect that machines will pass the FT, at least in some disciplines, around the same time as they pass the Turing test. Passing the FT in all disciplines is likely to take longer, however. This is why I see the 2030s as a period of consolidation, as machine intelligence rapidly expands its skills and incorporates the vast knowledge bases of our biological human and machine civilization."
The real question is whether a "super intelligent" machine would solve any math or scientific problems we have in the first place? Humans use "science" and "math" as abstractions to make sense of a world we cannot understand with our individual faculties alone. For the vast majority of human history, we didn't even have written language ... the alphabet is maybe a few thousand years old? Math and science were recent developments that we seem to use to "abstract" and "predict" patterns in the world in ways that may be unique to humans/primate cognition. Why would an arbitrary "intelligent" entity have use for such things? Wouldn't it be able to intuitively predict and manipulate matter etc at scales and ways we wouldn't even understand? Also, we greatly augmented our cognition by working socially in complex ways to invent tools such as computer or slide rules ... will a "superintelligence" be social? Will it form hierarchies in response to scarce resources bound by vague notions of kith and kin to work towards staving off starvation?
Great article. The discussion about AGI being a fuzzy frontier reminds me of a passage in Kurzweil's "The Singularity is Near". He writes:
"One of the many skills that nonbiological intelligence will achieve ... is sufficient mastery of language and shared human knowledge to pass the Turing test. The Turing test is important not so much for its practical significance but rather because it will demarcate a crucial threshold."
and follows up with:
"Because the definition of the Turing test will vary from person to person, Turing test-capable machines will not arrive on a single day, and there will be a period during which we will hear claims that machines have passed the threshold. Invariably, these early claims will be debunked by knowledgeable observers, probably including myself. By the time there is a broad consensus that the Turing test has been passed, the actual threshold will have long since been achieved."
This article helped me breathe a bit easier, especially taking a step back and thinking how long diffusion will likely take. I have this (dramatic) schema in my mind that it'll happen overnight and we will all be found with our pants down (metaphorically).
Amazingly lucid and pouring cold water on hype and fear.
AGI will be just one ingredient in the mix. The rate of change is, as always, dependent on the slowest component.
What if the slowest component isn’t slow any more: https://open.substack.com/pub/charleseldering/p/how-long-will-it-take-for-productivity?r=24o7sp&utm_medium=ios
Kurzweil predicted self driving cars by 2009. We will be lucky to have them ubiquous by 2030.
Things take time. Always the case.
Time will tell ;)
One of my favorite posts of the year so far, thanks.
This and the previous article have led me to regard your Substack as the sanest perspective on AI out there. Tremendously refreshing!
Thanks again for reorienting my understanding of AGI to something more realistic (you’ve done this in previous essays too).
One thing you don’t mention is the cost of training and running AI. If we reach some arbitrary metric of AGI, but the cost to interact with it is prohibitively high, does it really matter that we’ve gotten it?
Another thought I had based on your article is that even if progress on LLM base models stall (due to price or theoretical limitations), we will continue to see their use (and utility) increase for a long time as companies and people retool and retrain to learn how to use them properly.
Fantastic essay. Many thanks for providing it without a paywall.
"For example, it will be superhuman at playing chess, despite the fact that large language models themselves are at best mediocre at chess. Remember that the model can use tools, search the internet, and download and run code. If the task is to play chess, it will download and run a chess engine."
Would we not consider this cheating? A human can use a chess computer, too, but we don't consider the person smarter for it. Shouldn't we apply the same mental model to AI?
That's exactly our view! (In retrospect we should have been more explicit about it.)
In that section, we are taking AGI definitions at face value in order to critique those definitions, as reflected in the title of the section "It isn’t crazy to think that o3 is AGI, but this says more about AGI than o3".
In a later section, we point out that economic impact will take a while because humans are a moving target, and this is one reason why.
And in AI as Normal Technology, one of our central points is that "superhuman" and "superintelligence" don't mean what they're usually implied to mean because the right comparison is never human vs AI but rather human + AI vs AI.
This point applies with equal force to applications of generative AI in education, too.
Excellent article, and many compelling arguments about the rate of diffusion. However, what if the rate of adoption/change is a fraction of what we have experienced due to the exponential growth of technology and the shortening of time between breakthroughs? I explore that in my post: https://open.substack.com/pub/charleseldering/p/how-long-will-it-take-for-productivity?r=24o7sp&utm_medium=ios
In other words, could “true” AGI (whatever that means) be right around the corner?
I don't think you're right about the labs being incentivized to declare AGI achieved. They're incentivized to keep insisting it's coming any day now because that keeps the VC cash flowing in. But declaring it here means that all dries up. Any lab could claim they've released it whenever. They could do it today. But they won't because the investments pour in for the promises.
That's possible! But some counterarguments:
* If a startup makes this claim to get attention, the big players might be forced to follow suit to avoid looking like they've fallen behind (which we mention in the essay).
* If they declare AGI achieved, they could arguably raise even more cash than they're doing now if they claim that they are going to automate the whole economy and all they need to do is to build out the infrastructure for inference fast enough.
* Another way it could help them raise even more money is if they say that having achieved AGI puts them in the perfect place in the race to ASI.
I wonder what happens if a large group of otherwise independent AI/ML researchers band together and inject some "chaos" into the system: Write an op-ed or open letter or whatever with a large bunch of "aligned" scholars (see what I did there) and preemptively declare that AGI is here (say that o3-high or whatever) is it. That would be an interesting shock to the system! How would that change the dynamic between MS and ClosedAI? Of course it would potentially harm the reputations of said scholars ... but of course one can always just point and say "well would you like to prove us wrong?" Probably unethical, but I wonder if any of the tech executives would know any empathy or ethics if it came up behind them ...
I bought your book (twice now; I have the first copy to someone).
Are there any other books or blogs on AI (or any other issues) that you’ve read and would recommend?
Great read ! Too much hype on AGI because it’s driving investment; but with no clear definition AND purpose it feels more like a narrative than a true goal. Importantly I am quite convinced LLMs alone in their current forms CANNOT help achieve “human like intelligence” because they don’t have multi sensorial cognitive capabilities. LLMs are amazing at synthesising knowledge at hyper scale. But that’s all they are.
I write some perspectives here: https://www.koncentrik.co/p/the-illusion-of-intelligence-why
Such good stuff: Love the substack! This whole AGI vs. ASI thing is exhausting! For the "rationalists" and AI enthusiasts that claim that Superintelligence (TM)(R)(C) is just "out performing all humans at all tasks". To me this seems like total nonsense! A comparison is alwaysbetween humans and say other primates (as though evolution and intelligence is some neat continuum) ... but the issue is that if we are by definition more intelligent than other life then, and the metric is "cognitive tasks" that we outperform other animals. What metrics are those exactly?
A single spider can navigate more "varied" and perform more complex tasks than we can on a given day. It wouldn't make sense to give a squid a verbal analogies test. What determines which one of those tasks is more difficult than the other? Is there a handy scale for the "Algorithmic Complexity" of any arbitrary task (apparently it is famously difficult to prove that an arbitrary task is more complicated than another in a "universal way")?
And besides: would we truly be even able to communicate with a truly "super intelligence"!? what use would it have of language? Or abstractions/concepts we use to "hide" complexity of mental objects to make sense of the world that is vastly more complex than the artificial environments of our RL trained "agents" with limited "symbolic" processing capabilities! Would we even be able to differentiate their actions from randomness!? I think Cosma Shalizi's and Alison Gopnik's idea about LLMs being "cultural" technologies is more coherent than what the AI doomer cult's ideas!
As an example about "rate limitations/choke points" to diffusion of tech. I work for a large company people have heard of (just a regular boring ass non tech company) that is very profitable and has a presence in every continent.The company JUST MOVED from a paper based "approval" system ... to a "digital" one a few years ago! But the digitization was just Pdf versions of forms, "coversheets" and "drawings" that are scanned in ... so at best I use an approved LLM (a lobotomized o3-mini and claude 3.5) to translate and write emails etc etc. But the whole thing could have been designed with the technology of the 2000s to obviate the need for me to manually sign documents in the first place! Our "General Purpose Technology" is Excel: And I don't mean for number crunching or pivot tables ... but as a drawing tool, layout tool, presentation tool, communication tool ...
It's not like I need to solve canned Algebraic Topology or AIME/Frontier Math problems in my day to day! I bet few people do! In the real world, you won't be given a well crafted prompt of a math problem ... valuable knowledge is "diffuse" and "tacit" and most of the time it's about how you can use that knowledge to figure out what even needs to be solved.
Thanks for making the argument in plain & simple language - I'm convinced. The one place you did not cover (& I know you said you would get back to this later) is military uses of AI. Both diffusion & potential harmful use (including in the cyber domain) appear to me to be riskier than all the other domains where your arguments make sense.
One other quibble I have is on the speed of diffusion in China - from what anyone can observe with payment systems, e commerce, EV's..the Chinese are much quicker to innovate & adopt than anyone else. Your argument here does not bake all this evidence.
And more to the point- the models are still bad. Just so bad. I don't know what sorts of experiences other people are having that are enabling these evangelical awakenings, but every time I check back in with an LLM to see if there's some actual utility there, something dumb happens. I don't think I have ever had an LLM do a single task that was not sufficiently gnarled up in some way that it was an open question whether I actually saved any time or effort. Which is not to say it is not all a very impressive magic trick! But how are the 'the AI god is nigh' people not noticing that these models are worse at algebra than ten dollar calculators? They can seem very smart and answer last year's bar exam questions but brutally fail this year's- gosh, it's almost like this is mostly search. Which isn't nothing- but it's also not transformative, especially when the value of search is at least gesturing towards provenance, attribution, and alternative perspectives.
Kurzweil goes on to discuss the notion of a second threshold, after the Turing test, that will need to be crossed in order for machines to gain the ability to self-improve:
"Edward Feigenbaum proposes a variation of the Turing test, which assesses not a machine's ability to pass for human in casual, everyday dialogue but its ability to pass for a scientific expert in a specific field. The Feigenbaum test (FT) may be more significant than the Turing test because FT-capable machines, being technically proficient, will be capable of improving their own designs."
But then also notes:
"The entire history of AI reveals that machines started with the skills of professionals and only gradually moved toward the language skills of a child."
and, even better:
"Reasoning in many technical fields is not necessarily more difficult than the common sense reasoning engaged in by most human adults." (<--- THIS)
Finally:
"I would expect that machines will pass the FT, at least in some disciplines, around the same time as they pass the Turing test. Passing the FT in all disciplines is likely to take longer, however. This is why I see the 2030s as a period of consolidation, as machine intelligence rapidly expands its skills and incorporates the vast knowledge bases of our biological human and machine civilization."
The real question is whether a "super intelligent" machine would solve any math or scientific problems we have in the first place? Humans use "science" and "math" as abstractions to make sense of a world we cannot understand with our individual faculties alone. For the vast majority of human history, we didn't even have written language ... the alphabet is maybe a few thousand years old? Math and science were recent developments that we seem to use to "abstract" and "predict" patterns in the world in ways that may be unique to humans/primate cognition. Why would an arbitrary "intelligent" entity have use for such things? Wouldn't it be able to intuitively predict and manipulate matter etc at scales and ways we wouldn't even understand? Also, we greatly augmented our cognition by working socially in complex ways to invent tools such as computer or slide rules ... will a "superintelligence" be social? Will it form hierarchies in response to scarce resources bound by vague notions of kith and kin to work towards staving off starvation?
Great article. The discussion about AGI being a fuzzy frontier reminds me of a passage in Kurzweil's "The Singularity is Near". He writes:
"One of the many skills that nonbiological intelligence will achieve ... is sufficient mastery of language and shared human knowledge to pass the Turing test. The Turing test is important not so much for its practical significance but rather because it will demarcate a crucial threshold."
and follows up with:
"Because the definition of the Turing test will vary from person to person, Turing test-capable machines will not arrive on a single day, and there will be a period during which we will hear claims that machines have passed the threshold. Invariably, these early claims will be debunked by knowledgeable observers, probably including myself. By the time there is a broad consensus that the Turing test has been passed, the actual threshold will have long since been achieved."