This article helped me breathe a bit easier, especially taking a step back and thinking how long diffusion will likely take. I have this (dramatic) schema in my mind that it'll happen overnight and we will all be found with our pants down (metaphorically).
Thanks again for reorienting my understanding of AGI to something more realistic (you’ve done this in previous essays too).
One thing you don’t mention is the cost of training and running AI. If we reach some arbitrary metric of AGI, but the cost to interact with it is prohibitively high, does it really matter that we’ve gotten it?
Another thought I had based on your article is that even if progress on LLM base models stall (due to price or theoretical limitations), we will continue to see their use (and utility) increase for a long time as companies and people retool and retrain to learn how to use them properly.
"For example, it will be superhuman at playing chess, despite the fact that large language models themselves are at best mediocre at chess. Remember that the model can use tools, search the internet, and download and run code. If the task is to play chess, it will download and run a chess engine."
Would we not consider this cheating? A human can use a chess computer, too, but we don't consider the person smarter for it. Shouldn't we apply the same mental model to AI?
That's exactly our view! (In retrospect we should have been more explicit about it.)
In that section, we are taking AGI definitions at face value in order to critique those definitions, as reflected in the title of the section "It isn’t crazy to think that o3 is AGI, but this says more about AGI than o3".
In a later section, we point out that economic impact will take a while because humans are a moving target, and this is one reason why.
And in AI as Normal Technology, one of our central points is that "superhuman" and "superintelligence" don't mean what they're usually implied to mean because the right comparison is never human vs AI but rather human + AI vs AI.
I don't think you're right about the labs being incentivized to declare AGI achieved. They're incentivized to keep insisting it's coming any day now because that keeps the VC cash flowing in. But declaring it here means that all dries up. Any lab could claim they've released it whenever. They could do it today. But they won't because the investments pour in for the promises.
* If a startup makes this claim to get attention, the big players might be forced to follow suit to avoid looking like they've fallen behind (which we mention in the essay).
* If they declare AGI achieved, they could arguably raise even more cash than they're doing now if they claim that they are going to automate the whole economy and all they need to do is to build out the infrastructure for inference fast enough.
* Another way it could help them raise even more money is if they say that having achieved AGI puts them in the perfect place in the race to ASI.
I wonder what happens if a large group of otherwise independent AI/ML researchers band together and inject some "chaos" into the system: Write an op-ed or open letter or whatever with a large bunch of "aligned" scholars (see what I did there) and preemptively declare that AGI is here (say that o3-high or whatever) is it. That would be an interesting shock to the system! How would that change the dynamic between MS and ClosedAI? Of course it would potentially harm the reputations of said scholars ... but of course one can always just point and say "well would you like to prove us wrong?" Probably unethical, but I wonder if any of the tech executives would know any empathy or ethics if it came up behind them ...
Isn’t it logical to assume that if/when AGI is achieved, the company who invented it will control it and weild it themselves rather than handing out an "AGI subscription"? I don’t see anyone examining this angle of power/capability consolidation. In this scenario, no other CEO should be excited about AGI because it will only start a countdown to their own demise. They won't be able to share a slice of the action. For example, remember when Eric Schmidt talked about being able to say "AI: clone tiktok for me, make it better then deploy it." In simple terms then, couldn't OpenAI or whoever use AGI as normal software and just clone the top grossing apps and disrupt them all overnight? Why sell AGI as a product when you can use it to become King? Why do I feel like I'm the only one thinking this?
Great read ! Too much hype on AGI because it’s driving investment; but with no clear definition AND purpose it feels more like a narrative than a true goal. Importantly I am quite convinced LLMs alone in their current forms CANNOT help achieve “human like intelligence” because they don’t have multi sensorial cognitive capabilities. LLMs are amazing at synthesising knowledge at hyper scale. But that’s all they are.
Such good stuff: Love the substack! This whole AGI vs. ASI thing is exhausting! For the "rationalists" and AI enthusiasts that claim that Superintelligence (TM)(R)(C) is just "out performing all humans at all tasks". To me this seems like total nonsense! A comparison is alwaysbetween humans and say other primates (as though evolution and intelligence is some neat continuum) ... but the issue is that if we are by definition more intelligent than other life then, and the metric is "cognitive tasks" that we outperform other animals. What metrics are those exactly?
A single spider can navigate more "varied" and perform more complex tasks than we can on a given day. It wouldn't make sense to give a squid a verbal analogies test. What determines which one of those tasks is more difficult than the other? Is there a handy scale for the "Algorithmic Complexity" of any arbitrary task (apparently it is famously difficult to prove that an arbitrary task is more complicated than another in a "universal way")?
And besides: would we truly be even able to communicate with a truly "super intelligence"!? what use would it have of language? Or abstractions/concepts we use to "hide" complexity of mental objects to make sense of the world that is vastly more complex than the artificial environments of our RL trained "agents" with limited "symbolic" processing capabilities! Would we even be able to differentiate their actions from randomness!? I think Cosma Shalizi's and Alison Gopnik's idea about LLMs being "cultural" technologies is more coherent than what the AI doomer cult's ideas!
As an example about "rate limitations/choke points" to diffusion of tech. I work for a large company people have heard of (just a regular boring ass non tech company) that is very profitable and has a presence in every continent.The company JUST MOVED from a paper based "approval" system ... to a "digital" one a few years ago! But the digitization was just Pdf versions of forms, "coversheets" and "drawings" that are scanned in ... so at best I use an approved LLM (a lobotomized o3-mini and claude 3.5) to translate and write emails etc etc. But the whole thing could have been designed with the technology of the 2000s to obviate the need for me to manually sign documents in the first place! Our "General Purpose Technology" is Excel: And I don't mean for number crunching or pivot tables ... but as a drawing tool, layout tool, presentation tool, communication tool ...
It's not like I need to solve canned Algebraic Topology or AIME/Frontier Math problems in my day to day! I bet few people do! In the real world, you won't be given a well crafted prompt of a math problem ... valuable knowledge is "diffuse" and "tacit" and most of the time it's about how you can use that knowledge to figure out what even needs to be solved.
Thanks for making the argument in plain & simple language - I'm convinced. The one place you did not cover (& I know you said you would get back to this later) is military uses of AI. Both diffusion & potential harmful use (including in the cyber domain) appear to me to be riskier than all the other domains where your arguments make sense.
One other quibble I have is on the speed of diffusion in China - from what anyone can observe with payment systems, e commerce, EV's..the Chinese are much quicker to innovate & adopt than anyone else. Your argument here does not bake all this evidence.
And more to the point- the models are still bad. Just so bad. I don't know what sorts of experiences other people are having that are enabling these evangelical awakenings, but every time I check back in with an LLM to see if there's some actual utility there, something dumb happens. I don't think I have ever had an LLM do a single task that was not sufficiently gnarled up in some way that it was an open question whether I actually saved any time or effort. Which is not to say it is not all a very impressive magic trick! But how are the 'the AI god is nigh' people not noticing that these models are worse at algebra than ten dollar calculators? They can seem very smart and answer last year's bar exam questions but brutally fail this year's- gosh, it's almost like this is mostly search. Which isn't nothing- but it's also not transformative, especially when the value of search is at least gesturing towards provenance, attribution, and alternative perspectives.
This article helped me breathe a bit easier, especially taking a step back and thinking how long diffusion will likely take. I have this (dramatic) schema in my mind that it'll happen overnight and we will all be found with our pants down (metaphorically).
Amazingly lucid and pouring cold water on hype and fear.
AGI will be just one ingredient in the mix. The rate of change is, as always, dependent on the slowest component.
What if the slowest component isn’t slow any more: https://open.substack.com/pub/charleseldering/p/how-long-will-it-take-for-productivity?r=24o7sp&utm_medium=ios
Kurzweil predicted self driving cars by 2009. We will be lucky to have them ubiquous by 2030.
Things take time. Always the case.
Time will tell ;)
One of my favorite posts of the year so far, thanks.
This and the previous article have led me to regard your Substack as the sanest perspective on AI out there. Tremendously refreshing!
Thanks again for reorienting my understanding of AGI to something more realistic (you’ve done this in previous essays too).
One thing you don’t mention is the cost of training and running AI. If we reach some arbitrary metric of AGI, but the cost to interact with it is prohibitively high, does it really matter that we’ve gotten it?
Another thought I had based on your article is that even if progress on LLM base models stall (due to price or theoretical limitations), we will continue to see their use (and utility) increase for a long time as companies and people retool and retrain to learn how to use them properly.
Fantastic essay. Many thanks for providing it without a paywall.
Excellent, as always.
"For example, it will be superhuman at playing chess, despite the fact that large language models themselves are at best mediocre at chess. Remember that the model can use tools, search the internet, and download and run code. If the task is to play chess, it will download and run a chess engine."
Would we not consider this cheating? A human can use a chess computer, too, but we don't consider the person smarter for it. Shouldn't we apply the same mental model to AI?
That's exactly our view! (In retrospect we should have been more explicit about it.)
In that section, we are taking AGI definitions at face value in order to critique those definitions, as reflected in the title of the section "It isn’t crazy to think that o3 is AGI, but this says more about AGI than o3".
In a later section, we point out that economic impact will take a while because humans are a moving target, and this is one reason why.
And in AI as Normal Technology, one of our central points is that "superhuman" and "superintelligence" don't mean what they're usually implied to mean because the right comparison is never human vs AI but rather human + AI vs AI.
This point applies with equal force to applications of generative AI in education, too.
Excellent article, and many compelling arguments about the rate of diffusion. However, what if the rate of adoption/change is a fraction of what we have experienced due to the exponential growth of technology and the shortening of time between breakthroughs? I explore that in my post: https://open.substack.com/pub/charleseldering/p/how-long-will-it-take-for-productivity?r=24o7sp&utm_medium=ios
In other words, could “true” AGI (whatever that means) be right around the corner?
I don't think you're right about the labs being incentivized to declare AGI achieved. They're incentivized to keep insisting it's coming any day now because that keeps the VC cash flowing in. But declaring it here means that all dries up. Any lab could claim they've released it whenever. They could do it today. But they won't because the investments pour in for the promises.
That's possible! But some counterarguments:
* If a startup makes this claim to get attention, the big players might be forced to follow suit to avoid looking like they've fallen behind (which we mention in the essay).
* If they declare AGI achieved, they could arguably raise even more cash than they're doing now if they claim that they are going to automate the whole economy and all they need to do is to build out the infrastructure for inference fast enough.
* Another way it could help them raise even more money is if they say that having achieved AGI puts them in the perfect place in the race to ASI.
I wonder what happens if a large group of otherwise independent AI/ML researchers band together and inject some "chaos" into the system: Write an op-ed or open letter or whatever with a large bunch of "aligned" scholars (see what I did there) and preemptively declare that AGI is here (say that o3-high or whatever) is it. That would be an interesting shock to the system! How would that change the dynamic between MS and ClosedAI? Of course it would potentially harm the reputations of said scholars ... but of course one can always just point and say "well would you like to prove us wrong?" Probably unethical, but I wonder if any of the tech executives would know any empathy or ethics if it came up behind them ...
Isn’t it logical to assume that if/when AGI is achieved, the company who invented it will control it and weild it themselves rather than handing out an "AGI subscription"? I don’t see anyone examining this angle of power/capability consolidation. In this scenario, no other CEO should be excited about AGI because it will only start a countdown to their own demise. They won't be able to share a slice of the action. For example, remember when Eric Schmidt talked about being able to say "AI: clone tiktok for me, make it better then deploy it." In simple terms then, couldn't OpenAI or whoever use AGI as normal software and just clone the top grossing apps and disrupt them all overnight? Why sell AGI as a product when you can use it to become King? Why do I feel like I'm the only one thinking this?
Thinking is rare these days.
I bought your book (twice now; I have the first copy to someone).
Are there any other books or blogs on AI (or any other issues) that you’ve read and would recommend?
My A.I. Education content.
2024 podcast series:
1 of 3:
https://soberchristiangentlemanpodcast.substack.com/p/ai-deception-s1-ep-1-of-3?r=31s3eo
2 of 3:
https://soberchristiangentlemanpodcast.substack.com/p/ai-deception-s1-ep-2-of-3?r=31s3eo
3 of 3:
https://soberchristiangentlemanpodcast.substack.com/p/ai-deception-s1-ep-3-of-3?r=31s3eo
2025 podcast series:
1 of 3:
https://soberchristiangentlemanpodcast.substack.com/p/ai-deception-2025?r=31s3eo
2 of 3:
https://soberchristiangentlemanpodcast.substack.com/p/ai-deception-2025-pt-2?r=31s3eo
3 of 3:
https://soberchristiangentlemanpodcast.substack.com/p/ai-2025-update-the-window-is-closing?r=31s3eo
Essays:
A)
https://open.substack.com/pub/soberchristiangentlemanpodcast/p/essay-the-closing-window-the-real?utm_source=share&utm_medium=android&r=31s3eo
B)
https://soberchristiangentlemanpodcast.substack.com/p/essay-the-ai-deception-leo-to-gemini?r=31s3eo
C)
https://soberchristiangentlemanpodcast.substack.com/p/essay-the-ai-deception-2025-unraveling?r=31s3eo
Great read ! Too much hype on AGI because it’s driving investment; but with no clear definition AND purpose it feels more like a narrative than a true goal. Importantly I am quite convinced LLMs alone in their current forms CANNOT help achieve “human like intelligence” because they don’t have multi sensorial cognitive capabilities. LLMs are amazing at synthesising knowledge at hyper scale. But that’s all they are.
I write some perspectives here: https://www.koncentrik.co/p/the-illusion-of-intelligence-why
Such good stuff: Love the substack! This whole AGI vs. ASI thing is exhausting! For the "rationalists" and AI enthusiasts that claim that Superintelligence (TM)(R)(C) is just "out performing all humans at all tasks". To me this seems like total nonsense! A comparison is alwaysbetween humans and say other primates (as though evolution and intelligence is some neat continuum) ... but the issue is that if we are by definition more intelligent than other life then, and the metric is "cognitive tasks" that we outperform other animals. What metrics are those exactly?
A single spider can navigate more "varied" and perform more complex tasks than we can on a given day. It wouldn't make sense to give a squid a verbal analogies test. What determines which one of those tasks is more difficult than the other? Is there a handy scale for the "Algorithmic Complexity" of any arbitrary task (apparently it is famously difficult to prove that an arbitrary task is more complicated than another in a "universal way")?
And besides: would we truly be even able to communicate with a truly "super intelligence"!? what use would it have of language? Or abstractions/concepts we use to "hide" complexity of mental objects to make sense of the world that is vastly more complex than the artificial environments of our RL trained "agents" with limited "symbolic" processing capabilities! Would we even be able to differentiate their actions from randomness!? I think Cosma Shalizi's and Alison Gopnik's idea about LLMs being "cultural" technologies is more coherent than what the AI doomer cult's ideas!
As an example about "rate limitations/choke points" to diffusion of tech. I work for a large company people have heard of (just a regular boring ass non tech company) that is very profitable and has a presence in every continent.The company JUST MOVED from a paper based "approval" system ... to a "digital" one a few years ago! But the digitization was just Pdf versions of forms, "coversheets" and "drawings" that are scanned in ... so at best I use an approved LLM (a lobotomized o3-mini and claude 3.5) to translate and write emails etc etc. But the whole thing could have been designed with the technology of the 2000s to obviate the need for me to manually sign documents in the first place! Our "General Purpose Technology" is Excel: And I don't mean for number crunching or pivot tables ... but as a drawing tool, layout tool, presentation tool, communication tool ...
It's not like I need to solve canned Algebraic Topology or AIME/Frontier Math problems in my day to day! I bet few people do! In the real world, you won't be given a well crafted prompt of a math problem ... valuable knowledge is "diffuse" and "tacit" and most of the time it's about how you can use that knowledge to figure out what even needs to be solved.
Thanks for making the argument in plain & simple language - I'm convinced. The one place you did not cover (& I know you said you would get back to this later) is military uses of AI. Both diffusion & potential harmful use (including in the cyber domain) appear to me to be riskier than all the other domains where your arguments make sense.
One other quibble I have is on the speed of diffusion in China - from what anyone can observe with payment systems, e commerce, EV's..the Chinese are much quicker to innovate & adopt than anyone else. Your argument here does not bake all this evidence.
And more to the point- the models are still bad. Just so bad. I don't know what sorts of experiences other people are having that are enabling these evangelical awakenings, but every time I check back in with an LLM to see if there's some actual utility there, something dumb happens. I don't think I have ever had an LLM do a single task that was not sufficiently gnarled up in some way that it was an open question whether I actually saved any time or effort. Which is not to say it is not all a very impressive magic trick! But how are the 'the AI god is nigh' people not noticing that these models are worse at algebra than ten dollar calculators? They can seem very smart and answer last year's bar exam questions but brutally fail this year's- gosh, it's almost like this is mostly search. Which isn't nothing- but it's also not transformative, especially when the value of search is at least gesturing towards provenance, attribution, and alternative perspectives.