TLDR: why would anyone invest so much in something if they believe it will induce a scenario where their investment is worth 0?

Talking to people working at the AI labs, particularly Anthropic, can be incredibly unnerving. They are much closer to this technology than anyone else and they are MUCH more scared about its impact than your average Joe.

Try to pin them down on exactly what and you get the obvious ones:

  1. Someone uses AI to help them build a bio weapon or possibly a nuke.
  2. Someone uses AI to help them deliver a crippling cyber attack.

All very legitimate, understandable and definitely real concerns. Let's call these the tangible downside scenarios. They also of course talk about the tangible upside scenarios. No more disease, knowledge democratised, no more lawyers…there's a long list. There's some slightly more intangible downsides which look a bit like Terminator.

Then there's this weird category of a more intangible upside scenario, that's also potentially a major downside scenario. Elon Musk calls it Abundance, some people call it utopia, AGI or The Singularity. Various descriptions of it exist, but essentially what I can tell from people who think this could happen is that they don't really know what exactly it is but it's going to be BIG. Like really, really BIG. Like everyone in the future will talk about before AI and after AI, it will be up there with religion and agriculture in terms of impact on humanity. There will be massive job losses, a crisis in purpose, but it's gonna be totally worth it.

But press them more deeply on what this world actually looks like and NONE OF THEM KNOW. So all this money is being invested into the attempt to achieve something that we don't really know what it is.

This does freak me out a bit. But let's break it down. What could the age of abundance actually be like? Well by the name of it and Elon's description, anyone, anywhere could have anything they could possibly want at any moment in time. How exactly? Well you know, it's the Singularity, so anything is possible. So follow this logic and that means money ceases to have any relevance. There is no need for an economy. There's no need for jobs. There's no need for a political system. There's no need for anything really…

So why on earth would anyone invest hundreds of billions of dollars in something that will make their investment 100% worthless? I guess you might make that type of investment altruistically if it makes everyone effectively billionaires, but cynically, I doubt that's what the LPs of VCs want to hear, so I doubt that's what the CEOs and VC partners are telling them. I also don't think the world's top capitalists want what is effectively a socialist fantasy.

My theory is that fundamentally, deep down and despite all the fear mongering, none of the VCs or AI CEOs really believe the singularity is going to happen. They all believe that AI will kind of improve productivity, make businesses more efficient and maybe do some of the good things mentioned above but generally it will exist in the current framework we exist and operate in today. As in there will be no crazy step change but they are also going to make a shed load of money out of it. They have to see the world like this otherwise their investments are worth 0.

The only other explanation is that there is a secret conspiracy between the AI CEOs, the VC partners and shady government officials that they want the singularity to somehow be started with their own values and ideology. But then again it's a super intelligent AGI singularity thing and any attempt at alignment won't be effective against it. It can do what the hell it likes. So it doesn't matter what its starting position is, holding on to frivolous political and cultural perspectives are of very little use to it and are just meaningless human noise.

So all that is to say, I'm not convinced by this hype machine. What am I missing here?