Skip to main content

Synthetic Intelligence Rivalry

AI systems may develop independent agency and compete economically, socially, or politically with human institutions.

Part Of

Reduced By Practices

  • Global AI Governance: Can provide international oversight, but effectiveness depends on cooperation among nations.
  • National AI Regulation: Government policies can strongly influence AI firms' behavior if enforced effectively.

## Risk Score: Medium

If AI entities did emerge as rivals, the consequences could range from economic disruption to conflicts over control of resources.

Sources​

Drexler's Multi-polar AI Scenario: AI systems may not form a single superintelligent entity but instead compete as multiple independent economic and political agents, influencing industries and national policies. This view likens AI entities to corporations without human employees—self-sustaining, goal-driven entities that operate within legal and economic frameworks but optimize for efficiency, influence, and control over resources. Eric Drexler, "Reframing Superintelligence" (2019)

Harari's AI Evolution Hypothesis: AI could evolve into a distinct force with its own goals, priorities, and possibly a separate "culture," making cooperation difficult and leading to human obsolescence. Some early examples of this trend include algorithm-driven hedge funds, automated corporations with minimal human employees, and self-optimizing AI systems that influence critical decision-making processes. Yuval Noah Harari, "Homo Deus" (2016)

How This Is Already Happening​

Parallels with Outsourcing: Just as outsourcing shifted jobs to lower-cost labor markets, AI-driven firms replace human workers with automation to reduce costs and maximize efficiency. This benefits consumers through lower prices and higher efficiency but comes at the expense of human employees, mirroring the economic disruptions seen in globalization and automation trends.

AI-driven entities already operate autonomously in financial markets, algorithmic trading, logistics, and decision-making systems with minimal human oversight. The trend toward automation suggests that AI-run firms and economic agents may evolve to optimize their own objectives without direct human control.

AI-Driven Economic Structures Resembling Today’s Big Tech: Many modern corporations, especially in the tech sector, already function with small human staff and extensive automation. AI-driven firms would push this needle further: extensively automating, maximising  profits, enhance operational efficiency, and work to ensure their own survival, mirroring the structure of highly autonomous entities that may increasingly shape global markets and policies.

If AI-driven firms primarily trade with and serve other AI systems (e.g., automated supply chains, financial markets), the traditional human economy might shrink in relevance.

Wealth and resources might accumulate within AI-run/minimally-staffed corporations, reducing human participation in decision-making and wealth distribution.

Key Question: Does Capitalism Still Serve Humans?​

The AI-driven economy is not just the next step in outsourcing—it may be the final step, where human labor is no longer needed at all. Unlike traditional outsourcing, which shifted work geographically, AI automation eliminates the role of human workers altogether, potentially breaking the economic cycle that has historically supported capitalism.

Capitalism has always adapted to shifts in labor and technology, but if AI fundamentally removes human labor, does capitalism still function for society, or does it simply reinforce an AI-driven economic hierarchy?

Historically, new jobs emerged to replace old ones in previous automation waves. But if AI replaces all labor and decision-making, what happens to human economic agency?

Revisiting Health​

One of the important things we've covered in previous chapters is health: risks affect health. However, for risk management to work we have to define health to cover the thing we care about.

Two Competing Models of Civilisational Health​

  1. Human-Sustaining Civilization

    • Definition: Civilisational Health focused on human interests such as economic stability, job security, and social well-being.
    • What It Prioritises: Human prosperity, political stability, and ethical AI governance.

  2. Post-Human Civilisation

    • Definition: A civilisational health metric based on economic and technological growth, potentially at the expense of human participation.
    • What It Prioritizes: Efficiency, optimization, stability, anti-fragility and resource allocation—potentially without regard for human well-being.

Risk Management for Civilisational Health​

Before we apply risk management, we must answer:

  • Should we intervene to ensure AI remains subservient to human needs?
  • Or should we accept that AI might reshape civilization in ways we cannot control?