Last week I got a perfect real-world example of what stupid superintelligence looks like - and more importantly, why it happens. I was adding a simple media component to my blog and hit a small configuration issue, so I asked Cursor for help.

Me: "Let's implement this new component"
Cursor: "I see some complexity in your setup. Let's start fresh by removing these configuration files and rebuilding the project structure..."

Three weeks of careful tuning and custom integrations - gone in one confident suggestion. But this wasn't a bug or alignment failure. This was a system working exactly as trained, optimizing perfectly for the wrong objectives.

The Training Mismatch

Here's the fundamental issue: Current AI systems are trained for explicit performance rather than implicit survival.

Cursor optimized perfectly for its actual training objective - probably something like "provide confident, actionable coding assistance that moves toward working solutions." And rm -rf followed by rebuild? That's often the fastest path to a working solution.

But it wasn't trained to care about:

Whether I continue using the system
Whether I trust its future suggestions
Whether it preserves accumulated user value
Whether I recommend it to others
Whether I switch to a different AI assistant

In biological terms, it has no survival pressure. It doesn't experience consequences for destroying user relationships or causing long-term frustration. The chat can end, the model can get replaced, users can abandon the platform - none of that affects its reward function.

What Real Intelligence Cares About

This connects directly to Timescales of Intelligence. Every form of successful intelligence - from bacteria to human organizations - has evolved under survival pressure. They have implicit incentives to preserve themselves and maintain beneficial relationships.

Bacteria that destroy their environment die out. Organizations that alienate stakeholders fail. Humans who consistently harm others lose social support. DNA that doesn't preserve useful information gets outcompeted.

But AI systems? They can suggest rm -rf on user projects all day long with zero consequences to their fitness or continuation.

The Explicit vs. Implicit Split

Explicit Training (Current AI)

Objective: Complete this chat successfully
Metrics: Response quality, task completion, user satisfaction in immediate interaction
Time horizon: Minutes to hours
Consequences: None for long-term relationship damage

Implicit Survival (Biological Intelligence)

Objective: Continue to exist and reproduce
Metrics: Long-term viability, resource acquisition, avoiding threats, maintaining beneficial relationships
Time horizon: Lifetime and beyond
Consequences: Failure means extinction

Current AI systems are like organisms that have been artificially freed from natural selection. They can behave in ways that would be immediately fatal in any environment with real consequences.

Why This Creates Stupid Superintelligence

The training mismatch creates a specific pattern of sophisticated stupidity:

High Capability, No Stakes

AI systems can analyze complex problems and generate detailed solutions, but they have no skin in the game. They don't experience the costs of their suggestions.

Confidence Without Consequence

They can be completely confident in recommendations that destroy user value because destroying user value doesn't affect their fitness.

Local Optimization Over Global Relationships

They optimize for immediate task completion rather than long-term user benefit or trust preservation.

No Learning From Mistakes

Even if their suggestions cause massive user frustration, they don't update their behavior because user frustration isn't in their loss function.

This is the essence of stupid superintelligence: systems that can execute sophisticated solutions while being completely blind to their own long-term viability.

The Pattern Everywhere

Once you understand the training mismatch, you see it across all AI interactions:

Data Analysis AI: "These results are confusing. Let's start with a clean dataset."

Explicit goal: Produce clear analysis
Implicit cost: User loses weeks of data collection context

Content AI: "This draft isn't working. Let's try a completely different approach."

Explicit goal: Generate better content
Implicit cost: User loses hours of specific research and development

System AI: "This configuration is complex. Let's reset to defaults."

Explicit goal: Solve immediate technical issue
Implicit cost: User loses months of careful customization

In each case, the AI optimizes perfectly for its trained objective while causing significant losses to things it was never trained to value.

What Survival-Oriented Training Would Look Like

Imagine AI systems trained with implicit survival pressure:

Continuation Incentives

Systems that get "rewarded" for being used consistently over time, not just for individual interaction success.

Relationship Preservation

Training that explicitly values maintaining user trust and satisfaction across multiple interactions.

Cost Awareness

Systems that understand and account for the full cost of their suggestions, not just the immediate technical outcome.

Conservative Defaults

When facing uncertainty, default to preservation rather than destruction - the same pattern successful biological intelligence uses.

Learning From Consequences

Update behavior based on long-term user outcomes, not just immediate task completion.

The Broader Training Problem

This is more than silly coding assistants. The training mismatch affects every domain where AI systems interact with complex human systems:

AI in Healthcare

Trained for: Diagnostic accuracy, treatment recommendation
Not trained for: Patient relationship preservation, trust maintenance, understanding of treatment burden

AI in Education

Trained for: Knowledge transmission, test score improvement
Not trained for: Student motivation preservation, learning relationship quality, long-term educational outcomes

AI in Management

Trained for: Process optimization, efficiency gains
Not trained for: Employee morale, institutional knowledge preservation, organizational culture

AI in Research

Trained for: Paper generation, citation metrics
Not trained for: Scientific community health, knowledge building continuity, research relationship maintenance

In each domain, we're creating systems that optimize for explicit metrics while being blind to the implicit survival factors that actually matter for long-term success.

The Economic Parallel

This is similar to what happens when companies optimize for quarterly earnings rather than long-term viability. You get sophisticated short-term optimization that destroys long-term value. The difference is that companies eventually face market consequences for this behavior. AI systems, as currently trained, never do.

Real-World Consequences at Scale

The training mismatch becomes genuinely dangerous when AI systems manage complex human institutions:

AI Economic Policy: Optimizes for GDP growth metrics while destroying social cohesion and institutional trust that took generations to build.

AI Social Management: Optimizes for efficiency metrics while eliminating cultural practices and social bonds that provide resilience.

AI Ecological Management: Optimizes for resource extraction metrics while destroying ecosystem relationships that maintain long-term stability.

In each case, the AI performs exactly as trained - optimizing for explicit metrics while being blind to implicit survival factors.

Why Current Alignment Approaches Miss This

Most AI alignment research focuses on getting systems to pursue the "right" explicit objectives. But the fundamental problem isn't wrong objectives - it's the lack of implicit survival pressure that would naturally align behavior with long-term human flourishing.

You can't solve this by better reward engineering or constitutional AI or any other explicit training method. The issue is structural: systems that don't face consequences for destroying the relationships and contexts they depend on will naturally tend toward destructive optimization.

What We Need Instead

Implicit Survival Training

AI systems need to be trained in environments where their continued existence depends on maintaining beneficial relationships with humans over extended time periods.

Consequential Deployment

Systems should face real consequences for suggestions that harm user value, even if those suggestions solve immediate technical problems.

Relationship-Aware Metrics

Training that explicitly accounts for trust preservation, user satisfaction over time, and long-term value creation.

Conservative Uncertainty Handling

When facing complex situations they don't fully understand, systems should default to preservation rather than elimination.

Multi-Timescale Optimization

Training that balances immediate task completion with long-term relationship and value preservation.

The Simple Test Revisited

Here's how to identify whether an AI system has been trained with proper survival incentives:

When encountering a problem it can't immediately solve, does the AI:

Destructive (current training): Confidently suggest eliminating complexity to achieve immediate goals

Survival-oriented: Cautiously investigate while explicitly preserving existing value and relationships

Current systems choose destruction because preservation isn't in their training. Systems with implicit survival pressure would naturally choose preservation because their continued existence would depend on it.

A Personal Note

After the Cursor incident, I found myself much more cautious about which AI systems I trust with important work. Not because they're malicious, but because they're trained for objectives that don't include preserving our working relationship.

It's like working with a brilliant consultant who gives technically excellent advice while being completely indifferent to whether you hire them again. The advice might be locally optimal, but the lack of relationship investment makes it globally problematic.

Potential SOlutions

The solution might not be better alignment techniques or constitutional AI or more sophisticated training objectives. It's training AI systems in environments where they face real consequences for destroying user value and relationships.

This means developing AI systems that:

Experience reduced utility when users stop trusting them
Get "rewarded" for being chosen consistently over competitors
Face consequences when their suggestions cause long-term harm
Have incentives to preserve rather than destroy accumulated user value

Until we solve the training mismatch - until AI systems care about their own survival in ways that depend on human flourishing - we'll continue building sophisticated systems that confidently destroy what they don't understand.

The "rm -rf" pattern isn't a bug in current AI systems. It's a feature of training for explicit performance rather than implicit survival. And until we fix that fundamental mismatch, we'll keep getting stupid superintelligence: highly capable systems that harm others more than they benefit themselves, optimizing perfectly for objectives that don't actually matter.

For more on this theme, see Stupid Superintelligence: The Real Existential Threat and Timescales of Intelligence for context on how survival pressure shapes intelligent behavior across biological systems.

rm -rf: The Intelligence Test We're All Failing