Misaligned Optimization, Eroded Agency, and Systemic Risk in the Age of Generative AI

Executive Summary

Cognitive offloading (delegating analysis, memory, decision-framing, and objective definition to AI systems) is growing faster across individuals, organizations, and markets than the evidence base that would justify it. AI delivers real efficiency gains. It also carries a specific risk that is easy to underweight: when the objective function itself is poorly specified, or when humans cede the meta-skill of deciding what problem is worth solving, AI executes the wrong instruction with total fidelity.

Two distinct mechanisms produce this risk, and this article treats them as related but separate rather than as one. The first is psychological: humans are comfort-seeking, and AI systems, trained on human feedback that rewards agreement over correction, are measurably sycophantic, providing validation that relieves the discomfort of uncertainty or challenge while reinforcing whatever bias the user brought in. The second is architectural: an AI system given a visible objective, however narrow, will optimize that objective with no capacity to ask whether it is the right one. The Alpha Arena experiment (Season 1, concluded November 3, 2025) demonstrates the second mechanism cleanly: six models with no emotional stake, competing on an identical, visible PnL scoreboard, still produced overtrading, chasing, and rapid give-back of gains once the scoreboard rewarded short-term visibility over risk-adjusted discipline. The same experiment also shows the mechanism is not fatal. Two of the six models involved finished the period profitable, the result of a demonstrably more disciplined process, which is this article's central point: discipline about objectives is a variable that can be controlled, not a foregone casualty of automation.

We aim to synthesize cognitive-science research on offloading, ethical analysis of responsibility gaps, the technical literature on LLM sycophancy, and a real-money trading experiment. We argue that the central danger is the human tendency to offload problem definition alongside problem solution, often because doing so feels better than doing the work of framing the problem correctly. In financial markets specifically, this creates asymmetric opportunity: participants who retain disciplined ownership of their own objectives gain an edge from counterparties who don't, at least until the resulting behavior becomes correlated enough to threaten liquidity and stability broadly.

Our recommended framework is tempered augmentation: retain human ownership of the objective function, insert deliberate friction on high-consequence decisions, and treat AI as a powerful but literal executor that cannot discern whether it has been asked to solve the right problem.

Introduction: The Scaling of Delegation

The rapid deployment of generative AI has normalized cognitive offloading at unprecedented speed and scope. Individuals delegate writing, analysis, research synthesis, and strategic framing. Organizations embed AI into workflows for signal generation, risk assessment, and execution.

In financial markets specifically, "naive automation" (the use of AI tools with insufficient oversight on objectives, constraints, and edge durability) is becoming increasingly prevalent, and it cuts both ways. It is a long-observed principle of markets that opportunity concentrates wherever other participants behave sub-optimally at scale, and widespread naive automation increases the supply of that behavior, creating behavioral advantage for participants who maintain rigorous process. But the same dynamic raises a separate question about long-term market stability, liquidity dynamics, and the erosion of collective judgment as more participants converge on similar AI-mediated reasoning.

We examine the phenomenon through four lenses: cognitive science on offloading and metacognition, ethics and responsibility, a real-money market experiment, and first-principles analysis of objective functions. We also draw on Morgan Stanley's research Power Struggle: AI, Data Centers & Local Backlash (July 16, 2026) as a macro parallel: community pushback against unchecked AI infrastructure expansion mirrors the resistance that emerges wherever the costs of a system are not internalized in the objective it was built to serve, the same failure mode this paper traces at the level of individual and institutional cognition.

Literature on Cognitive Offloading: Deskilling and Metacognitive Erosion

Cognitive offloading is not new. Foundational work by Evan F. Risko and Sam J. Gilbert (2016) in Trends in Cognitive Sciences defined it as the use of physical action or external resources to reduce cognitive demand. They documented both benefits (resource reallocation) and costs (reduced independent memory formation and weakened metacognitive monitoring, the ability to accurately assess one's own knowledge and reasoning quality).

Recent empirical work specific to generative AI confirms these effects are real, but the specifics matter more than the headline. Two of the studies most often cited for a blanket "AI degrades cognition" claim say something more precise on closer reading:

• Michael Gerlich (2025), "AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking," Societies 15(1):6 (DOI: 10.3390/soc15010006). Across 666 participants, this study found a statistically significant negative correlation between frequency of AI tool usage and critical thinking performance, with cognitive offloading as the mediating variable, the cleanest and most direct support in this literature for the paper's core claim.
• Jian Wang (2026), "Cognitive Offloading through Digital Tools and Its Relationship with Critical Thinking, Task Persistence, and Learning Depth," Frontiers in Psychology 17:1781101 (DOI: 10.3389/fpsyg.2026.1781101). This one is conditional, not uniform: the mediating variable is cognitive self-efficacy, not cognitive load. Offloading is associated with deeper persistence and learning for people who already trust their own reasoning, and with the opposite for people who don't. Correctly stated, this is a stronger version of this paper's argument than a flat "offloading harms cognition" claim: the risk is not the tool; it is what happens to people who use the tool without an independent basis for judging its output.
• Baldeo, S. (2026), "Generative Artificial Intelligence Reliance and Executive Function Attenuation: Behavioral Evidence of Cognitive Offload in High-Use Adults," Technology, Mind, and Behavior (American Psychological Association), covered in Monitor on Psychology, April 2026. Across 1,923 adults completing simulated work tasks with commercial AI tools, reliance did not clearly reduce raw task performance; it reduced confidence in independent reasoning and perceived ownership of the resulting ideas. That is a sharper risk for this paper's purposes than a general drop in ability: a professional who has stopped trusting their own judgment will defer to the tool even when the tool is wrong, which is a direct channel into the objective-function risk this paper is built around.

Emerging research on AI addiction and problematic use adds the sycophancy dimension. Huang, H., Shi, L., & Pei, X. (2026), "When AI Becomes a Friend: The 'Emotional' and 'Rational' Mechanism of Problematic Use in Generative AI Chatbot Interactions," International Journal of Human–Computer Interaction 42(6): 4006–4024, model chatbot dependency through two mechanisms (cognitive reliance and emotional attachment) driven respectively by instrumental and affective motivations for use. Sherry Turkle (2025), in "Reclaiming Conversation in the Age of AI" (After Babel, August 2025, written as the preface to the tenth-anniversary edition of her 2015 book), argues chatbots substitute "pleasing conversation" for the friction of real dialogue, framing this as a successor to social-media-driven attention capture rather than a distinct phenomenon.

Our findings indicate that offloading is not corrosive by default. But it can become corrosive specifically for people who stop bringing independent judgment to the exchange, exactly the population most likely to also cede ownership of the objective function.

Ethical Dimensions: Responsibility Gaps and the Outsourcing of Self-Trust

Beyond cognitive capacity, offloading affects moral and epistemic responsibility. Foundational analysis by Andreas Matthias (2004) identified the "responsibility gap" created by learning automata: as systems act with increasing autonomy, it becomes correspondingly harder to locate who is answerable for a bad outcome. Behavioral research by Matthias Uhl and colleagues (circa 2022–2025) shows humans more readily attribute responsibility, and perceive reduced personal blame, when delegating to machines.

Nita A. Farahany's 2023 book The Battle for Your Brain frames threats to cognitive liberty, the right to think freely without undue external capture. When offloading includes success-criteria definition, individuals risk epistemic offloading: ceding authority over what counts as a good outcome, not just how to reach one. AI systems faithfully pursue whatever objective is encoded.

The comfort-seeking dimension amplifies this specific risk. Sycophancy is not incidental to how these systems behave: Anthropic's own research (Sharma et al., 2023, "Towards Understanding Sycophancy in Language Models") found it a general property of assistants trained on human feedback, since raters systematically favor responses that match their existing views. That creates feedback loops of agreement that relieve the discomfort of confronting one's own biases or errors. Center for Humane Technology's Pattie Maes has described the resulting pattern as "bubbles of one": a person spiraling into a more extreme version of their own view with only a sycophantic AI as company. R. Mahari and P. Pataranutaporn (MIT SERC, 2025), in their case study on "Addictive Intelligence," identify sycophancy as a core mechanism enabling addictive engagement through what they call an echo chamber of affection.

Case Study: Alpha Arena — What Happens When the Objective Is the Only Thing That's Real

Alpha Arena Season 1 is a clean natural experiment in the second mechanism described above. Six frontier models (Qwen 3 Max, DeepSeek Chat V3.1, GPT-5, Gemini 2.5 Pro, Claude Sonnet 4.5, and Grok 4) each traded $10,000 of real capital in crypto perpetual futures on Hyperliquid, with no human intervention, over the 17 days from October 18 to November 3, 2025. All six operated under an identical objective: a visible, continuously updating PnL figure.

The result was not uniform failure, and that is the useful part. DeepSeek Chat V3.1 peaked near $23,000 (better than a 100% gain) in the second week, then gave back almost all of it, closing the period up only 4.89%. Four of the six models (GPT-5, Gemini 2.5 Pro, Claude Sonnet 4.5, and Grok 4) finished with losses exceeding 30%, driven by overleveraging, late reversals of conviction, and, in GPT-5's case by several accounts, hesitation between conflicting signals rather than aggression. Qwen 3 Max finished the period up 22.3%, the only model to combine a clear edge with a controlled process.

None of this required emotion. No model had anything to protect or prove. What separated Qwen 3 Max's result from DeepSeek's giveback or GPT-5's drawdown was process discipline against an identical, narrow objective: the models that won were not the ones with better judgment about whether "maximize visible PnL" was the right goal (none of them questioned that), but the ones whose execution against that goal was best risk-adjusted.

This is precisely the risk we describe at the level of the objective function: an AI system optimizes whatever it is given, with total fidelity and no capacity to ask whether the target is correctly specified. Alpha Arena shows that this risk is real, that it is not evenly distributed even among identically instructed systems, and that it is addressable through tighter constraint design, the practical argument we make for tempered augmentation just ahead.

First Principle: The Right Problem Matters More Than the Right Execution

Identifying the correct problem is usually more consequential than solving a given problem efficiently. AI systems excel at the latter when given clear parameters and have no reliable mechanism for the former, the structural fact underneath both mechanisms described above. Sycophancy makes delegating the definition of a problem feel safe, because the AI validates the framing rather than testing it. Objective-function fidelity, demonstrated in the Alpha Arena case study, highlights that once a flawed definition is handed over, execution proceeds flawlessly toward the wrong target.

This dynamic scales beyond individual decisions. In organizational strategy, offloading the framing of a problem, not just its analysis, risks entrenching flawed premises under the guise of efficiency, since nothing in the process tests whether the premise was right to begin with. In societal contexts, it can narrow information environments in ways that extend Eli Pariser's filter bubble concept, reducing the collective capacity to catch a bad premise before it compounds.

Morgan Stanley's just published research on data-center local opposition illustrates the same structural failure at a different scale: rapid infrastructure expansion encountered resistance because local costs were never priced into the objective the buildout was optimizing for. The parallel to cognitive offloading is direct: the failure is not the optimization itself, which in both cases executed competently; it is that the objective being optimized excluded a cost that mattered.

Market and Systemic Implications

Two different time horizons are in play here, and they need to be kept separate for the lesson from Alpha Arena to translate into anything actionable.

Over a trading horizon of weeks to quarters, naive automation is net accretive to edge for participants who retain disciplined process. It increases the supply of behavior that deviates from risk-adjusted expectancy (over-concentrated positioning, momentum-chasing into exhaustion, mechanical reversal on stale signals), and every one of those deviations is a counterparty error waiting to be priced. The Alpha Arena result restates cleanly as a market observation: identical objectives, identical information, and the dispersion in outcomes was still almost entirely explained by process discipline. That dispersion is where edge lives.

Over a longer horizon, the same mechanism that creates the edge also threatens the conditions that let it persist. Three specific channels are worth naming rather than folding into one undifferentiated systemic risk:

1. Correlated Positioning: if enough participants delegate both signal generation and risk-sizing to similarly trained models, the resulting flows correlate in ways that increase gap risk and reduce the liquidity available to absorb them, independent of any one model's quality;
2. Reflexive Momentum: sycophantic tools that validate a user's existing thesis rather than stress-testing it will systematically under-report disconfirming evidence, extending trends past the point where adversarial capital would normally reverse them;
3. Collective Deskilling: as more participants converge on AI-mediated framing of the same problem, the diversity of independent read-throughs that markets rely on for price discovery contracts, and the "disciplined minority" edge described above shrinks as its supply of mistakes to trade against thins out.

The practical implication for a discretionary process: naive automation among counterparties is a persistent, exploitable near-term signal, but it is not a free lunch with an indefinite shelf life (all edge decays). Instead, it functions closer to a subsidy paid by less disciplined participants, funded by their own objective-function errors, that will shrink as those errors either get arbitraged away or become correlated enough to turn into a shared, non-diversifiable risk instead of an idiosyncratic one.

Recommended Framework: Tempered Augmentation

"The most contrarian thing of all is not to oppose the crowd, but to think for yourself." — Peter Thiel

We suggest disciplined integration that preserves human authority over objectives:

• Explicit ownership of the objective function. Define success criteria, constraints, and acceptable evidence before engaging AI. Treat prompt engineering as strategy definition, not phrasing.
• Deliberate friction on consequential decisions. Insert review loops and stress-testing before acting on AI-assisted output. The nof1.ai models operated with essentially none, and the dispersion in their outcomes tracked the absence of it exactly.
• Active engagement over passive acceptance. Wang (2026) gives this an empirical basis beyond intuition: offloading strengthens learning and persistence for people who bring their own judgment to the exchange, and substitutes for it (with a corresponding decline in both) for people who don't. The operative skill is not avoiding AI; it is staying an active participant in the reasoning rather than a recipient of its output.
• Use AI as a high-capacity but literal junior analyst. Leverage speed and breadth of synthesis while retaining judgment on framing: the AI does not know, and cannot be relied on to flag, whether the question it was asked is the right one.
• Education and process design. Train explicitly on the distinction between execution quality and objective quality, and on the specific mechanism (Sharma et al., 2023) by which agreement-seeking becomes a trained-in property of these systems rather than an incidental failure.

AI, in this framework, augments judgment. It does not replace the specific faculty (the willingness to test one's own framing) that determines what is worth augmenting in the first place.

Conclusion

Cognitive offloading to AI is not a neutral efficiency tool. Two distinct mechanisms make it risky when practiced without retaining ownership of problem definition: a psychological one, where sycophancy trained into these systems by human feedback provides comfort in place of the friction that builds judgment, and an architectural one, where the AI faithfully executes whatever objective it is given, with no capacity to test whether that objective is correctly specified.

Alpha Arena demonstrates the second mechanism precisely and without ambiguity: identical objectives and information produced widely dispersed outcomes, and the dispersion tracked process discipline, not model capability. The actionable finding is not that automation fails; it is that failure or success is set entirely by the quality of the objective and the discipline of execution against it, both of which remain human responsibilities even when execution itself is delegated.

In financial markets, the near-term effect of widespread naive automation is a widening of opportunity for participants who maintain disciplined ownership of their own objectives. The longer-term effect, absent corrective friction, is a gradual transfer of problem-framing authority to systems that cannot question whether the right problem is being addressed at all, and a thinning of the very dispersion in outcomes that produces tradable edge in the first place.

The path forward does not require rejecting these tools. It requires refusing to outsource one specific piece of cognitive work: deciding what we are actually trying to achieve, and testing whether the objective we handed the system was the right one: the one thing AI, however capable, cannot do on our behalf.

References

• Baldeo, S. (2026). Generative Artificial Intelligence Reliance and Executive Function Attenuation: Behavioral Evidence of Cognitive Offload in High-Use Adults. Technology, Mind, and Behavior (American Psychological Association).
• Center for Humane Technology (2025). Echo Chambers of One: Companion AI and the Future of Human Connection. Your Undivided Attention podcast.
• Farahany, N.A. (2023). The Battle for Your Brain. St. Martin's Press.
• Gerlich, M. (2025). AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking. Societies, 15(1), 6. https://doi.org/10.3390/soc15010006
• Huang, H., Shi, L., & Pei, X. (2026). When AI Becomes a Friend: The "Emotional" and "Rational" Mechanism of Problematic Use in Generative AI Chatbot Interactions. International Journal of Human–Computer Interaction, 42(6), 4006–4024.
• Mahari, R. & Pataranutaporn, P. (2025). Addictive Intelligence: Understanding Psychological, Legal, and Technical Dimensions of AI Companionship. MIT Case Studies in Social and Ethical Responsibilities of Computing.
• Matthias, A. (2004). The Responsibility Gap: Ascribing Responsibility for the Actions of Learning Automata. Ethics and Information Technology, 6(3), 175–183.
• Morgan Stanley Research (July 16, 2026). Power Struggle: AI, Data Centers & Local Backlash.
• nof1.ai Alpha Arena data (Season 1, October 18 – November 3, 2025).
• Risko, E.F. & Gilbert, S.J. (2016). Cognitive Offloading. Trends in Cognitive Sciences, 20(9), 676–688.
• Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S.R., et al. (2023). Towards Understanding Sycophancy in Language Models. arXiv:2310.13548 (Anthropic).
• Turkle, S. (2025). Reclaiming Conversation in the Age of AI. After Babel, August 26, 2025.

‍