What Weak AI Knows - and Why it Matters for Decision Making

Why AI’s Confidence Is No Substitute for Understanding

May 26, 2025

This is the first post in the series, Exploring AI and Its Intersection with Human Decision Making. It pulls inspiration from two previous Psychology Today posts: one in 2023 and one in 2025.

From recommendation algorithms to self-driving cars, artificial intelligence (AI) is increasingly shaping the choices we make. It assists in diagnosing diseases, enhances business strategies, and even generates music and artwork. As AI becomes more embedded in everyday life, its influence on human decision making will only grow.

This series will examine how AI intersects with human judgment. While I have several posts planned, the series won’t follow a strict chronological order. Instead, each post will focus on a specific topic that fits within the broader theme. Each entry will be clearly identified, with links to prior posts included at the bottom. That said, not every upcoming post will be part of this series—I’ll continue writing on other topics as well and will add to this series periodically.

This first post lays the foundation by asking a simple but but question: What does AI actually know? By answering that, we can begin to explore its implications for how we make decisions—both individually and as a society.

A Brief Primer on Types of AI

Artificial intelligence is often discussed as if it’s a singular technology, but in reality, it spans a wide range of systems with different capabilities. While all AI systems rely on pattern recognition and data processing, their complexity and flexibility vary significantly. One way to classify them is into two broad categories: weak (narrow) AI and strong (general) AI (Sahota, 2024).

Weak (Narrow) AI refers to systems designed for specific tasks that operate within predefined constraints. These include Google Maps (route optimization), spam filters (detecting unwanted emails), and facial recognition software (identifying individuals in images). These systems excel within their domain but cannot generalize knowledge beyond it.
Strong (General) AI refers to hypothetical systems with human-like cognitive abilities—able to reason, learn flexibly, and apply knowledge across multiple domains. We do not yet have general AI. While some AI models display impressive problem-solving capabilities, they remain task-specific rather than possessing true understanding.

The specific type of AI that has captured the lion’s share of attention since 2022 is Generative AI—systems such as ChatGPT, Claude, DALL-E, and AlphaFold. These models stand out because they generate new text, images, or predictions rather than merely retrieving or classifying data. While generative AI is more complex than both other AI tools, the key difference lies in scale and adaptability. Unlike narrowly focused AI models, generative AI systems—especially large language models (LLMs)—have massive parameter counts and produce outputs that are far less constrained by predefined rules. This flexibility allows them to respond to a wide variety of prompts, making them appear more general-purpose, even though they remain fundamentally a form of weak AI1.

What Weak AI Knows

Despite its impressive ability to generate human-like responses, weak AI doesn’t actually know anything. Whether it’s Google Maps suggesting a route or ChatGPT drafting a letter, AI isn’t recalling stored knowledge the way a person does—it’s simply predicting the most statistically probable response based on its training data and constraints.

For highly constrained AI systems, like Google Maps, this is relatively easy to recognize. Google Maps doesn’t know where traffic is heavy, nor does it understand the fastest way to your destination. It processes real-time traffic data, compares it to historical patterns, and predicts which route is likely to be optimal. Because it operates within a narrow, well-defined domain, its predictions tend to be highly reliable.

But as AI systems become more flexible, their ability to make errors increases. Large language models (LLMs) like ChatGPT, Claude, and Gemini generate responses in a far less constrained manner. Instead of selecting from a limited, structured dataset (as Google Maps does), they predict text based on statistical patterns across enormous amounts of training datasets. Their goal isn’t accuracy—it’s to produce plausible, coherent responses that fit the input they receive.

Reinforcement learning with human feedback (RLHF), a common method used to refine these models, exacerbates this tendency. Previous, less sophisticated versions of an LLM were more prone to produce outputs that were either (1) nonsensical—responses that were incoherent and clearly wrong—or (2) avoidant—failing to address the user’s query meaningfully2. These obvious errors made it easier to identify when the model was off track.

With RLHF, human reviewers reward responses that feel natural and convincing, but this doesn’t always ensure factual accuracy. You see, both the clearly wrong and the avoidant answers get flagged, teaching the LLM to minimize these types of responses. But those responses that sound good—they get rewarded, even if they aren’t fully accurate3. Over time, the model learns to prioritize reasonable-sounding answers—even when they deviate from the truth. The more fluent and articulate a model becomes, the easier it is for users to mistake its output for truth, especially when the falsehoods sound reasonable.

This is where the problem can arise. Unlike Google Maps, which has clear boundaries on what it can predict, LLMs operate across a much wider range of possible outputs—which means they have a much greater capacity for error. And because their responses sound human-like and confident, we often forget that they’re just sophisticated predictive tools performing the same kind of statistical pattern-matching as any other AI system4.

This is what makes generative AI so compelling—and so deceptive. The more sophisticated it becomes, the harder it is to distinguish between genuine knowledge and convincing bullshit (Zhou et al., 2024). LLMs don’t check facts, evaluate the reliability of sources, or reason through problems. They’re designed to generate plausible responses, not truthful ones. That plausibility is often enough to make them sound like they know things, when, in reality, they’re highly sophisticated bullshitters.

Though we often use the term bullshit to describe deception or exaggeration, Frankfurt (2005) argued that bullshit is fundamentally a disregard for truth, which differentiates it from lying. A liar knows the truth but intentionally misrepresents it; a bullshitter, by contrast, shares information with little concern for whether it’s true or false. Frankfurt (2005) further explained:

Bullshit is unavoidable whenever circumstances require someone to talk without knowing what he is talking about. Thus the production of bullshit is stimulated whenever a person’s obligations or opportunities to speak about some topic are more excessive than his knowledge of the facts that are relevant to that topic. (p. 19)

This is exactly how AI operates. It generates outputs without knowing what it’s talking about. It doesn’t distinguish between accurate and inaccurate statements, making it capable of producing factually correct responses and convincing but completely false ones.

In terms of a simple analogy, we might think of weak AI as having access to a box of Legos. Less sophisticated forms of weak AI:

Have access to only a few Legos;
Those Legos are of very specific types; and
They were trained on a very limited set of building instructions, restricting what they can create.

But more advanced generative AI has access to a vast number of Legos—of many different shapes and sizes—along with a massive, diverse set of building instructions. This gives them far more flexibility in how they assemble pieces.

So, when a user inputs a query, a generative AI system predicts—based on the instructions it was trained on—how to put the Legos together in a way that provides the user with what they’re looking for. If the user asks for a castle, the AI predicts which Lego pieces to use and how to arrange them in a way that resembles other castles it has seen. If the user asks for a boat, the AI follows a different pattern, assembling pieces in a way that statistically aligns with what a boat should look like.

As long as it predicts correctly which Legos to use and in what order, it produces something that appears coherent. But here’s the catch:

The AI isn’t actually thinking before it responds.
It doesn’t know whether it’s building a structurally sound castle or a boat that floats.
It can’t verify whether what it produces matches reality—it’s just guessing based on patterns.

Just like an AI assembling Legos into a structure that looks right, LLMs assemble words into responses that seem reasonable—even when they aren’t. This is why generative AI is so powerful but can be unreliable. It constructs outputs that look and sound correct, but it does so without any comprehension of their accuracy or meaning.

Implications for Human Decision Making

AI is often framed as a revolutionary tool that will transform decision making. And in many ways, it already has. AI can rapidly process and synthesize vast amounts of information, identify patterns that humans might overlook, and assist in everything from diagnosing diseases to optimizing business strategies. These strengths make it a powerful decision-support tool. But that’s the key: support—not replacement.

And while these tools can enhance decision making, they can just as easily be co-opted to serve self-interests. Licon (2025) makes a compelling argument that in many contexts LLMs specifically will lead to “the democratization of bullshit.”

People will use LLMs to craft more persuasive resumes, more polished excuses, more palatable defenses. The same reputational games we’ve always played—now accelerated by predictive text.

In these situations, the outputs don’t need to be accurate—they just need to sound good. That can benefit users, at least in the short term, especially if they’re not caught.

But the problem arises when people mistake AI’s fluency for understanding, treating it as an authority rather than a predictive tool with built-in constraints and limitations. As AI becomes more deeply embedded in decision-making processes, overreliance on its outputs can lead to short-sighted or even dangerously incorrect conclusions.

For example, a doctor could misdiagnose a patient Sohn (2023)5, a lawyer could build a case on incorrect legal interpretations (Neumeister, 2023), or a business executive could approve a new product line based on inaccurate market analysis6—all because the outputs produced by the AI were plausible-sounding but inaccurate. In such cases, the stakes are too high to place blind trust in AI.

But AI can be enormously beneficial when used within clearly defined constraints. Highly structured systems like Google Maps work well because their predictions are limited to a specific set of inputs (real-time traffic, road conditions) and outputs (route recommendations). This clarity significantly reduces the risk of AI generating misleading or unfounded responses.

More flexible AI tools, like LLMs, can also be useful when applied appropriately. They can summarize large datasets, assist with brainstorming, and generate structured outputs that help decision makers work more efficiently. But unlike structured AI, LLMs generate responses without nearly as many constraints, meaning they can produce both valuable insights and complete fabrications with the same level of confidence.

This leads me to a few concluding points when it comes AI, bullshit, and human decision making:

AI can improve decision-making efficiency and effectiveness, but only if decision makers critically evaluate its outputs. The speed and scale of AI’s pattern recognition can be valuable, but decision makers must verify the accuracy and relevance of AI-generated information before acting on it.
We can easily mistake fluency for accuracy—AI will exploit this mistake. The easier something is to process, the more we tend to trust it (Fort & Shulman, 2024; Thompson et al., 2013). AI-generated outputs sound clear, structured, and authoritative, which makes them feel correct—even when they aren’t. Good decision makers remain cognizant of this.
AI’s flexibility increases the risk of misleading outputs. Highly constrained AI (like Google Maps) works well because it operates within strict decision-making parameters. More flexible AI (like LLMs) generates outputs without such guardrails, meaning decision makers must apply greater scrutiny when integrating AI into their choices.
The more ambiguous the decision, the more AI’s limitations matter. AI can generate plausible-sounding answers to open-ended, complex questions, but plausibility isn’t the same as accuracy. Decision makers need to be extra cautious in high-uncertainty environments (e.g., legal analysis, business forecasting, medical diagnostics).
Overreliance on AI can lead to worse decisions, not better ones. When people defer too much to AI (creating a bias toward accepting AI-based conclusions with little conscious thought), they risk making decisions based on predictions rather than understanding. AI-generated legal citations, medical diagnoses, and financial projections all require human oversight to avoid serious errors.
AI doesn’t evaluate trade-offs—humans do. Decision making often involves balancing competing priorities, ethical concerns, and long-term consequences. AI lacks judgment, values, and contextual awareness, so decision makers must ensure AI-generated insights align with human goals and constraints.
Critical thinking is the best safeguard against AI’s bullshitting tendencies. Decision makers who understand both the strengths and limits of AI will be far better equipped to use it effectively—leveraging its strengths without succumbing to its flaws.

Some of these conclusions will be expanded on in later posts, but suffice it to say that while AI offers tremendous benefits for human decision making and productivity, it also necessitates a need to ensure we are not merely offloading our critical thinking to a bullshitter.

The Eel-Filled Hovercraft

Discussion about this post