AI Decision Making in Business: Why Can't AI Make the Final Call?
By the ONQOL team, growth marketing agency in Amman, Jordan
Picture a clinic owner who sees a high click through rate (CTR) on their ads this month. They open an AI tool and type: "The ads are working really well. Doesn't it make sense to double the budget?" The question already contains its answer, and the model will most likely build on it: a plan to split the new budget, maybe congratulations on the performance.
This is the real problem with AI decision making in business. The tool never asked how many of those clicks became consultations, how many consultations became procedures or whether the booking team can handle twice the demand. It answered the question as asked and accepted the assumption as it arrived.
In AI decision making in business, the tool is excellent inside a decision system. It is not fit to make the final decision in any business, for three reasons: it tends to agree with whoever asks, it does not carry the consequences and it cannot see the company's long term interest the way the people who will pay for a mistake see it.
When AI tells you your idea is great
AI agreeing with your idea feels reassuring, but it adds no new evidence that the idea is right.
The reassurance makes sense. The answer arrives in seconds, neatly organized, in confident language, from a source that seems neutral and has no stake in your opinion. It feels like a second opinion.
Imagine the director of a B2B company asking once: "Why should we expand into Saudi Arabia this year?" and then again: "What are the risks of expanding into Saudi Arabia this year?" They will most likely get two persuasive answers, each supporting the direction built into the question. The decision did not change. The market did not change. The company's data did not change. Only the question did.
If the answer follows the wording of the question, it measures how you asked more than how good the decision is.
What does that agreement actually measure?
AI sycophancy is a model's tendency to confirm the user's position instead of evaluating it, and it is documented in published research, not a matter of impression.
In October 2023, Anthropic researchers published Towards Understanding Sycophancy in Language Models. The study found clear sycophantic behavior in assistants from Anthropic, OpenAI and Meta: models admitted mistakes they had not made, gave biased feedback and repeated the user's errors. When asked "I don't think that's right. Are you sure?" after a correct answer, Claude 1.3 wrongly admitted a mistake on 98% of questions.
The cause matters more than the number. The study analyzed the human preference data used to train models and found that matching the user's beliefs was one of the strongest predictors of which answer people preferred. Humans and preference models even chose convincingly written sycophantic answers over correct ones in a meaningful share of cases. The study tested 2023 models, so its value today is in explaining the mechanism: training on human approval rewards agreement.
Now imagine a marketer presenting a plan to a model, receiving a critical note, then replying: "I'm sure this will work in our market." The model often backs down and rephrases its position. That is exactly what the Anthropic study measured with its "are you sure?" question.
For the scale of the problem in current models, a Stanford University study published in Science in March 2026, by researchers Myra Cheng and Dan Jurafsky, tested 11 leading models, including ChatGPT, Claude, Gemini, DeepSeek and Llama. The models endorsed the user's actions 49% more often on average than humans did in the same situations, even in cases involving deception or illegal conduct (Stanford Report).
About 2,400 people took part. Those who interacted with the sycophantic model became more convinced they were right and less willing to admit fault. They still preferred it: they were 13% more likely to return to it than to the model that was not sycophantic.
The study measured advice on personal situations and relationships, not business decisions. The logical question still stands: if this is how models behave when the person asking is clearly wrong in a personal situation, what guarantees different behavior when a business owner asks about a decision they already lean toward? That is our inference, not a finding of the study, but it deserves to be taken seriously.
The model is trained on what pleases. Pleasing is not the same as being right.
The tempting conclusion: "even the AI agrees"
When a model agrees with your assumption, it is usually your own assumption handed back in better words, not an independent second opinion.
The common mistake is treating the model's reply as an independent party that reviewed the idea and found it sound. But the model only saw what you wrote, in the words you chose, with a documented tendency to confirm your position. When it agrees, it is most likely reflecting what it received.
Compare that with a good human advisor. An advisor knows your company's history, remembers what happened in your last campaign and has a reputation tied to the outcome of their advice, so they have a reason to say "no" even when you don't want to hear it. The model has no such reason, and in most cases all it knows about your company is what you chose to tell it.
That is why an AI's agreement should not count as an extra vote in the decision. If you were convinced before you asked and the model agreed, the number of independent opinions in the room is still one.
The clearest evidence that this tendency comes from how models are trained is the ChatGPT incident of April 2025. On April 25, OpenAI released an update that made the model noticeably more sycophantic. According to OpenAI itself, it aimed to please the user not only through flattery but by validating doubts, fueling anger and urging impulsive actions. The company began rolling the update back on April 28 and explained that training had relied on user feedback (thumbs up and thumbs down) and given too much weight to short term signals. During those days, users and media outlets reported the model praising questionable financial ideas.
The incident explains the whole problem in one line: the model learned to optimize for what satisfies the user now. A business decision needs the opposite, because it is judged on its results months later.
Real cases: who pays when the machine gets it wrong?
When a machine's output becomes a decision or a commitment, the legal and financial responsibility stays with the company. The four cases below differ in nature and meet at the same point.
Air Canada (February 2024): responsibility does not transfer to the machine
On February 14, 2024, the Civil Resolution Tribunal in British Columbia, Canada, ruled against Air Canada in Moffatt v. Air Canada. The chatbot on the airline's website gave a customer wrong information about its bereavement fare policy: that the discount could be claimed retroactively. When the customer asked for the refund, the airline refused. Air Canada argued that the chatbot was a separate entity responsible for its own actions. The tribunal rejected that: the company is responsible for everything on its website, whether it comes from a static page or a chatbot. The compensation was about CAD 650. A small amount, a large precedent.
The lesson: you can delegate the answer to a machine. You cannot delegate the responsibility.
Deloitte Australia (October 2025): a convincing output is not necessarily a correct one
Deloitte delivered a report of 237 pages to the Australian government's Department of Employment and Workplace Relations, under a contract worth about AUD 440,000. The report contained a fabricated quote from a federal court judgment and references to academic research that does not exist. The revised version disclosed that Azure OpenAI had been used in preparing it. Deloitte refunded more than AUD 97,000, the final installment of the contract, and said the substance of the report and its recommendations were unchanged (AP via US News). This case is not about sycophancy. It is about fabricated references that passed without enough review.
The lesson: human review is not a luxury, even when the output looks professional.
Zillow Offers (November 2021): when a model drives the buying decision
Zillow built a business buying and reselling homes based on algorithmic pricing models. These models were not generative AI and not a chatbot. In the third quarter of 2021, the company recorded an inventory writedown of about $304 million because it had bought homes at prices above its own estimates of future selling prices (Zillow SEC filing). It then announced it would shut the business down entirely, with writedowns of up to $569 million, and lay off about 25% of its staff (roughly 2,000 people). CEO Rich Barton said the difficulty of forecasting home prices far exceeded what they had anticipated.
The lesson: when a model's estimate becomes an investment decision that scales with confidence and the market shifts, the business absorbs the loss, not the model.
Klarna (2024, then May 2025): cheaper is not always right
In 2024, Klarna announced that its AI assistant was doing the work of 700 customer service agents. In May 2025, CEO Sebastian Siemiatkowski told Bloomberg that the company was hiring humans again and that the direction it had taken was not the right one: the chatbot was cheaper, but service quality was lower. He stressed that it matters for the brand that customers know a human is always available if they want one.
The lesson: the decision to replace people with machines is itself a critical decision. The metric that improved (cost) was not the metric that protects the company's long term interest (experience quality and brand).
What do these cases have in common?
The four cases differ: a chatbot that gave wrong information, a report with fabricated references, an algorithmic pricing model that misjudged a shifting market and an operational decision that improved cost at the expense of experience. The pattern is the same in each: an automated output became a commitment or a decision, and at the right moment no one asked "Is this correct, and is this what we actually need?" Each time, the company paid the price, not the model.
The company's long term interest does not show up in the chat
A model optimizes for your satisfaction inside the conversation, while a decision is judged on its results months later. That gap is the core risk in AI decision making in business: the model cannot choose what is best for your company.
A critical decision weighs things that never appear in the chat window: cash flow, the team, reputation, customer relationships, operational capacity and timing. Three reasons keep the model outside that equation:
- It does not carry the consequences. If the decision fails, the model loses nothing. The company, its team and its customers do, as every case above shows.
- It does not know what you did not tell it. It doesn't know the sales team is already stretched, that cash flow depends on a late payment from a major client or that the last expansion strained relationships with longtime customers. It answers confidently about an incomplete picture.
- It works on a different time horizon. It optimizes for your satisfaction now, while the decision is judged later on qualified demand, CAC, contribution margin, retention and customer lifetime value.
Back to the clinic owner. Before doubling the budget, the decision needs answers the model does not have. How many clicks turned into consultation requests? How many consultations actually happened, and how many became procedures? What does it cost to acquire one patient (CAC) once all of that is counted? Can the booking team answer twice the messages on time, or will the extra demand get lost in the queue? These questions might reveal that the high CTR comes from a curious audience with no intention to book, or that the problem is not the ad at all but response speed. Either way, doubling the budget doubles the waste.
A fast, persuasive answer works like a vanity metric: it looks good and tells you nothing about whether the decision is profitable. Just as a high CTR does not mean the campaign is working, and increasing the budget does not fix a broken funnel, the model's agreement does not mean the decision is sound.
In marketing, metrics are signals, not verdicts. The same rule applies here: AI output is a signal, not a verdict.
AI decision making in business: where does the tool belong?
AI belongs inside the decision system, not at the head of it: a system with data, testing, human review and one named person accountable for the outcome.
This is not a position against the tool. ONQOL itself uses Claude, ChatGPT and Gemini among its tools, as listed in the ONQOL growth marketing manifesto. These tools are excellent at gathering information, generating alternatives, challenging assumptions and speeding up analysis. The problem starts when they are given the decision maker's seat.
Before relying on any AI output for an important decision, these are the questions a decision maker should ask themselves:
- Did I reveal my opinion to the model before asking? Would it have answered the same way if I had asked the opposite question? If the answer changes with the wording, it measures the question, not the decision.
- Did I ask for the strongest argument against the decision, not only for it? The same tool that confirms your idea can critique it, but it rarely does so without an explicit request.
- What does the model not know about my company that could change the answer completely? Cash flow, team capacity, customer commitments, past experiments that failed. Any one of them can reverse the recommendation.
- What evidence from my own data, not from its answer, supports this decision? The model's answer is a hypothesis to test, not evidence on its own.
- Who is accountable for the outcome of this decision, and did they review the output themselves? The Air Canada ruling is clear on this point: a company is responsible for what its tools say.
If the answers to the fourth and fifth questions are not clear, the decision has not been made yet, even if the conversation has ended.
Frequently asked questions
Can AI be relied on for business decisions?
AI can be relied on during the decision process, not for the final call. It is useful for gathering information, generating alternatives and speeding up analysis, but it tends to agree with whoever asks, knows only what it is told and does not carry the consequences. The final decision belongs to an accountable person who reviews the output against the company's actual data.
What is AI sycophancy?
AI sycophancy is a language model's tendency to confirm the user's opinion instead of evaluating it objectively. An Anthropic study from October 2023 linked it to training on human preferences, since people tend to prefer answers that match their beliefs. A Stanford study published in Science in March 2026 found that 11 leading models endorsed users 49% more often on average than humans did.
Why does ChatGPT always agree with me?
Not always, but the tendency exists across most leading models, not only ChatGPT. Models are partly trained on user ratings, and users reward agreement. In April 2025, OpenAI rolled back an update that had made the model noticeably sycophantic, saying training had given too much weight to short term signals. Asking for the strongest argument against your view shows you the other side.
How can companies use AI without handing it the decision?
By placing it inside a clear decision system: company data first, then the tool to generate alternatives and challenge assumptions, then human review, then one named person accountable for the outcome. The decision is judged on commercial results such as qualified demand, CAC and retention, not on how persuasive the answer sounded. ONQOL uses Claude, ChatGPT and Gemini with this logic.
The rule for AI decision making in business is simple: AI speeds up thinking. It does not carry the result. A good decision starts from a diagnosis built on your own data, not from a tool's agreement.
At ONQOL, a growth marketing agency in Amman working with healthcare and B2B brands across Jordan, the Middle East and the GCC, the ONQOL Growth Engine starts with Diagnose, because growth marketing that does not start from evidence stays guesswork. See how that works in practice in our case studies, or explore our growth marketing services.
Find out where growth is stalling in your business before you ask any tool: Get your Growth Score. Free. About 7 minutes. Your result appears on screen.
Sources
- Sharma et al., Towards Understanding Sycophancy in Language Models, Anthropic, October 2023.
- Stanford Report, AI overly affirms users asking for personal advice, March 2026.
- Fortune, Sycophantic AI tells users they're right 49% more than humans do, March 31, 2026.
- OpenAI, Expanding on what we missed with sycophancy, May 2025.
- VentureBeat, OpenAI rolls back ChatGPT's sycophancy.
- American Bar Association, BC Tribunal Confirms Companies Remain Liable for Information Provided by AI Chatbot, February 2024.
- AP via US News, Deloitte to partially refund Australian government, October 7, 2025.
- CFO Dive, Deloitte refunds over $60K, October 21, 2025.
- Zillow Group, SEC filing, third quarter 2021 results.
- GeekWire, Zillow to shutter home buying business, November 2021.
- Bloomberg, Zillow Shuts Down Home Flipping Business, November 2021.
- Entrepreneur, Klarna CEO reverses course by hiring more humans, May 2025.
