← Browse all articles
Leadership IQ · Research & insights
How To Check AI Advice Before You Act On It
A practical report on verifying sources and evaluating the evidence behind AI recommendations
Published
Add us as a preferred source on Google
Suppose you ask an AI assistant whether a coaching pilot worked. You give it a spreadsheet, and it comes back with a clear recommendation to expand the program. The teams whose supervisors received coaching escalated fewer customer problems, and the AI has calculated the difference correctly. It also supplies a few research citations that appear to support the recommendation.
You could have a useful result. But you could also have a comparison between teams that handle very different customers. Or the coached teams might already have had fewer escalations before the pilot began. The arithmetic could be right while the explanation for the difference is wrong.
Checking AI advice requires you to follow the recommendation back to the evidence. You need to know whether the source exists, what it actually establishes, and which additional assumptions the AI used to reach its conclusion. This report gives you a process for doing that, with prompts you can reuse and a hypothetical example you can work through.
Start by separating the recommendation from its supporting claims
An AI answer can mix facts, estimates, and interpretations in the same paragraph. Before checking every sentence, identify the claims that matter to the decision. If a claim were wrong, would you choose something different?
For the coaching pilot, the observed escalation rates matter. So does the assumption that coaching caused the difference. A general statement about the importance of leadership development adds little to that particular decision.
Prompt to expose the evidence behind a recommendation
Review this recommendation: [paste recommendation]. Identify the claims that the decision depends on. For each claim, state whether it is a supplied fact, an externally sourced finding, a calculation, an interpretation, or an assumption. Identify its source and the specific passage, table, or data fields that support it. Explain what remains unverified. If you cannot access a source, say so. Don’t invent a citation or treat your own earlier answer as evidence. Finish by identifying which unverified claim would most change the recommendation if it proved wrong.
Use the response as a list of things to check. An AI’s account of its sources can contain errors too. You still need to open the important sources and inspect the relevant evidence.
Four checks for consequential AI advice
| Check | What you need to establish |
|---|---|
| Source existence | The cited paper, report, document, or dataset is real and accessible enough to inspect. |
| Claim support | The source actually supports the factual statement attributed to it. |
| Reasoning | The recommendation follows from the evidence without an unsupported assumption doing the important work. |
| Company relevance | The finding applies well enough to your circumstances to inform the decision. |
The depth of the review should reflect what you’re about to do. You can tolerate more uncertainty in a small, reversible experiment than in a companywide change that will be expensive to undo. Start with the claims that could change the decision and work outward as needed.
Check that the source exists
A citation can look entirely plausible without referring to a real publication. It can include an author, journal, and title that sound right together, while no such paper exists.
In a 2023 study by William H. Walters and Esther Isabelle Wilder, researchers examined 636 references generated in literature reviews by GPT-3.5 and GPT-4. Eighteen percent of the GPT-4 references were fabricated. Those results concern the models and tasks tested at that time; they are not a current error rate for every AI tool. They illustrate why a polished reference deserves verification.
Open the original publisher’s page or an institutional repository. Check whether the title and authors match. A DOI can help you locate a scholarly work, but you need to follow it and confirm where it leads. For a company statistic, look for the original filing or report. For an internal figure, identify the actual file and version.
If you can’t find the source, mark the claim as unverified. Failure to locate it immediately doesn’t prove fabrication, but it does leave you without evidence you can rely on.
Prompt to locate and identify sources
For each externally sourced claim in this answer, provide the original source title, author or issuing organization, date, and URL or DOI. Identify the relevant section, table, or passage. State whether you accessed the full source, an abstract, a search result, or a secondary summary. If you cannot locate the original source, label the claim unverified. Do not fill missing citation details with plausible guesses.
You can use this prompt with an AI that has browsing or retrieval access. If it doesn’t have access, it should tell you what it can and cannot inspect. A statement that it “checked the research” is too vague to establish what happened.
Check that the source supports the claim
A real study can still be attached to advice it never tested. For instance, an association between two variables might be described as proof that changing one will change the other. Or a finding about a narrow task might become a claim about executive performance generally.
Consider the 2025 study led by Hao-Ping Lee on generative AI and critical thinking. It surveyed 319 knowledge workers. Greater confidence in AI was associated with less self-reported critical thinking. That supports a discussion about how people describe their effort when using AI. It doesn’t establish that AI permanently reduced their intelligence or that every use of AI weakened their reasoning.
When you inspect a source, identify who or what was studied and what the researchers actually measured. Pay particular attention to whether the evidence comes from an experiment, a survey, or an analysis of existing records. These approaches can answer different questions.
An abstract may be enough to reject an exaggerated claim. It may not provide enough detail to justify a consequential decision. If the relevant methods or results are unavailable, keep that limitation attached to the claim.
Prompt to compare a claim with its source
Here is the claim: [claim]. Here is the source material I have actually obtained: [text, table, or approved attachment]. Use only this material. Identify the strongest support for the claim and any wording that goes beyond the source. State who or what was studied, what was measured, and whether the source establishes a causal effect or an association. If the material does not answer one of those questions, write “not specified in the supplied material.” Give me the narrowest accurate version of the claim. Identify the exact passage or table I should inspect myself.
This is also useful when the source is an internal memo. If a meeting note says that several customers asked for faster responses, an AI shouldn’t turn that into a finding that most customers prefer a fully automated service.
Check the data before accepting the calculation
When advice rests on a spreadsheet, begin by establishing what each row and column represents. An escalation might mean a transfer to a specialist, a complaint to a manager, or any case requiring a second contact. Those definitions produce different measures.
The same applies to blanks. A blank field can mean zero, not yet recorded, or not applicable. Replacing every blank with zero may change the result in a way the original data doesn’t justify.
Keep an unchanged copy of the source file. Ask for proposed corrections before allowing the AI to alter it, and retain a record of any changes you approve. Removing a duplicate transaction can be appropriate. Removing a difficult case because it looks unusual can conceal exactly the problem you’re trying to understand.
Prompt to inspect a dataset before analysis
Describe what each row represents and what each field appears to measure in this dataset: [approved data or description]. Identify definitions you need me to confirm. Check for duplicate records, missing values, inconsistent units or time periods, and denominators that may not match the question. Do not change the source data. List proposed corrections separately and explain how each could affect the result. Do not silently replace missing values with zero, drop unusual records, or assume that two similarly named fields mean the same thing.
Then check at least one consequential calculation yourself, using the source records or a calculation you can inspect. Ask the AI to show which rows it included, the numerator and denominator, and the formula. An accurate formula applied to the wrong population still gives you the wrong answer to your question.
Use company information only in tools and accounts approved for that information. Removing names can reduce exposure, but combinations of details can still identify people or reveal confidential business information. Public or synthetic data is sufficient for practicing the process in this report.
Work through a coaching pilot example
The following numbers are invented for illustration. They are not findings from a Leadership IQ study or an actual client engagement.
A company examines customer cases handled during a pilot. Some supervisors received coaching, and others didn’t. Its summary shows:
| Group | Customer cases reviewed | Cases escalated | Escalation rate |
|---|---|---|---|
| Teams whose supervisors received coaching | 100 | 25 | 25% |
| Comparison teams | 100 | 35 | 35% |
An AI could accurately calculate that the coached teams’ escalation rate was 10 percentage points lower. The relative difference is about 28.6%, calculated as (35% minus 25%) divided by 35%. Those are two ways to describe the same comparison, and they should be labeled correctly.
But if the AI writes that coaching “reduced escalations by 28.6%,” it has introduced a causal conclusion that these figures alone don’t establish. The table compares groups. It doesn’t tell us what would have happened to the coached teams without coaching.
Before recommending expansion, you would want to know how the supervisors were selected, whether the teams handled comparable cases, and what their escalation rates looked like before the pilot. You would also want to check whether an escalation was defined consistently. Perhaps the coached supervisors were simply more willing to handle a problem themselves, with no improvement in resolution or customer experience.
The recommendation could still turn out to be sensible. You just haven’t established it from this table.
Examine the step from finding to recommendation
This is where many reviews stop too soon. The source exists and the number is correct, so the recommendation begins to feel verified. But a business recommendation usually adds assumptions about what will happen next.
For the coaching pilot, expanding the program assumes that the difference will persist, that the program can be delivered effectively to other supervisors, and that fewer escalations represent a useful outcome. Each assumption deserves attention if it affects the decision.
Prompt to examine the reasoning
Here are the verified findings: [findings and sources]. Here is the proposed recommendation: [recommendation]. Explain the reasoning that connects them. Identify any assumptions required to make that connection and any plausible alternative explanations for the findings. For each important assumption, state what information could confirm or weaken it. Do not describe an association as a causal effect. If the evidence supports only a narrower recommendation, explain what that would be and why.
You can now separate the claims in the hypothetical pilot:
| Claim | Status based on the supplied table | What would need checking |
|---|---|---|
| The coached teams escalated 25 of 100 cases. | Supported by the illustrative table. | Verify the actual records and definition of escalation. |
| Their escalation rate was 10 percentage points lower. | The arithmetic is correct for these figures. | Confirm comparable cases, periods, and counting rules. |
| Coaching caused the lower rate. | Not established by the table. | Examine selection, prior performance, case mix, and other changes. |
| Expanding coaching will improve customer outcomes. | Not established by the table. | Examine resolution quality, customer outcomes, and whether the results transfer. |
An appropriate next step might be to examine the baseline and case mix, then design a further pilot with a more informative comparison. The right choice depends on the cost of learning, the urgency of the decision, and what the company can safely reverse.
Check whether the advice applies to your situation
Even a strong study can leave important questions about your company unanswered. An intervention that helped experienced supervisors with one task may require adaptation for newly promoted managers with a different problem.
Be specific about the mismatch. “Our culture is different” can become a convenient reason to dismiss evidence. A more useful concern would be that the study’s participants received weekly practice and your proposed program provides one session with no opportunity to apply the skill.
Prompt to check relevance to your company
Here is what the source actually found: [verified finding and relevant study details]. Here is our situation: [population, task, constraints, and intended outcome]. Identify the similarities and differences that could affect whether the finding applies. Explain which differences have a reason to matter and which are merely descriptive. State what a limited local test could help us learn. Do not assume that a successful result elsewhere guarantees the same result here, and do not dismiss the evidence simply because our organization is different.
Where the advice depends on specialist judgment, involve the person responsible for that decision. An AI’s summary of a contract, financial model, or employment requirement doesn’t establish that you’ve completed the review those subjects need. The report’s evidence checks help you identify what to bring to that reviewer.
Use a second AI to locate disagreements
A second AI can point out a missing assumption or a citation that deserves attention. It can also repeat the first model’s mistake. Treat agreement as agreement between outputs, then inspect the evidence supporting it.
Give the second system the same source material and ask it to identify unsupported claims. If the two systems disagree about a number, return to the data and calculation. If they disagree about what a paper establishes, inspect the relevant passage. The question is which explanation the evidence supports.
Record the review and decide what to do
Keep a brief record of the important checks. You don’t need a dossier on every low-stakes suggestion, but a consequential recommendation should have a trail that another person can follow.
Evidence review record to copy
Proposed decision and deadline:
Claim that most affects the decision:
Original source and version or publication date:
Source location checked, including page, table, or data fields:
What the source actually supports:
Calculation independently checked, if relevant:
Assumptions or alternative explanations still unresolved:
Differences between the source setting and our situation:
Reviewer and date checked:
Decision, next check, and person responsible:
For the hypothetical coaching pilot, the record might say that the difference in escalation rates has been checked, while the reason for that difference remains unresolved. The company could then authorize a defined next investigation without describing the first pilot as proof of an effect.
When you accept AI advice, you should be able to explain what supports it and where your own judgment begins. If the recommendation changes after a source check, keep that correction in the record so the original claim doesn’t reappear in the next presentation.
About this report and its sources
This practical report draws on source-verification and data-review methods taught in Leadership IQ’s AI leadership curriculum. Its prompts organize a review; they do not guarantee correct outputs. The worked example uses invented data to demonstrate the difference between an observed comparison and a causal claim.
William H. Walters and Esther Isabelle Wilder, Fabrication and Errors in the Bibliographic Citations Generated by ChatGPT, Scientific Reports, 2023. Study of generated references; the historical error rates apply to the tested models and tasks. DOI: 10.1038/s41598-023-41032-5.
Hao-Ping Lee and colleagues, The Impact of Generative AI on Critical Thinking, CHI, 2025. Survey evidence about self-reported critical thinking; not an experiment demonstrating lasting cognitive decline. DOI: 10.1145/3706598.3713778.
NIST, Artificial Intelligence Risk Management Framework Generative Artificial Intelligence Profile, 2024. Further reading for organizations developing a broader approach to evaluating and managing generative AI risks. This framework is not a validation study of the prompts in this report.