OSINT and AI, validate using CRAAP test (& II)

PLAUSIBLE REAL LIFE EXAMPLE

Scenario

An analyst is tasked with assessing the credibility of claims about a sudden shift in China’s foreign investment policy in Africa. The analyst uses an AI tool, ChatGPT or a similar Long Language Model (LLM), to gather a summary and relevant sources, but needs to quickly validate the AI-generated information before using it in a policy briefing.

Step-by-Step CRAAP Test Workflow

1. Currency
   • AI Output: The AI summary states, “In April 2025, China announced a freeze on new infrastructure investments in Sub-Saharan Africa.”
   • Actions:
     • Check the date of the AI’s knowledge cutoff (e.g., October 2023 for many LLMs).
     • Search for up-to-date news (Google News, Reuters, official Chinese government sites) to confirm if any such announcement was made in April 2025.
     • Result: No reputable sources confirm this event; the most recent official statements date back to 2024.
   • Conclusion: The AI’s information is likely outdated or fabricated.
2. Relevance
   • AI Output: The summary includes background on China’s Belt and Road Initiative, but only one paragraph relates to the alleged investment freeze.
   • Actions:
     • Assess whether the content directly addresses the analyst’s requirement: “Has China changed its investment policy in Africa in 2025?”
     • Result: Most of the AI’s output is general context, not specific to the current policy shift.
   • Conclusion: Only a small portion of the AI’s output is relevant.
3. Authority
   • AI Output: The AI cites “reports from the China-Africa Research Initiative at Johns Hopkins University” and “statements from China’s Ministry of Commerce.”
   • Actions:
     • Check if these reports exist and are recent (visit the official China-Africa Research Initiative website, search for press releases from China’s Ministry of Commerce).
     • Result: No such reports or statements found for 2025; cited sources either do not exist or are outdated.
   • Conclusion: The AI’s citations lack authority and cannot be verified.
4. Accuracy
   • AI Output: The AI claims, “Chinese investments in Africa dropped by 40% in the first quarter of 2025.”
   • Actions:
     • Cross-check this statistic with data from the World Bank, IMF, or African Development Bank.
     • Look for corroborating reports from reputable financial news outlets (e.g., Financial Times, Bloomberg).
     • Result: No supporting data for a 40% drop; available statistics show only minor fluctuations.
   • Conclusion: The AI’s claim is inaccurate and unsupported.
5. Purpose
   • AI Output: The AI presents information in a neutral tone but includes dramatic language (“a seismic shift in China’s Africa policy”).
   • Actions:
     • Analyze the language for bias or sensationalism.
     • Consider that AI-generated content may reflect patterns from its training data, amplifying dramatic narratives even when unwarranted.
     • Result: The dramatic framing is not supported by evidence.
   • Conclusion: The purpose may be to attract attention rather than inform accurately.

CONCLUSIONS

This workflow demonstrates how the CRAAP test can be systematically applied to AI-generated information in real world intelligence and policy contexts, helping analysts avoid the pitfalls of bias, hallucination, and disinformation or misinformation.

• Always verify AI outputs against up-to-date, authoritative external sources before including them in intelligence products.
• Use the CRAAP test as a rapid checklist to filter out unreliable or misleading AI-generated content, especially when working under time constraints or with limited resources.
• Document your validation steps to support transparency and accountability in your analysis.