Document Q&A with Citations: Implementation Guide

The difference between a basic RAG system and one that produces reliable citations is verification: you must check that the LLM actually used the passage it cited, rather than hallucinating the reference or attaching a citation to a claim the passage doesn't support.

How Document Q&A with Citations Works

The process follows five sequential steps:

1. Index your documents — Chunk documents into passages (typically 200-500 tokens), generate embeddings for each chunk, and store them in a vector database alongside metadata (document title, page number, section heading).

1. Retrieve relevant passages — When a user asks a question, embed the query and retrieve the top 5-10 most similar chunks based on cosine similarity.

1. Generate an answer with citation markers — Pass the retrieved chunks and the question to an LLM with a system prompt that requires inline citations in a specific format, such as [1], [2], or [source_id].

1. Return citations with metadata — For each citation marker in the answer, return the corresponding passage text, document name, and location (page number or URL).

1. Verify citation accuracy — Run a verification step that checks whether each cited passage actually supports the claim it's attached to, either through prompt-based verification or a separate entailment model.

The verification step is what separates a system that shows citations from one that produces trustworthy ones. Without it, LLMs will frequently cite passages that don't support the claim, cite the wrong passage, or invent citation numbers that don't correspond to any retrieved document.

Retrieval Approaches for Document Q&A

You have three viable retrieval strategies, each with different accuracy and complexity trade-offs.

Dense retrieval works for most general knowledge questions but struggles when users ask about specific product names, error codes, or technical terms that appear infrequently in your corpus. Hybrid retrieval solves this by adding a BM25 sparse retrieval layer that performs exact keyword matching, then merging the results using reciprocal rank fusion. Reranking adds a second-stage model that scores each retrieved passage against the query, reordering them by true relevance rather than embedding similarity.

For production systems handling technical documentation or legal contracts, hybrid retrieval is the baseline. Reranking is worth the added latency when precision matters more than speed, such as compliance Q&A or medical documentation.

Prompt Engineering for Citation Generation

The LLM must be instructed to cite sources in a consistent format and to only make claims it can support with the provided passages. A working system prompt includes:

• Explicit citation format — Specify exactly how citations should appear, such as "Use [1], [2], [3] immediately after each claim, where the number corresponds to the passage index."

• Prohibition against unsupported claims — Instruct the model: "If the provided passages do not contain information to answer the question, state that explicitly. Do not use outside knowledge."

• Passage numbering in context — When you pass retrieved chunks to the LLM, number them clearly in the prompt: "Passage [1]: ...", "Passage [2]: ...", so the model can reference them unambiguously.

Even with careful prompting, LLMs will occasionally cite the wrong passage or insert a citation number that doesn't exist. This happens more frequently with longer answers or when the model synthesizes information across multiple passages. The solution is verification, not better prompting.

Verification: Checking Citation Accuracy

After the LLM generates an answer with citations, you must verify that each citation is valid. There are two methods:

Prompt-based verification — For each claim and its cited passage, send a follow-up prompt to the LLM: "Does the following passage support this claim? Passage: [text]. Claim: [claim text]. Answer yes or no." This works but is slow and prone to false positives, as the same model that generated the citation will often confirm it even when the support is weak.

Entailment model verification — Use a natural language inference (NLI) model fine-tuned for entailment tasks. These models take a premise (the cited passage) and a hypothesis (the claim) and classify the relationship as entailment, contradiction, or neutral. Models like DeBERTa-v3-base fine-tuned on MNLI or ANLI datasets perform this task reliably. If the model returns "neutral" or "contradiction," flag the citation as unsupported.

Entailment models add 50-100ms per citation but catch errors that prompt-based methods miss. For systems where citation accuracy is non-negotiable, this is the correct approach.

Handling Multi-Document Citations

When an answer synthesizes information from multiple documents, you must decide whether to cite each document separately or combine them under a single citation. The clearer approach is separate citations: "The system supports OAuth 2.0 [1] and SAML authentication [2]." This makes it obvious which document supports which claim.

The alternative is to list all sources at the end of the answer, but this reduces traceability. Users cannot easily determine which document supports a specific claim without reading all cited sources. Inline citations per claim are more useful.

Common Implementation Mistakes

Three errors cause most citation failures:

Retrieving too few passages — If you only retrieve 3-5 chunks, the LLM often lacks the context to answer fully and will either refuse to answer or hallucinate information. Retrieve 8-12 passages and let the LLM select which ones to cite. The added context improves answer quality more than it increases hallucination risk, provided your verification step is in place.

Inconsistent citation formatting — If your prompt says "cite sources as [1]" but the LLM sometimes outputs "(Source 1)" or "[Source 1]", your citation extraction logic will fail. Enforce formatting by parsing the output and rejecting malformed citations, then regenerating the answer.

No fallback for unanswerable questions — When the retrieved passages don't contain the answer, the system should say so explicitly: "The provided documents do not contain information about X." Without this, the LLM will attempt to answer anyway, often producing an unsupported claim with a fabricated citation.

What Citation Accuracy Looks Like in Practice

A well-built document Q&A system with citations achieves 85-95% citation accuracy, meaning that 85-95% of cited passages actually support the claims they're attached to. The remaining 5-15% are edge cases where the passage provides partial support or the claim is a reasonable inference but not explicitly stated.

You measure this by sampling 100 question-answer pairs, manually checking each citation, and calculating the percentage of valid citations. If accuracy falls below 80%, the issue is usually retrieval quality (the right passage wasn't retrieved) or verification gaps (unsupported citations aren't being caught).

Frequently Asked Questions

What is the difference between document Q&A with citations and standard RAG?

Document Q&A with citations extends retrieval-augmented generation by adding structured citation tracking and verification. Standard RAG retrieves passages and generates an answer but doesn't link specific claims to specific sources or verify that the LLM used the retrieved passages correctly. Citation systems add inline reference markers, return the cited passages with metadata, and run a verification step to ensure each citation is accurate. This makes the system auditable and allows users to check the source of each claim.

How do you prevent LLMs from hallucinating citations?

You prevent hallucinated citations through prompt engineering and post-generation verification. The prompt must explicitly prohibit unsupported claims and specify the exact citation format. After generation, run each claim and its cited passage through an entailment model to verify that the passage actually supports the claim. If the entailment check fails, flag the citation as unsupported and either remove it or regenerate the answer. Prompt engineering alone is insufficient because LLMs will still occasionally cite passages incorrectly, especially in long answers or when synthesizing information.

What retrieval method works best for technical documentation?

Hybrid retrieval combining dense vector search and BM25 sparse retrieval works best for technical documentation. Technical queries often include specific product names, error codes, version numbers, or API endpoints that appear infrequently in the corpus, which dense retrieval alone will miss. BM25 provides exact keyword matching for these terms, while dense retrieval handles semantic similarity for conceptual questions. Merge the results using reciprocal rank fusion, which combines rankings from both methods without requiring manual weight tuning. For higher precision, add a reranking layer using a cross-encoder model, though this increases query latency by 100-200ms.

Build Citation Systems That Stay Accurate

Document Q&A with citations requires more than retrieval and generation — it requires verification infrastructure that checks every citation before returning it to the user. The systems that work in production are the ones that assume the LLM will make mistakes and build guardrails accordingly.

If you're building document Q&A for a product that needs citation accuracy you can defend, WeaveAI builds RAG systems with built-in verification layers that catch unsupported citations before users see them. We handle retrieval tuning, citation formatting, and entailment-based verification so your Q&A system produces answers you can trust. See how we build reliable document Q&A at weaveai.dev/products/seo.