How to verify AI-written docs against your source code

A language model writes from patterns, so it can produce a plausible command, option or environment variable that your project does not have. Readers copy those lines into a terminal, and an invented flag costs them time and trust.

A better prompt does not fix this. Checking every concrete statement against the source does.

• Commands and subcommands exist in your CLI.
• Flags and options exist on the command they are shown with.
• Environment variables and configuration keys are read somewhere in the code.
• Install commands name your real package.
• File paths, function names and other identifiers exist in the repository.
• Links point to pages that exist.
• Code citations point to lines that exist.

• Search the source for every flag, variable and identifier the docs mention. A match proves it exists. No match is a defect.
• Compare your CLI's own help output with the documented options, on a machine where you are happy to run it.
• Run a link checker on every release.
• Re-read a page whenever the code it describes changes. Note which source files each page covers so you know what to revisit.

DocUp reads your GitHub repository as files and extracts facts from the code: CLI commands and their options, environment variable names, install commands and exported symbols. It does not install, build or execute anything, on any plan.

Pages are written with markers that point at the source lines each statement rests on. A checker then compares the page with the extracted facts, and pages with defects go through a repair step. It looks for:

• commands and flags the code does not define
• identifiers, file paths and package names the repository does not contain
• install commands that name the wrong package
• links that do not resolve
• line references that do not exist or cover too many lines
• malformed Markdown and empty marketing words such as powerful, seamless and robust

A static check proves that something exists in the source. It cannot prove that an explanation is clear or complete, or that behaviour described in prose matches what the code does at run time. Read the draft before you publish it. How much can be checked also depends on what DocUp can extract from your repository.

• Every command in the docs runs as written.
• Every flag appears in the source or the CLI help.
• Every environment variable is read by the code.
• Every link opens.
• Someone who did not write the code has read the page once.