Foldline is a made-up API company that turns messy PDFs into structured JSON. Its docs live on docs.foldline.example, in four versions and two languages, next to a marketing site and a blog.
What was bought
- Product
- Standard audit, €1,160 ex-VAT
- Cap
- up to 500 URLs
- Call
- 45 minutes, booked in a CET/CEST window
- Turnaround
- 10 working days from a complete brief and access
- Includes
- AI-crawler check, hreflang check
- Fed in
- crawl of the docs subdomain and the marketing site up to the cap, the sitemap, robots.txt, read access to Search Console
This is an excerpt. A real Standard audit has more verdicts than we show here, and covers your actual URLs.
Fix first
Fix first
Every docs page exists once per API version, and none of them says which one is the real one.
/v2/webhooks/retries/, /v3/webhooks/retries/ and /v4/webhooks/retries/ carry near-identical text. Each canonical points at itself. Nothing points at /latest/.
High. Search engines pick a version for you, and the pick is often an old one. People land in docs for an API version they can no longer use.
Low to medium. A template change: canonical to the latest equivalent, plus a visible “you're reading v2” banner on old versions. The oldest version may be better off with noindex. That's a decision for Foldline.
Fix first
The German docs point hreflang at pages that don't exist.
<link rel="alternate" hreflang="de" href="https://docs.foldline.example/de/guides/auth/">That URL returns a 404. Only the quickstart and the auth guide exist in German, but the tags are on every English page.
High. When a hreflang set is full of dead ends, search engines are likely to ignore it. German searchers then land on English pages.
Low. Output the tag only where the translated page exists.
Fix first
The docs search results are crawlable, and they're thin.
/search?q=webhook is linked from the header search box, returns a 200, has no noindex and uses the title “Search | Foldline Docs” every time.
Medium to high. Crawlers can spend their time on endless search URLs instead of the guides.
Low. Noindex the results template and disallow /search in robots.txt.
Worth fixing
Worth fixing
The quickstart is an orphan.
The blog never links to /v4/quickstart/. The only route in is a version dropdown that needs JavaScript.
Medium. It's your most useful page and it's hard to reach, for people and for crawlers.
Low. Add plain links from the homepage, the pricing page and the three most relevant blog posts.
Worth fixing
Titles follow one pattern, and the pattern says nothing.
Pages under /guides/ are titled “Guide | Foldline Docs”, with the topic missing.
Medium. Same title across many pages makes them hard to tell apart in results.
Low. Template change: topic first, product name last.
Worth fixing
Code-sample pages have breadcrumbs markup and nothing else.
JSON-LD found: BreadcrumbList only. Nothing describes the pages as documentation or articles.
Low to medium. Extra markup helps machines read the page. It doesn't make anything rank.
Medium. A template job, once.
The robots.txt rule nobody remembers adding
The robots.txt carries this block, above a comment reading “block AI scrapers”:
robots.txt excerpt
User-agent: GPTBot
Disallow: /Plain version: GPTBot has been told not to read any of the docs. That may be what Foldline wants. It may be a blanket rule from an earlier year that nobody revisited, and nobody ever decided on it for the docs. Whether to let AI bots read your docs is a business decision, not an SEO one. We'd put the question to the company and write whichever answer they choose into the file.
No schema field or batch of pages can force an AI assistant to recommend you. We work to improve your chances of being cited, and we show you what we checked.
What to give the developer first
Ranked. Do them in this order.
- Search results: noindex plus a robots.txt disallow.Effort: low
- hreflang: output only where the translated page exists.Effort: low
- Canonical to /latest/ on old versions, with a version banner.Effort: low to medium
- Plain links to the quickstart from three places.Effort: low
- Real titles on /guides/ pages.Effort: low
- Documentation markup on code-sample pages.Effort: medium
- robots.txt: decide on GPTBot, write the answer down.Effort: low, once someone decides
One round of questions, after delivery
Every Standard audit comes with one clarification round within 7 days of delivery. Here's what one looks like.
Client asks
We block GPTBot on purpose for the API reference but want it on the guides. Can you say what the rules should look like?
We reply
Yes. Disallow /reference/ for GPTBot and leave /guides/ open. Put each rule under its own User-agent line, then check the file at yoursite.com/robots.txt. One caution: robots rules are a request, and well-behaved bots follow them. Nothing in the file enforces them.
Clarifications, evidence, “show me where”, and “this fact is wrong” are inside the round. New URLs and new scope are a new order.
Same kind of report, on your site
Standard audit: €1,160, up to 500 URLs, a 45-minute call, 10 working days from a complete brief and access.