AI Startup Website Checklist: Crawlers, Trust Pages and UK Rules

An AI startup website needs to do more than look sharp. It should be crawlable by the AI search tools your buyers use, explain clearly what the product does and doesn't do, answer security and privacy questions with real trust pages, ship with sensible security headers, and make only claims you can prove under UK advertising and consumer law.

AI companies face a particular mix of pressures. Buyers ask ChatGPT and Perplexity for recommendations, so AI visibility matters more than for most sectors. Enterprise pilots arrive with security questionnaires. And regulators are paying close attention to how AI products are marketed. This checklist covers the website side of all three.

The AI startup website checklist

1. Decide your AI crawler policy

AI crawlers don't all do the same job. Some collect training data, some fetch pages so an assistant can cite them in answers, and some fetch a page when a user asks about it. For an AI startup that wants to be recommended, blocking the wrong bot can quietly remove you from the answers your buyers read.

A sensible default for most AI companies:

• Allow AI search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, on marketing, docs and blog pages.
• Decide separately on training crawlers such as GPTBot, ClaudeBot and Google-Extended. That's a business choice. Blocking them doesn't remove you from AI search answers.
• Block app, account, API playground and user-generated content routes for all bots, and put anything private behind authentication.
• Never block Googlebot or Bingbot as part of an AI block list, or you'll drop out of normal search too.

Our guide to blocking or allowing AI crawlers lists each bot, its owner and what blocking it changes. Check what your current robots.txt actually allows with the AI crawler checker, build a clean file with the robots.txt generator, and test individual paths with the robots.txt tester. Many framework templates and CDN settings block AI bots by default, so check rather than assume.

2. llms.txt: optional, not a ranking lever

llms.txt is a proposed file that gives AI tools a Markdown summary of your site. It isn't an official standard, and Google Search ignores it. It can be useful for AI companies with developer documentation, because some coding tools read it. Our explainer, what is llms.txt, covers when it's worth adding and what goes in it, so we won't repeat that here. Treat it as a small extra, not a substitute for crawlable pages.

3. Structure pages so AI tools can describe you accurately

AI answers can only be as accurate as the pages they draw on. Vague hero copy like "the future of intelligent work" gives a model nothing to quote. Make sure your site states plainly:

• What the product does, in one sentence, on the home page.
• Who it's for, with separate use-case pages for each main audience.
• How it works, including which models or approaches you use, at the level of detail you're comfortable sharing.
• What it costs, with pricing in HTML text rather than inside a script or image.
• What it doesn't do, and its known limitations.

Serve these pages as server-rendered or static HTML. Many AI crawlers don't run JavaScript, so a client-rendered marketing site can look empty to them. Add Organization schema on the home page and SoftwareApplication on the product page, then test both with the schema markup validator.

4. Trust pages that answer buyers before they ask

Business buyers of AI products ask the same questions every time: where does our data go, is it used for training, who are your sub-processors, how do you secure it? Answer them on the site:

• Security page: hosting, encryption in transit and at rest, access controls, MFA, incident response, certifications you genuinely hold such as Cyber Essentials, and how to report a vulnerability.
• Privacy notice: what personal data you collect, why, your lawful basis, retention, international transfers and individuals' rights. A privacy policy generator gives you a starting draft to review with your adviser.
• Data use and model training: whether customer inputs and outputs are stored, for how long, and whether they're used to train or improve models. Say whether customers can opt out.
• Sub-processors list: model providers, cloud hosting, analytics, support tools, with locations.
• Data processing agreement: downloadable or available on request.
• Status page: current and past incidents.
• security.txt: under RFC 9116, place it at /.well-known/security.txt, include a Contact and an Expires field, and keep the expiry less than a year ahead.

Write these pages for accuracy, not marketing. If a statement on your security page turns out to be untrue, it's a misleading claim as well as a sales problem.

5. Security headers and the public surface

Enterprise prospects and their security teams often check your public site first. A marketing site launched on a template in a weekend usually lacks the basics. Check:

• HTTPS everywhere, with HSTS once you're confident every subdomain supports it.
• Content-Security-Policy, starting in report-only mode if your site loads many third-party scripts. Our CSP guide explains how.
• Frame protection through frame-ancestors or X-Frame-Options, especially on login pages.
• X-Content-Type-Options: nosniff and a sensible Referrer-Policy.
• No exposed files such as .env or .git on the marketing or app domain.

Run the security headers checker on the marketing site, the app login and the docs. Our HTTP security headers guide explains each header. These checks don't replace a penetration test, but they give honest answers to the parts of a questionnaire anyone can verify from outside.

6. Email before the launch

Startups often launch on a new domain and email the whole waitlist on day one. Set up SPF, DKIM and DMARC first, check them with the DMARC checker, then send in batches that grow over days. Google's sender guidelines require DMARC for anyone sending more than 5,000 messages a day to Gmail. Our guide to warming up an email domain covers the ramp.

7. Claims that stand up to the ASA and CMA

AI marketing is under scrutiny. The ASA applies the CAP Code to claims on your own website and social channels, not just paid ads. If you also sell in the US, the FTC takes a similar line: it expects a reasonable basis for objective claims about AI products and, through Operation AI Comply, has taken action against companies that overstated what their AI can do.

In an August 2026 article, Regulating the illusion of intelligence in ads, CAP's advice to AI advertisers was to "hold evidence to back up any objective claims about what your product can do", and to make sure content is responsible. It flagged particular concerns about AI products in homework, health and wellbeing, companionship, body image and income claims. Separately, the ASA has said advertisers shouldn't falsely claim AI features or exaggerate what AI can do.

For an AI startup website, that means:

• Hold evidence before you publish claims such as "99% accurate", "saves 10 hours a week" or "replaces your analyst". Know how the figure was measured, on what data and under what conditions.
• State limitations alongside capabilities, such as supported languages, file types or accuracy ranges.
• Don't call a feature AI if it isn't.
• Be careful with "free". If a free scan or trial leads to a paywall for the result, say so clearly.
• Keep testimonials and reviews genuine. Fake reviews and concealed incentivised reviews are banned under UK consumer law. In the US, the FTC's rule on consumer reviews and testimonials (16 CFR Part 465) bans fake reviews and incentives conditioned on a positive or negative review.

Consumer law enforcement has also changed. Since April 2025, the Digital Markets, Competition and Consumers Act 2024 lets the CMA decide for itself whether consumer law has been broken and fine businesses up to 10% of global turnover. That covers misleading claims and practices such as drip pricing aimed at consumers.

8. Transparency about AI and automated decisions

Explain on your site where AI is used and where people stay involved:

• Automated decisions about people. The Data (Use and Access) Act 2025 rules on automated decision-making came into force on 5 February 2026, according to the ICO. Significant decisions made solely by automated means need safeguards: telling people about the decision, letting them make representations, offering human intervention and allowing them to contest it. If your product supports decisions in hiring, lending or similar areas, explain how your customers can meet those duties.
• Chatbots. Tell users when they're talking to an AI system rather than a person. If you serve users in the EU, the EU AI Act's transparency obligations also matter; law firms report most of them have applied since 2 August 2026, so take advice if that's relevant.
• AI-generated content. If your product creates images, audio or text that could be mistaken for real, explain how outputs are labelled.
• Human oversight. Say plainly which decisions stay with people. Buyers in regulated sectors will ask.

The ICO's guidance on AI and data protection covers fairness, transparency and lawful basis in more depth. The ICO has also said it's starting work on a statutory code of practice on AI and automated decision-making.

9. Launch assets and social previews

Launch posts on LinkedIn and X depend on Open Graph tags. Check every key page with the Open Graph checker before you announce, and make sure the share image is the right size. Generate a full favicon set so the site looks finished in browser tabs. Our Open Graph tags guide covers the details.

Our free tools for AI companies page collects the checks in this guide. Make It Flow is an AI automation agency itself, and we work with software companies on automation with people kept in charge of decisions. If that's useful, get in touch.

FAQ

What should an AI startup website include?

A clear statement of what the product does and who it's for, use-case and pricing pages in crawlable HTML, trust pages covering security, privacy, data use and sub-processors, a security.txt file, sensible security headers, authenticated email, and a deliberate robots.txt policy for AI crawlers. Every capability claim should be backed by evidence.

Should an AI company allow AI crawlers?

Most should allow AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot on marketing and docs pages, so assistants can cite them. Training crawlers are a separate decision. Block app and account routes for all bots, and never block Googlebot or Bingbot as part of an AI rule.

Does an AI startup need llms.txt?

No. It's an optional, proposed file, and Google Search ignores it. It can help AI companies with developer documentation, because some coding tools read it. Crawlable, server-rendered pages that state clearly what the product does matter far more for AI visibility.

What trust pages do B2B buyers expect from AI companies?

Typically a security page, privacy notice, a statement on whether customer data is used for model training, a sub-processors list, a data processing agreement, a status page and a way to report vulnerabilities. Keep them accurate and current, because buyers' security teams will check them.

Can the ASA act on claims on our own website?

Yes. The ASA applies the CAP Code to marketing claims on companies' own websites and social media. For AI products, it expects evidence for any objective claim about what the product can do, and warns against falsely claiming AI features or exaggerating capabilities.

Do UK rules on automated decision-making apply to AI startups?

They apply when your product, or your customer using it, makes significant decisions about people solely by automated means. Since 5 February 2026, the Data (Use and Access) Act rules allow this in more cases but require safeguards, including information, the chance to contest and human intervention. Special category data has stricter limits.