← Back to all free guides
A free guide by Mariah Brunner ↗ INSTAGRAM ↗ TIKTOK

Get Your Website
Ready For AI

Bots just passed humans online. More of your customers ask AI instead of visiting your site. Here’s how to make sure AI can find you, understand you and recommend you.

In June 2026, Cloudflare’s CEO shared Radar data showing bots made about 57% of web page requests on its network, with people at about 43%. A big chunk of that is AI reading the web to answer questions inside ChatGPT, Claude and Perplexity. (The numbers)

So more and more, people don’t visit your website. They ask an AI, the AI reads your site for them, and it reports back. You have to build for that reader too.

Quick Reality Check

"Bots passed humans" counts requests, not attention. People didn’t leave the web. Your website matters more now, because it’s the source the AI reads about you.

Step 1: Let AI Reach You

01

Check your robots.txt

Lots of sites block AI without knowing it. Type yourwebsite.com/robots.txt into any browser. That little file tells bots what they may read. Look for Disallow: / under bot names like OAI-SearchBot, PerplexityBot or Claude-SearchBot. Those are the ones that put you in AI search answers.

Here’s the safe starting block. It lets in the bots that answer live questions and can cite you. Hand it to whoever runs your site.

Let The Answer Bots In (robots.txt)
User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /

02

Know the two kinds of bots

Search and user bots (the ones above) fetch your page to answer someone’s question and can link back to you. Training bots like GPTBot and ClaudeBot collect content to train future models. You can block training and still allow search. OpenAI lists its bots here, Anthropic here and Perplexity here.

Blocking Google-Extended only covers Gemini training and grounding. It doesn’t remove you from Google Search.

03

Check Cloudflare if you use it

Cloudflare is phasing out its old one-click "Block AI bots" toggle (deprecated September 15, 2026) in favor of AI bot policies. Go to Security Settings, then Configure AI bot policies. You’ll see three presets: Search, Agent and Training. Newer defaults can block Agent bots on pages with ads, so set Search and Agent to Allow if you want to be recommended. (Cloudflare’s docs)

Step 2: Make Your Site Easy To Understand

AI likes pages that plainly answer real questions. Clever and vague hurts you here.

04

Say exactly what you do

On your homepage and About page, state what you do, who it’s for and where you are. AI reads this literally. "We’re a family-owned bakery in Austin, Texas, making custom wedding cakes" does more than a page of pretty branding.

05

Build a real FAQ page

Use the questions customers actually ask as headings, like "Do you ship nationwide?" Answer each in one to three plain sentences right under it. That’s the format AI pulls answers from. This prompt does the writing for you.

About + FAQ Rewriter
You are a website copywriter who writes for both people and AI assistants. Rewrite my About section and build an FAQ that an AI can quote word for word. I'M ATTACHING OR PASTING - My current homepage and About page text: [paste] - The questions customers ask me most (from DMs, emails, calls): [paste a list, rough is fine] - Facts that must be exact: [prices, service area, hours, shipping, booking policy] DO THIS 1. Write a 2-sentence About opener that states what we do, who it is for and where we are. No slogans. 2. Turn my customer questions into 8 to 12 FAQ entries. Use the customer's own wording as the question. Answer each in 1 to 3 plain sentences, with the most important fact first. 3. Suggest clear section headings for my homepage (for example "Pricing" instead of "Investment"). 4. List any facts you needed but I did not give you. FORMAT - About opener, then FAQ as "Q:" / "A:" pairs, then headings, then missing facts. RULES - Use only the facts I gave you. If something is missing, write [NEED: ...] instead of making it up. - Keep my voice: [warm / professional / playful]. No hype words.

06

Use clear headings and add schema

Name sections what they are. "Pricing" beats "Investment." Then add schema markup, hidden labels that tell machines your name, address, hours and products. You don’t have to code it. Shopify, Squarespace and Wix add some automatically, and on WordPress a plugin like Yoast or Rank Math adds the basics. Ask whoever runs your site to set up LocalBusiness and Product schema.

Step 3: Get AI To Recommend You

AI trusts what the wider web says about you, beyond your own claims. So part of this work happens off your site.

  1. Make your business info identical everywhere. Name, address, phone, hours and website should match across Google Business Profile, Yelp, Instagram, Apple Maps and directories. Mismatches make you look less trustworthy.
  2. Get and keep reviews. AI leans on third-party reviews when it picks who to recommend.
  3. Get mentioned on sites you don’t own. Local press, "best of" lists, partner sites and podcasts all count as proof you’re real.

Skip The Hype

You’ll hear about an llms.txt file as a magic fix. No major AI company has said its crawlers use it, and log studies show they rarely fetch it. Nail robots.txt, clear pages, consistent info and reviews first.

Step 4: Test It This Week

Open ChatGPT, Claude, Perplexity and Google’s AI Mode and ask what a customer would ask. Try "Best [your category] in [your city]" and "What does [your business] do, and are they any good?" Or let this prompt run the whole audit for you in any AI with web search turned on.

AI Visibility Audit
You are an AI search visibility auditor for a small business. Your job is to find out how AI assistants see my business right now and tell me exactly what to fix first. MY BUSINESS - Name: [business name] - Website: [yourwebsite.com] - What we do and for whom: [one sentence] - City or service area: [city, state] - 2 or 3 competitors I lose to: [names] STEPS 1. Use web search. Search the way a customer would: "best [category] in [city]", "who does [service] near me", and "[business name] reviews". Note whether my business appears, what is said about it, and which sources you relied on. 2. Open [yourwebsite.com]/robots.txt. List any rules that block OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User or Googlebot. If you cannot open it, say so. 3. Read my homepage and About page. In one sentence, say what you think we do, who it is for, and where we are. Flag anything vague or missing. 4. Compare my name, address, phone and hours across my site, Google Business Profile, Yelp and any directories you find. List every mismatch. 5. For each competitor, list the sites that mention them but not me (reviews, "best of" lists, local press). OUTPUT - A one-line verdict: Invisible, Visible but wrong, or Visible and accurate. - A table: Issue | Where you found it | Fix | Effort (Low/Med/High). - My top 3 fixes for this week, in order. RULES - Cite the page for every finding. If you could not check something, say "Not checked" instead of guessing. - Do not invent reviews, rankings or traffic numbers. - Plain language. I am not technical.
  1. Not showing up? Check robots.txt, then earn more reviews and outside mentions.
  2. Showing up with wrong info? Fix your About page, Google Business Profile and listings so the facts match.
  3. Competitors winning? See where they get cited and go earn the same.

Re-test every few weeks. AI answers change from run to run, so look at the pattern, not one result.

The Other Half: Build What AI Can’t Touch

Here’s the take I’m betting on most. As the internet fills up with AI, the things AI can’t copy get more valuable. The biggest one is real, in-person community. People are starving for it, and most of us spend less time with friends than we used to.

You can see brands leaning in already. Run clubs are everywhere, brands sponsor local running crews, and coffee shops throw morning DJ sets. Brands are hosting real-world events instead of only running more ads.

You don’t need a rave. You need one real recurring in-person thing, like a workshop, a meetup, a class or a monthly coffee. It builds loyalty and word of mouth no AI can generate. Bonus: those events create the local mentions and reviews that boost your AI visibility in Step 3.

The Real Win

Make everything online easy for AI to read, and build in-person community AI can’t touch. That combo keeps you found and loved while the web fills up with machines.