Two different tools handle this, and they do different jobs. robots.txt is a request, not a lock: well-behaved bots read it and stay out of the paths you list, but nothing stops a bot from ignoring it. A server-level block, done through .htaccess or your firewall, actually refuses the connection. If a crawler is hammering your server or scraping content you don't want scraped, robots.txt alone won't stop it, you need a real block.
Most sites only need robots.txt. Reach for a server-level block when a specific bot is ignoring your rules, spiking your server load, or you've confirmed it's a scraper rather than a legitimate crawler.
robots.txt: the polite ask
Your robots.txt file lives at the root of your domain, for example yoursite.com/robots.txt. Edit it through the Files tile in the portal (Services → click your hosting service → Files) or upload a new one over FTP, see FTP and SFTP accounts or uploading files to your hosting if you need a refresher on getting a file onto the server.
Basic syntax blocks a bot by name and lists paths it shouldn't touch:
User-agent: BadBot Disallow: / User-agent: * Disallow: /admin/ Disallow: /cart/ Allow: /
The first block above tells BadBot specifically to stay off the entire site. The second applies to everyone else, blocking only /admin/ and /cart/. Order and specificity matter: a bot matches the most specific User-agent line that applies to it, not every matching block.
Common AI crawler user-agents you'll see requesting your robots.txt: GPTBot and ChatGPT-User (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Google's AI training, separate from regular Googlebot), PerplexityBot, and CCBot (Common Crawl, which many AI labs train on downstream). Each respects its own User-agent block, so you can allow some and block others.
Server-level blocks: the actual lock
If a bot ignores robots.txt, or you want to block by IP range or request pattern rather than declared identity, block it at the server. In .htaccess (reachable via the Files tile, or the File Manager tool in cPanel):
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} BadBot [NC]
RewriteRule .* - [F,L]
This returns a 403 to any request whose user-agent string contains BadBot, regardless of what robots.txt says. It's heavier-handed and appropriate when a crawler is a genuine load problem, since it stops the request before your PHP or database ever runs, unlike a robots.txt rule which the bot has to choose to honor.
Keep in mind user-agent strings are self-reported. A scraper that wants past your block can claim to be a browser. For persistent abuse from a specific source, blocking by IP or range is more durable than matching on user-agent, though it takes more upkeep as ranges change.
The AI crawler tradeoff
This is genuinely a judgment call, not a settled best practice, and the honest answer cuts both ways.
Arguments for allowing AI crawlers: if your business depends on being found, being cited in an AI-generated answer is a real discovery channel now, alongside search. Blocking GPTBot or ClaudeBot means your content can't be referenced in ChatGPT or Claude responses at all. For a documentation site, a knowledgebase, or anything hoping to be an authoritative source, that visibility has value.
Arguments for blocking them: AI crawlers don't send you traffic the way a search result click does, someone gets an answer without ever visiting your site. If you monetize via ads or want visits you can measure, that's a real cost with no visit in return. Crawl volume is also a legitimate concern: some AI crawlers are aggressive and will re-crawl frequently, adding load without any benefit to you if you've decided you don't want to be training data or a citation source.
There's no universally correct answer. A SaaS company's docs site probably wants the citations. A site selling something where the value is the visit itself, not the information, may reasonably decide the tradeoff isn't worth it. Decide based on what you're actually trying to get out of visitors, then write the rule to match, and revisit it if either side of the tradeoff changes.
Checking what's actually crawling you
Before you block anything, check whether it's worth blocking. Server access logs show you the real request volume and user-agent strings hitting your site. If you have shell access, a quick way to see who's showing up most:
grep -c "GPTBot" access.log grep -c "ClaudeBot" access.log
If a particular bot is a small fraction of your traffic, blocking it probably isn't worth the maintenance. If one bot is a large share of requests and clearly not converting into anything useful for you, that's your signal to write the rule.
When to get help
If you're seeing unusual load and can't tell whether it's a legitimate crawler, a scraper, or something worse, open a ticket from the portal (Support → New ticket) or use live chat. Our team can help you read the logs and figure out whether a robots.txt rule is enough or you need a firewall-level block.