How to Hide Your Site From Semrush and Ahrefs (2026 Guide)
By SM Mehedi Hasan
To hide your site from Semrush and Ahrefs, block their crawlers (SemrushBot and AhrefsBot) using robots.txt, an .htaccess rule, or a Cloudflare firewall filter. This stops them from collecting fresh data about your pages. Incoming backlinks from sites you do not control will still show up in their databases.
Most website owners think hiding from these tools is a one-switch flip. It is closer to a set of trade-offs you choose between.
Block too much, and you lose your own rank tracking and site audits. Block too little and competitors still pull your full playbook.
This guide walks through every working method in 2026, what each one actually hides, and where the limits sit.
I have set these blocks up on my own sites and client sites, so the notes here come from logs and dashboards, not theory.
What does it mean to hide your site from Semrush and Ahrefs?
Hiding your site from Semrush and Ahrefs means stopping their crawlers from accessing your pages so they cannot collect fresh SEO data about your domain. You do this by blocking two specific bots: SemrushBot and AhrefsBot.
These crawlers visit websites the same way Googlebot does. They follow links, read pages, and store what they find.
The difference is that their data feeds paid tools, so any subscriber, including your competitors, can study what they collect.
When you block these bots, you are not deleting yourself from the internet. You are telling those two companies that their robots are not allowed to read your pages directly.
Google still crawls you. Your visitors still see everything. Only the third-party data pipeline gets cut.
If you run a site in a tight niche, this matters because your backlink sources, top pages, and keyword gains are usually the first things a rival checks before copying you.
Why would you want to hide your site from these SEO tools?
People hide from Semrush and Ahrefs for three main reasons: protecting strategy from competitors, saving server resources, and keeping private sites out of public databases. Each reason fits a different type of site.
Protecting your strategy from competitors
Competitive privacy is the number one reason. When your data is exposed, any rival with a subscription can see which pages earn your traffic, which links power your rankings, and which keywords you are climbing for.
I noticed this clearly on a client site in a small affiliate niche. Within weeks of a ranking jump, two competitors had near-identical articles targeting the same terms. They had simply read the gains in a tool and reverse-engineered the play.
Blocking the crawlers slows that down. It does not make you invisible, but it removes the easy map that points straight at what is working for you.
Saving server resources and bandwidth
Crawlers cost you server load. AhrefsBot is one of the busiest bots online, and on shared hosting, that constant activity competes with real visitors for the same resources.
When I was reviewing access logs for a low-budget hosting setup, SEO bots were eating a noticeable slice of daily requests. Real users got slower responses during peak crawl windows. Cutting non-essential bots freed that headroom.
If you are on weak hosting, this matters because every request a bot makes is a request your actual customer is waiting behind.
Keeping private or staging sites out of public databases
Staging sites, client previews, and development builds should never appear in a public tool. Blocking crawlers in these environments keeps unfinished work and private structures hidden.
Honestly, this is the use case I trust most. There is no downside. You are not tracking SEO on a staging site anyway, so blocking every external crawler is the clean choice.
Can you fully hide a website from Semrush and Ahrefs?
No, you cannot fully hide a website from Semrush and Ahrefs. You can only hide the data they collect by crawling your own pages directly.
Links pointing to you from other websites stay visible because those tools crawl those external sites, too.
This is the part most articles skip, and it trips people up. Picture your backlink profile as two halves.
The first half is data from your own pages: your internal links, your outgoing links, your page content, your site structure. Block the bots, and this half goes dark over time.
The second half is data from other domains linking to you. A blog in another country links to your homepage. Semrush crawls that blog, sees the link, and records it. You never touched that site, so you cannot stop that record.
Most people assume blocking the bot erases their backlink profile. It does not. It only thins the data that comes from your side of the web. The links others built to you keep showing up.
So the honest framing is this. Blocking gives you partial privacy and meaningful server relief. It does not give you a clean slate. Anyone determined enough can still piece together a rough picture from external sources.
How do you block AhrefsBot and SemrushBot using robots.txt?
- Open your site root folder using your hosting File Manager, FTP, or your CMS file editor.
- Find the file named robots.txt. If it does not exist, create a blank one in the root directory.
- Add the following block rules to the file:
User-agent: AhrefsBot
Disallow: /
User-agent: AhrefsSiteAudit
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
- Save the file and upload it back to the root if you edited it locally.
- Visit yourdomain.com/robots.txt in a browser to confirm the rules are live.
Robots.txt is the simplest method, and it works because both Ahrefs and Semrush officially respect these directives. The Disallow: / line tells each bot it cannot crawl any path on your site.
After saving, you should see the bots stop appearing in fresh crawl logs within a few days. Their existing stored data fades slowly rather than vanishing instantly.
One caution. Robots.txt is a polite request, not a wall. Honest bots obey it.
A spoofed bot pretending to be AhrefsBot will ignore it completely. For that problem, you need the server-level methods below.
How do you block these crawlers with .htaccess?
- Confirm your hosting runs Apache, since .htaccess only works on Apache and LiteSpeed servers.
- Open the .htaccess file in your root folder, or create one if it is missing.
- Paste this rule near the top of the file, before other directives:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT}
(AhrefsBot|SemrushBot|MJ12bot|DotBot|rogerbot) [NC]
RewriteRule .* – [F,L]
- Save the file and reload your site to make sure nothing breaks.
- Check your access logs and confirm that those bots now receive a 403 Forbidden response.
This method is stronger than robots.txt because it rejects the request at the server before the bot reads any content. The bot has no choice in the matter, assuming it uses its real user-agent name.
When I switched a heavy-traffic site from robots.txt to an .htaccess rule, the difference in logs was obvious. Instead of polite skips, the bots got hard 403 responses and stopped retrying as often.
The [NC] flag makes the match case-insensitive, and [F,L] forces the forbidden response and stops further rule processing. Add or remove bot names inside the brackets to fit your needs.
How do you block these crawlers using Cloudflare?
- Log in to your Cloudflare dashboard and select your domain.
- Go to Security, then WAF, then Custom rules.
- Click Create rule and name it something clear, like Block-SEO-Bots.
- In the expression editor, paste this filter:
(http.user_agent contains “AhrefsBot”) or (http.user_agent contains “SemrushBot”) or (http.user_agent contains “MJ12bot”)
- Set the action to Block, then click Deploy.
Cloudflare blocks the bot before it even reaches your origin server, which is the biggest advantage here. Your hosting never processes the request, so you save the most resources of any method.
If you are running several sites, this matters because one Cloudflare rule set can cover many domains with consistent behavior. You manage bot policy in one place instead of editing files on every server.
Compared to what I have tried before, the Cloudflare route handled aggressive crawlers the cleanest.
When a bot sent dozens of requests at once to test a robots.txt block, Cloudflare stopped them at the edge instead of letting them hammer the server.
How do you block crawlers in WordPress with Rank Math?
- In your WordPress dashboard, go to Rank Math, then General Settings.
- Open the Edit robots.txt tab.
- Paste the same Disallow rules shown in the robots.txt section above into the editor box.
- Click Save Changes to push the rules live.
- Verify by loading yourdomain.com/robots.txt in your browser.
Rank Math edits your virtual robots.txt directly from inside WordPress, so you never touch a file by hand. For most WordPress owners, this is the safest entry point.
The thing that surprised me most was how many people break their robots.txt by editing the wrong file. Rank Math removes that risk because it manages the file for you and shows you the live output before you save.
But remember the same limit applies. This is still a robots.txt method, so it only stops bots that choose to obey. For force-level blocking on WordPress, pair it with Cloudflare.
Should you block or just limit these crawlers?
You should limit rather than fully block when you still want accurate Ahrefs and Semrush data for your own work, but need to control server load. Full blocking only makes sense when privacy matters more than your own access to the data.
This is not an all-or-nothing choice, and treating it that way is where people lose value. There are three positions: allow, limit, or block.
If you limit instead of block, you slow the crawler down without cutting it off. You add a crawl-delay or restrict it to certain folders. The bot still feeds data, just at a gentler pace.
Here is a simple way to decide based on your situation.
| Your situation | Best choice | Why it fits |
|---|---|---|
| You rely on Ahrefs or Semrush for your own audits | Limit | Keeps your data accurate while easing server load |
| You are in a cutthroat niche and want privacy | Block | Hides your strategy from rival subscribers |
| You run staging or private sites | Block | No SEO tracking needed, full privacy preferred |
| Your hosting is strong and crawls are light | Allow | No real reason to restrict useful bots |
To limit instead of block, use a crawl-delay rule in robots.txt like this:
User-agent: AhrefsBot
Crawl-delay: 10
User-agent: SemrushBot
Crawl-delay: 10
You can also restrict bots to skip heavy folders while still reading your main content:
User-agent: AhrefsBot
Disallow: /wp-admin/
Disallow: /cgi-bin/
Ahrefs also lets verified site owners set a custom crawl rate inside Ahrefs Webmaster Tools, which gives you tighter control than robots.txt alone.
In My Experience
After running these setups for a long stretch across several sites, the biggest lesson was that limiting beats blocking is the most effective for most real businesses.
I started by hard-blocking everything on a client site, then realized I had blinded my own monthly audits in the process.
The site audit feature in Ahrefs stopped working entirely because the audit bot could no longer reach the pages. I had to whitelist the audit crawler again just to run my own reports.
That back-and-forth cost a week of confused troubleshooting. So now my default is to limit on money sites I actively track, and full-block only on staging or sites I do not analyze.
The privacy gain from a full block rarely outweighs losing your own data on a site you are trying to grow.
Which SEO bots should you block? (full 2026 user-agent list)
The main SEO crawlers to know are AhrefsBot, SemrushBot, MJ12bot, DotBot, and Rogerbot. Blocking only one leaves the others free to crawl, so privacy work means handling each tool separately.
Blocking AhrefsBot does nothing to stop Semrush. Blocking Semrush does nothing to stop Moz. Each company runs its own bots under its own name, and they update those names over time.
| SEO tool | Crawler name | Notes |
|---|---|---|
| Ahrefs | AhrefsBot, AhrefsSiteAudit |
Respects robots.txt and crawl-delay |
| Semrush | SemrushBot, SemrushBot-SA |
SA variant powers site audits |
| Majestic | MJ12bot | Distributed crawler, hard to block by IP |
| Moz | DotBot, rogerbot | Two separate bots for index and analysis |
Here is a non-obvious point most guides get wrong.
Ahrefs published its own research on how often each SEO bot gets blocked across millions of robots.txt files, and SemrushBot came out as the most explicitly blocked SEO crawler, ahead of Moz and even ahead of AhrefsBot itself.
Compared to similar assumptions I held before reading that data, it flipped my thinking. People talk about blocking Ahrefs first, but more site owners actually single out Semrush. If full privacy is your goal, you cannot just block the one name everyone mentions.
How do you verify real bot traffic versus fake spoofed bots?
You verify real bot traffic by checking the IP address and reverse DNS, not the user-agent name alone. The name AhrefsBot can be faked by any scraper, so a name match proves nothing on its own.
This is the gap that catches people in 2026, when bot spoofing is common. A malicious scraper can wear the AhrefsBot name to look harmless while ignoring every rule you set.
To confirm a real AhrefsBot visit, the request should come from an IP inside Ahrefs published ranges, and a reverse DNS lookup on that IP should resolve to a hostname ending in ahrefs.com or ahrefs.net.
I ran into an issue when a site showed huge AhrefsBot activity even after a robots.txt block. The logs looked alarming until I checked the IPs.
They did not match Ahrefs’ ranges at all. It was a spoofed scraper, and only a server-level firewall rule stopped it.
So the takeaway is layered. Robots.txt handles honest bots. Reverse DNS plus a firewall handles the liars. If your block rule is not working, a spoofed bot is the usual reason.
What are the downsides of blocking Semrush and Ahrefs?
The main downsides are losing your own SEO data, breaking your own site audits, and appearing weaker to potential link partners. Blocking is easy to turn on, but the data gaps it creates are slow to undo.
When you block these bots, your own dashboards go stale. Your rank tracking, backlink monitoring, and content reports inside those tools stop updating with fresh data from your pages.
This works well as a privacy move, except when you depend on those same tools for your daily work. You cannot block the crawler and still expect a complete site audit from it. The two goals fight each other.
There is also a perception cost. SEO professionals check Ahrefs and Semrush before offering you a guest post or a link.
If your data looks thin or outdated, your site may seem less authoritative than it really is, and you could lose outreach interest.
What I did not expect was how long the data took to recover after I unblocked a bot. The gaps did not refill instantly. It took weeks of recrawling before the numbers looked normal again, so this is not a switch you flip casually.
Common Pitfalls When Hiding Your Site From SEO Crawlers
Beginners make a handful of repeat mistakes here, and most come from misunderstanding what blocking actually does. These errors either break the site or create a false sense of privacy.
- Blocking Googlebot by accident. A sloppy Disallow: / aimed at all bots can drop you from Google entirely. This happens when people copy a broad rule without targeting specific bot names. Always name the SEO bots; never block everything blindly on a live site.
- Editing the wrong robots.txt file. Many sites have a virtual robots.txt generated by a plugin. Editing a physical file that the plugin overrides means your rules never go live. Check the live output at the browser URL after every change.
- Assuming robots.txt stops every bot. Robots.txt is voluntary. Spoofed and rogue bots ignore it. People feel protected while scrapers walk right through. Add a firewall layer if real privacy is the goal.
- Blocking only one tool. Stopping AhrefsBot while leaving SemrushBot and Moz wide open gives almost no real privacy. Competitors just switch tools. Handle every crawler or accept that the block is mostly symbolic.
- Forgetting the site audit bot. Blocking the audit crawler kills your own ability to run technical reports. People block it, then wonder why their audits fail. Decide whether you still need your own audits before cutting that bot.
Each of these comes from the same root cause. Owners treat blocking as one universal action instead of a set of targeted rules with clear trade-offs.
Workflow Example: Hiding a Site From SEO Crawlers Step by Step
Here is a full, realistic flow showing how a privacy-focused block actually plays out from start to finish.
Input: A small affiliate site on shared hosting in a competitive niche. The owner wants competitor privacy and lower server load, but still wants their own monthly audits to work.
Process: They start in Cloudflare and add a WAF custom rule blocking AhrefsBot, SemrushBot, and MJ12bot by user-agent.
Then they edit robots.txt through Rank Math to add Disallow rules for the same crawlers as a polite second layer. They deliberately leave the Ahrefs site audit bot allowed by whitelisting its name, so their own reports keep running.
Finally, they check server access logs to confirm the blocked bots return 403 responses and the audit bot still gets through.
Output: Competitor-facing crawlers are stopped at the Cloudflare edge before touching the server. The site audit bot still reaches the pages. Server logs show a clean split between blocked and allowed bots.
Result: Over the next month, competitor tools show stale, thinning data for the site, which is the privacy win. Server load from bots drops noticeably, helping real visitors’ speed.
The owner keeps full access to their own audit reports because they whitelisted that one bot. Incoming backlinks from external domains still appear, which is expected and unavoidable.
This flow shows the balanced approach in action. Block what exposes you, allow what serves you, and accept the one limit you cannot beat.
Frequently Asked Questions
No. Both are third-party crawlers that work independently from Google. Blocking them has zero effect on how Googlebot crawls, indexes, or ranks your site. Your organic search performance stays exactly the same.
Not fully. Your own page data fades over time, but backlinks pointing to you from other websites still appear because those tools crawl those external domains separately. You cannot remove links from sites you do not control.
Yes. Each tool uses its own named crawler, so you block them individually. Blocking AhrefsBot does not stop SemrushBot, and the reverse is also true. Real privacy means handling every crawler you care about.
No. Controlling which bots access your own website is a normal, legal technical SEO practice. You are managing your own server, not breaking any law or violating search engine guidelines by setting crawler rules.
Usually, because the bot is spoofed or robots.txt is being ignored. Robots.txt is voluntary, so fake bots wearing real names bypass it. Use an .htaccess rule or a Cloudflare firewall for force-level blocking instead.
Is an SEO Specialist and AI Tools Researcher with over 4 years of hands-on experience in search engine optimization. As the founder of Smart AI Helper Pro, he tests and reviews AI writing, SEO, and marketing tools to help creators and business owners grow faster with practical, research-backed strategies.
