This Monday, 4th August 2025, big news came up from the top leading internet security provider, Cloudflare. In a recent blog post, the company has published regarding the stealth crawling behavior from Perplexity, an AI-powered search engine. Even though it initially crawls from a declared user agent.
The network infrastructure provider has observed that Perplexity is obscuring its identity to circumvent the website’s preferences and disregarding protocols such as robots.txt, controlling how bots interact with web content. This action is a response to multiple user complaints regarding the violation of robots.txt protocols and the creation of WAF rules to block Perplexity’s declared crawlers, PerplexityBot and Perplexity-User.
Cloudflare offers a Verified Bots program that authorizes bots to crawl websites protected by Cloudflare. The verified bots should abide by the companies' policies, which include robot.txt protocols, to maintain their privileged status in the Cloudflare system. It was found that Perplexity violated Cloudflare’s policies.
How Perplexity is Bypassing Blocks?
According to research by Cloudflare, the AI search engine Perplexity bypassed website restrictions by:
- Spoofing its identity
- Rotating IP addresses
- Switching Autonomous System Numbers (ASN)
These innovative methods were employed when websites attempted to block their crawlers. This incident has affected tens of thousands of domains and resulted in millions of requests per day. Alongside, Perplexity changed its user agent to mimic a human visitor using Google Chrome or macOS.
Cloudflare’s Ultimate Response
Considering the scenario and stealth tactics to bypass restrictions by Cloudflare, the internet security provider has officially:
- Delisted Perplexity’s bots from its Verified Bots program
- Deployed new blocking methods to stop the stealth crawling activity.
This represents a significant step by Cloudflare against AI scraping that can harm both the web and publishers.
Now, Perplexity went against the allegations. Jesse Dwyer, Perplexity’s spokesperson, responded to Cloudflare’s post, saying, “public stunt,” the company did not use any content, and the bot detected by Cloudflare is not ours. This isn’t the first time. In 2024, Perplexity was caught stealing content from sites such as The New York Times.
Cloudflare CEO Matthew Prince has shared his thoughts on the vast impact that AI scraping is having on the internet economy. Additionally, he has shed light on AI startups using publisher content without consent, thereby affecting publishers' revenue. AI companies are booming rapidly in the market, working on training LLM models using data scraped from the web, while also addressing consent issues and other concerns.
Ultimately, the action against Perplexity highlights the growing conflict between digital publishers and AI firms. Even though Perplexity denies any misconduct, Cloudflare's move is a significant step toward creating web standards and providing content owners with a safeguard against scraping, especially in the AI age.
Stay tuned for more tech trending news here: SecureITWorld
Also Read: What is Perplexity AI Model? Is Perplexity AI Better than ChatGPT?
