Robots.txt just turned 30 – cue the existential crisis! Like many hitting the big 3-0, it’s wondering if it’s still relevant in today’s world of AI and advanced search algorithms. Spoiler alert: It ...

The Xserver server panel has had an item called 'AI Crawler Blocking' since January 2026. It is a feature that allows you to ...

Generative AI is breaking established internet etiquette to satisfy a bottomless appetite for training data. For example, Microsoft-backed OpenAI and Amazon-supported Anthropic ignore robots.txt to ...

Perplexity was discovered to be actively bypassing blocks from websites to scrape content in 2024, and a new report shows that it has continued with increasing sophistication as the company defends ...

Robots.txt files can be centralized on CDNs, not just root domains. Websites can redirect robots.txt from main domain to CDN. This unorthodox approach complies with updated standards. Google's Gary ...

Perplexity, a company that describes its product as "a free AI search engine," has been under fire over the past few days. Shortly after Forbes accused it of stealing its story and republishing it ...

There is this interesting conversation on LinkedIn around a robots.txt serves a 503 for two months and the rest of the site is available. Gary Illyes from Google said that when other pages on the site ...

Large language models are trained on massive amounts of data, including the web. Google is now calling for “machine-readable means for web publisher choice and control for emerging AI and research use ...

Do you use a CDN for some or all of your website and you want to manage just one robots.txt file, instead of both the CDN's robots.txt file and your main site's robots.txt file? Gary Illyes from ...