How robots.txt Affects Web Archiving
The ethical and technical implications of robots.txt for web crawlers and archival bots.
The robots.txt Standard
The Robots Exclusion Protocol (robots.txt) is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
The Archival Dilemma
For search engines like Google, obeying robots.txt is standard practice. If a site owner doesn't want a page indexed, Googlebot respects that. However, for web archivists fighting Link Rot, the calculation is different.
If a newspaper goes out of business and the domain squatter puts up a Disallow: / rule, should the historical record of that newspaper be erased? Different archives have answered this differently.
The Wayback Machine Policy
Historically, the Wayback Machine strictly adhered to robots.txt, going so far as to retroactively hide decades of archives if a new owner added a block. In 2017, they changed this policy and now ignore robots.txt for retrospective archiving, though they still honor direct DMCA and removal requests.
How Archives Handle Exclusions
| Archive | Respects robots.txt? | Notes |
|---|---|---|
| Wayback Machine | Mostly No | Changed policy in 2017; still processes manual removal requests. |
| archive.today | Never | Acts like a normal user browser, aggressively ignoring all crawler directives. |
| Common Crawl | Yes | The largest open dataset for AI training strictly adheres to crawler blocks. |
robots.txt FAQ
Can robots.txt prevent malicious scraping?
No. It is a polite request, not a technical barrier. Malicious bots and many scraping tools simply ignore the file.
How does this impact Legal Citations?
If you rely on an archive that respects robots.txt, your Legal Citations could vanish overnight if the target site changes its crawler policy. This is why services like Perma.cc are vital.