Self-Hosting Web Archives
How to deploy Archivebox or pywb for personal, private web archiving.
Why Self-Host?
While public archives like the Wayback Machine and archive.today are invaluable, they have drawbacks:
- They are public, making them unsuitable for archiving private intranets, paywalled content, or personal data.
- They may honor robots.txt restrictions or DMCA takedown requests.
- They can disappear or go offline.
Self-hosting guarantees that you maintain custody of your own data in standard WARC formats.
Legal Warning
Self-hosting archives of paywalled or copyrighted content is generally protected under personal fair use, but distributing those archives publicly is not. See our guide on Archiving Paywalled Content.
Recommended Tools
1. ArchiveBox
ArchiveBox is the most popular solution for personal web archiving. It takes URLs and automatically generates multiple formats: HTML, PDF, PNG screenshots, and WARC files.
Deployment (Docker):
docker run -v $PWD/data:/data -p 8000:8000 archivebox/archiveboxArchiveBox is excellent for broad, automated capture of articles and bookmarks to prevent link rot.
2. pywb (Python Wayback)
If you want to run your own Wayback Machine, pywb is the software that powers large institutional archives. It excels at high-fidelity replay of complex Javascript applications.
pywb requires more configuration than ArchiveBox but provides a much more authentic "time travel" experience, seamlessly routing requests for assets through your stored WARC files.
| Feature | ArchiveBox | pywb |
|---|---|---|
| Ease of Use | High (Web UI) | Moderate (CLI focus) |
| Formats | PDF, PNG, WARC, HTML | WARC replay only |
| Fidelity | Good for static text | Excellent for SPAs/Interactive |
Self-Hosting FAQ
How much storage do I need?
It depends entirely on your habits. A typical news article might consume 2-5MB in a WARC file. Archiving heavily interactive sites or video content can consume gigabytes rapidly.