Linux Commands & Shell Skills
wget: The Downloader That Never Sleeps
wget is curl for jobs measured in hours: mirrors, archives, recursive pulls, and cron batches that run while nobody watches. It recurses, honors robots.txt, resumes interrupted transfers by default, and paces itself with rate limits and wait intervals, which is a politeness curl leaves to the caller. Between the two tools the division is clean: wget for bulk and patience, curl for precision, and a team that knows both rarely needs a third installer.
Key points
- Mirroring is one line: wget --mirror --convert-links --adjust-extension --no-parent https://docs.example.com; omit --no-parent and a documentation archive quietly becomes a crawl of everything above it.
- Resume and retry for free: wget -c picks an interrupted transfer back up, and --tries=10 --waitretry=5 rides out servers that hiccup mid-bulk instead of failing the job at the first twitch.
- Politeness has flags: --limit-rate=200k, --wait=1, and --random-wait keep a mirror from tripping upstream rate limits or a shared-server abuse clause; shared hosting plus an unthrottled crawler is a short friendship.
- robots.txt is respected by default; -e robots=off overrides it deliberately, and using that override against someone else's production site is how bulk downloads turn into incident tickets.
- wget -O /dev/null -nv https://host/health is a one-line monitoring probe whose exit codes carry the diagnosis: 4 is a network failure, 8 is a server error, and both beats a silent cron.
- Conversion matters after the fact: --convert-links rewrites a mirrored site for offline navigation, and --adjust-extension saves CMS-generated pages under extensions local tools can serve.
- Bulk jobs in cron get lock files, or overlapping runs write into each other's half-finished files while --waitretry quietly stretches the job past its window; wget has no internal mutex.
- Know the boundary: forms, cookie-heavy sessions, and API endpoints belong to curl, and JavaScript-built pages mirror as empty shells because wget follows links, not scripts.
Field note
One mirror without --no-parent wandered into forty gigabytes of a client's media library overnight and nearly filled the disk. The politeness flags are set once and copied forever after.
Frequently asked questions
Why does my mirror contain only one page?
Scope flags did the work someone meant to undo: robots.txt restrictions, --no-parent, a depth limit, or the site being JavaScript-rendered. wget follows links it can see; check -o wget.log before blaming the tool.
When should I pick wget over curl for a single download?
For anything resumable or cron-reliable: -c continuation, built-in retries, and timestamping (--timestamping) make wget the low-attention span choice; curl gives you the same only if you assemble it yourself.
Is it legal or polite to mirror a public site?
Ownership decides legal, robots.txt and load decide polite: honor the robots file, throttle to a human-ish rate, and prefer an official export or API the moment the target offers one.
Why do some sites return 403 to plain wget?
User-agent filtering, usually intentional; --user-agent='Mozilla/5.0' gets you through, but a filtered 403 also means the operator is watching, so a dialogue or an API is the better next step.
How Kinetikatech helps with linux commands & shell skills
We've handled this hundreds of times: The commands and scripting habits that decide whether an administrator operates a server or fights it: search, transform, transfer, and supervise text and processes with the tools that live in every base install. With Linux Server Security and Server Management, you get the full loop, we assess, we implement, we watch it around the clock, and we pick up within 30 minutes when it matters.