When a shop blocks access

Most shops work fine. Plenty of stores are perfectly happy to be read — you paste a product link and it just works. Whether you'll ever run into a block depends entirely on the store; it's a possibility to be aware of, not something that happens everywhere.

Affiliate Egg reads live pages because these shops offer no API and no product feed to pull from. If a shop does publish a product feed, importing it with Content Egg's Feed module is cleaner and never gets blocked — reach for that first when it's an option. Affiliate Egg is for everything else.

Here's what's going on: Affiliate Egg makes a real request to the shop each time it searches for a product or refreshes a price, much like a browser. Most shops don't mind. Some, though, run anti-bot protection or limit how many requests one visitor can make — cross that line and the shop starts refusing requests, usually with a 503 or 403 error.

The kinds of blocks you might hit

• Rate limits. Sending too many requests too quickly — from bulk-adding products or frequent price updates — can get your server's IP temporarily blocked. These usually clear on their own within about 24 hours.

• Anti-bot walls. Some shops (often behind Cloudflare and similar) challenge automated visitors with captchas or JavaScript checks. These don't clear by waiting.

• A few shops won't cooperate at all. A small number are locked down tightly enough that reliable, unattended reading isn't realistic.

None of this is the norm — but when it happens, here's what to do.

What actually helps

• Use direct product URLs, not keyword search — the single biggest improvement. Far fewer requests, and none of the fragile search pages.

• Go slow. Add products a few at a time, and avoid aggressive price-update schedules.

• Ask the shop to whitelist your server's IP. You're sending them sales, so many merchants are happy to — and asking permission is good form either way. Your host can tell you the server's IP address.

• Use a scraping service for the harder cases. Your requests route through the service's rotating IPs (and can render JavaScript), which gets past most rate limits and many anti-bot walls. It's paid, but it's what makes difficult shops workable. See Scraping services.

• Set custom cookies for shops that need a region or session cookie to show prices. See Custom cookies.

What won't help

• Waiting out an anti-bot wall. Captchas and JavaScript challenges don't clear on a timer — you need a scraping service or a whitelisted IP.

• A custom parser. A parser is for shops whose data can't be read (see Is your shop supported?); it doesn't change whether a shop lets you in. If a shop blocks you, the fix is a scraping service or a whitelist, not a parser.