> For the complete documentation index, see [llms.txt](https://ae-docs.keywordrush.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ae-docs.keywordrush.com/getting-data-reliably/extractor-settings.md).
# Scraping services
The plugin integrates third-party scraping services to help bypass IP restrictions or blocks:
* [Scrapingdog](https://www.keywordrush.com/go/scrapingdog)
* Scrapeowl.com
* [Scraperapi](https://keywordrush.com/go/scraperapi)
* [Bright Data](https://www.keywordrush.com/go/brightdata)
* Crawlbase
* Decodo
{% hint style="warning" %}
These services are paid. See free-tier and pricing details below.
{% endhint %}
### Scraping Service Pricing
| Service | Free Usage | Free Allowance Renews? | Minimum Paid Plan | Pay-as-you-go |
| --------------- | ----------------------: | :--------------------: | ----------------: | :----------------------: |
| **ScraperAPI** | 1,000 credits | ✅ Monthly | $49/month | ❌ |
| **ScrapingDog** | 200 credits | ✅ Monthly | $40/month | ✅ From $10 |
| **ScrapeOwl** | 1,000 credits | ❌ One-time | $29/month | ❌ |
| **Decodo** | 2,000 standard requests | ✅ Monthly | $19/month | ❌ |
| **Crawlbase** | 1,000–5,000 credits | ❌ One-time | $99/month | ✅ $0 monthly fee |
| **Bright Data** | 5,000 requests | ✅ Monthly | $499/month | ✅ $1.50 / 1,000 requests |
**Notes:**
* **ScraperAPI:** Standard requests normally cost 1 credit, but difficult websites may consume more — Amazon specifically uses 5 credits per request.
* **ScrapingDog:** The free plan includes 200 monthly credits. Pay-as-you-go credits can be purchased without a monthly subscription.
* **ScrapeOwl:** 1,000 free credits are provided when you create an account; they do not appear to renew monthly.
* **Decodo:** The free plan provides up to 2,000 standard requests per month. Premium proxies and JavaScript rendering consume the allowance faster.
* **Crawlbase:** New accounts receive 1,000 free credits and can unlock up to 5,000 initial credits. After that, you can continue with pay-as-you-go without a subscription. Only successful requests are billed.
* **Bright Data:** The free tier includes 5,000 requests every month. After the free allowance, Web Unlocker can be used on a pay-as-you-go basis at $1.50 per 1,000 successful requests.
*Pricing and terms can change at any time — always verify current details directly on each provider's website.*
### How to Route Requests Through a Scraping Service
#### 1. Add Your API Keys
1. In your WordPress admin, go to:\
**Affiliate Egg → Settings → Extractor Settings**
2. Enter the API key for each provider you want to use.
* **Bright Data** also needs a **zone** name from your Bright Data control panel. Click **Test connection** next to the zone field to check your key and list your account's active zones.
3. Save changes.
You can enable **one or multiple providers** at the same time.\
Which provider is used for each URL is controlled by **routing rules** (see below).
#### 2. Configure Routing Rules
Routing rules tell the plugin **which scraping service** to use for specific domains or URL patterns.
1. In **Affiliate Egg → Settings → Extractor Settings**, scroll to the **Routing rules** table.
2. Click **Add rule**.
3. Fill in the fields:
* **Pattern** – the domain or URL pattern to match.
* **Provider** – select the scraping service to use.
* **Extra params (optional)** – additional query parameters for the provider API.
4. Save your changes.
Whenever a URL matches a rule, that request will be sent through the selected provider.
**Pattern Examples**
You can **match domains** or more specific URL paths. Some common patterns:
* `example.com`\
Matches the domain example.com. This is the **most common pattern** and is usually what you’ll use when creating a rule for a specific site.
* `*.example.com`\
Matches any subdomain, e.g.:
* `shop.example.com`
* `de.example.com`
* `example.com/path/*`\
Matches only URLs that start with `/path/`, e.g.:
* `https://example.com/path/product-123`
* `https://example.com/path/category/`
**Additional Parameters**
Additional parameters are simply added to the provider’s API request as query parameters.
* Each provider has its **own parameter names and supported values**.
* Use these to enable features like geo-targeting, premium proxies, or JavaScript rendering.
Below are examples for the providers currently supported.
**ScraperAPI Parameters**
You can pass these in the **Extra params** field:
* `country_code=us`\
Use US-based proxies (geo-targeting).
* `premium=true`\
Use premium residential/mobile IPs.
* `ultra_premium=true`\
Use the advanced bypass mechanism for harder sites.
* `render=true`\
Enable JavaScript rendering for dynamic pages.
**Scrapingdog Parameters**
You can pass these in the **Extra params** field:
* `country=de`\
Use German IPs (geo-targeting).
* `premium=true`\
Use premium residential proxies.
* `dynamic=true`\
Enable JavaScript rendering for dynamic pages.
**Combining Parameters**
You can combine multiple parameters using `&` just like a normal query string.
Example:
```
country_code=us&premium=true&render=true
```
Recommended settings for mercadolivre.com.br via Decodo:
```
proxy_pool=premium&headless=html&geo=Brazil&locale=pt-br
```
Bright Data also supports geo-targeting via `country` (ISO 3166-1 alpha-2), e.g. `country=de` to fetch through German IPs.
### Rule Priority and Order
Routing rules are evaluated **from top to bottom**:
1. The plugin checks the first rule.
2. If the URL matches the rule pattern, that rule’s provider (and parameters) are used.
3. If it doesn’t match, it moves to the next rule, and so on.
The **first matching rule wins**.
**Tips:**
* Put **more specific patterns (e.g. `example.com/path/*`) above** more general ones (e.g. `example.com`).
* If a URL does not match any rule, the request will be made **without a scraping provider**.
#### Custom Parameters for Scraping Services (Programmatically)
```php
// Add premium parameter to ScrapingDog requests
add_filter('affegg_parse_url_scrapingdog', function($url) {
return add_query_arg('premium', 'true', $url);
}, 10, 1);
// Add country parameter to ScrapingDog requests
add_filter('affegg_parse_url_scrapingdog', function($url) {
return add_query_arg('country', 'au', $url);
}, 10, 1);
// Add country_code parameter to ScraperAPI requests
add_filter('affegg_parse_url_scraperapi', function($url) {
return add_query_arg('country_code', 'de', $url);
}, 10, 1);
```
