How to Rotate Proxies in Python (Working Code, Both Methods)
Gateway rotation, manual pool rotation, exponential backoff, concurrency limits and block detection — a complete, runnable walkthrough.
By APIRaven Team
Rotation is the difference between a scraper that gets rate-limited after 60 requests and one that runs all night. This is how to do it properly in Python — covering both the gateway approach and manual pool rotation.
Two ways to rotate
Before writing code, pick your model. They are not interchangeable.
Gateway rotation. You connect to a single proxy endpoint with one set of credentials. The gateway decides which address your request exits from. Rotation is a server-side concern, and your code never changes.
Manual pool rotation. You hold a list of proxy endpoints and pick one per request yourself. You control the timing exactly, and you pay in bookkeeping.
Gateway rotation is simpler and is the better default. Manual rotation is worth it when you need deterministic control — for example, retrying a specific request on a specific address.
The examples below cover both.
Setup
pip install requests httpx
For the async examples you will want httpx, which supports proxy configuration natively and works cleanly with asyncio.
The simplest possible request
If your provider rotates per request, this is the whole implementation:
import requests
PROXY = "http://youruser:yourpass@PROXY_HOST:9281"
response = requests.get("https://api.ipify.org", proxies={"http": PROXY, "https": PROXY})
print(response.text)
Run it twice and you should get two different addresses back. If you don't, rotation is not enabled on that endpoint — check with your provider before building on top of it.
Confirming rotation actually happens
Do not assume. Verify:
import requests
PROXY = "http://youruser:yourpass@PROXY_HOST:9281"
proxies = {"http": PROXY, "https": PROXY}
seen = set()
for _ in range(10):
ip = requests.get("https://api.ipify.org", proxies=proxies, timeout=15).text
seen.add(ip)
print(f"{len(seen)} unique addresses out of 10 requests")
If that prints 1, you are hitting a single exit node. If it prints something close to 10, rotation is working.
Pinning an address for a session
Everything above breaks the moment you need a session to survive more than one request — a login, a cart, a multi-page form.
import requests
# Most providers expose a sticky endpoint or a session token in the username.
STICKY_PROXY = "http://youruser-session-abc123:yourpass@PROXY_HOST:9281"
session = requests.Session()
session.proxies = {"http": STICKY_PROXY, "https": STICKY_PROXY}
session.get("https://example.com/login")
session.post("https://example.com/login", data={"user": "me", "pass": "secret"})
# Same address for every request in this session
The exact sticky syntax varies by provider. On our network, rotation is the default and pinning is configured from the dashboard, so the credentials in your code stay the same either way.
Manual rotation across a pool
If you have a list of proxies rather than a gateway, wrap the selection in a small class. This keeps the rotation policy in one place instead of scattered through your scraper:
import itertools
import requests
class ProxyPool:
def __init__(self, proxies):
self._cycle = itertools.cycle(proxies)
def next(self):
proxy = next(self._cycle)
return {"http": proxy, "https": proxy}
pool = ProxyPool([
"http://user:pass@proxy-a.example:9281",
"http://user:pass@proxy-b.example:9281",
"http://user:pass@proxy-c.example:9281",
])
response = requests.get("https://api.ipify.org", proxies=pool.next(), timeout=15)
itertools.cycle gives you round-robin with no index bookkeeping. For a pool of one — which is what an IPv6 /48 gives you — you do not need this at all, because the gateway handles it.
Retries that actually help
Blind retries against a rate limit make things worse. Back off, and rotate before you try again.
import random
import time
import requests
RETRY_STATUSES = {429, 500, 502, 503, 504}
def fetch(url, proxies, attempts=4):
for attempt in range(attempts):
try:
response = requests.get(url, proxies=proxies, timeout=20)
if response.status_code in RETRY_STATUSES:
raise requests.HTTPError(f"retryable status {response.status_code}")
response.raise_for_status()
return response
except (requests.RequestException, requests.HTTPError) as exc:
if attempt == attempts - 1:
raise
# Exponential backoff with jitter: 1s, 2s, 4s (+/- 30%)
delay = (2 ** attempt) * (0.7 + random.random() * 0.6)
time.sleep(delay)
raise RuntimeError("unreachable")
Three things matter in that function: it distinguishes retryable status codes from fatal ones, it backs off exponentially rather than hammering, and it adds jitter so parallel workers do not synchronise their retries.
Concurrency without tripping yourself up
Sequential requests are slow. The usual mistake when going concurrent is firing 200 workers at once and wondering why everything returns 429.
import asyncio
import httpx
PROXY = "http://youruser:yourpass@PROXY_HOST:9281"
CONCURRENCY = 20
async def fetch(client, url, semaphore):
async with semaphore:
response = await client.get(url, timeout=20)
return response.status_code, response.text
async def main(urls):
semaphore = asyncio.Semaphore(CONCURRENCY)
async with httpx.AsyncClient(proxy=PROXY) as client:
return await asyncio.gather(*(fetch(client, url, semaphore) for url in urls))
results = asyncio.run(main([f"https://example.com/page/{i}" for i in range(500)]))
print(f"fetched {len(results)} pages")
The semaphore is the important part. It caps in-flight requests at 20 regardless of how many URLs you queued, which keeps you under the target's tolerance while still being far faster than sequential.
Tune CONCURRENCY downward until you stop seeing 429s, not upward until you start seeing them.
Detecting blocks vs. ordinary failures
A 200 response does not mean you succeeded. Anti-bot systems frequently return a challenge page with a success status code.
Check the body, not just the status:
BLOCK_MARKERS = (
"captcha",
"are you a robot",
"unusual traffic",
"access denied",
"verify you are human",
)
def looks_blocked(response):
if response.status_code in {403, 429}:
return True
body = response.text[:5000].lower()
return any(marker in body for marker in BLOCK_MARKERS)
Log how often looks_blocked returns true. If it climbs, you are sending too fast, and the fix is lower concurrency or a slower schedule — not more retries.
Putting it together
import httpx
from urllib.parse import urlparse
BLOCK_MARKERS = ("captcha", "are you a robot", "unusual traffic", "verify you are human")
def looks_blocked(response):
return response.status_code in {403, 429} or any(
marker in response.text[:5000].lower() for marker in BLOCK_MARKERS
)
def scrape(urls, proxy, concurrency=10):
ok, blocked, failed = 0, 0, 0
with httpx.Client(proxy=proxy, timeout=20) as client:
for url in urls:
try:
response = client.get(url)
if looks_blocked(response):
blocked += 1
else:
ok += 1
yield url, response.text
except httpx.HTTPError:
failed += 1
print(f"ok={ok} blocked={blocked} failed={failed}")
Keep the counters. A scraper with no visibility into its own block rate is a scraper you cannot tune.
The short version
- Prefer gateway rotation. There is less to get wrong.
- Verify rotation empirically before trusting it.
- Pin an address when a session has to survive more than one request.
- Back off exponentially with jitter on 429s.
- Cap concurrency with a semaphore and lower it when blocks start.
- Check the response body, not just the status code.
If you want to try the examples against a real IPv6 pool, the IPv6 proxy network gives you 1.2M addresses with rotation handled at the gateway — so most of the code above stays exactly as written.