Obscura is not. It is a browser engine written from scratch in Rust that embeds V8 for JavaScript, ships no Chromium and no Node.js, and claims to run in about 30 MB of RAM. I already put its fingerprint on the bench once, in THE LAB #111, where it scored well in a controlled probe. A lab score is one thing. What I wanted this time was to point it at a real anti-bot on a real site and see what a featherweight engine actually buys you.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

The site is Radisson. Their booking pages hide an availability API that returns every room rate for a hotel and date range, and that API sits behind Akamai Bot Manager. If you can talk to it, you can do something useful with it. You can track a single hotel’s nightly price over the next few months and find the cheapest and most expensive days to go. I picked Munich, because Oktoberfest 2026 runs from September 19 to October 4, and nothing moves a city’s hotel prices like a festival that pulls millions of visitors into one place.

So there are two questions stacked on top of each other. Can Obscura clear Akamai at all? And if it can, is a 30 MB engine a sensible way to mint the session that a fast HTTP client then reuses? That second pattern is not new. In THE LAB #100 I built exactly this hybrid, one browser login feeding thousands of curl_cffi requests, but with Camoufox, a real Firefox. So this is really a hands-on review of Obscura for one concrete job. Can the cheap engine take Camoufox’s place, and where it cannot, is that the tool’s fault or the target’s? That second question turned out to be the interesting one.

The tools on the table

Three pieces do the work.

Obscura is the from-scratch Rust engine. Because it implements its own DOM and Web APIs around V8 rather than shipping a real browser, it is small and fast, and that same design is the source of its quirks. It exposes a Chrome DevTools Protocol server, so we can drive it with Playwright over connect_over_cdp exactly like a headless Chrome. Its --stealth flag turns on a consistent spoofed fingerprint, which is the feature under test.

curl_cffi is the replay layer. It is a Python HTTP client that impersonates a real browser’s TLS fingerprint (JA3/JA4, HTTP/2 settings, header order) without running a browser at all. When TLS fingerprinting is the wall, it walks through it in milliseconds. When JavaScript execution is the wall, it cannot help. Knowing which wall is which is most of the job.

The third piece is the fingerprint bench from #111, which reads a browser’s own fingerprint surface from a local probe page and scores it out of 100. I reused it here to get a fresh, reproducible number for Obscura before sending it into the fight.

Two more browsers turn up later as controls, both from that same bench and both scoring a clean 100. CloakBrowser is a Chromium patched at the C++ level, and Camoufox is a Firefox fork with a coherent injected fingerprint. I ran the same proxy test through them to separate a tool problem from a proxy problem, and all three browsers gave the identical result.

How Radisson is actually protected

Before running anything, it pays to model the target. Radisson does not protect everything the same way, and the whole strategy depends on that split.

The HTML pages, like the room-display page you land on after picking a hotel and dates, come back fine to a good TLS impersonation. A plain curl gets nothing, HTTP 000, because its TLS handshake does not look like a browser. curl_cffi with a Chrome profile gets HTTP 200 and 380 KB of real HTML. So the first tier is TLS fingerprinting, and curl_cffi alone handles it.

The XHR APIs are a different story. Two matter here. content-api/hotels/{code}/rooms returns a hotel’s bookable rooms, and availability-api/stays-by-rooms returns the rates for a set of rooms on given dates. Both sit behind Akamai Bot Manager. A request with no cookies gets HTTP 403 with a 421-byte body. Akamai wants its cookie set first: _abck, bm_sz, ak_bmsc, bm_sv, bm_so, bm_s. Those cookies are minted by the Akamai sensor script, a chunk of obfuscated JavaScript that runs on page load, collects a pile of signals, and posts them back. Only after that handshake does _abck reach a validated state and the API start answering.

There is a third layer that shows up when you push too hard. Load the wrong path, or hit the site too often from one address, and Akamai serves a small interstitial instead of the page. It is the sec-cpt behavioral challenge, with a sec-if-cpt-container div and a friendly “Powered and protected by Akamai” line. That page runs its own script and expects the client to solve it before moving on.

So the layer we target is clear. curl_cffi covers the TLS tier and the raw HTTP replay. The one thing it cannot do is run the Akamai sensor to mint _abck. That is the job we hand to Obscura, and it is the exact point where a from-scratch engine either earns its place or does not.

Check the TWSC YouTube Channel

Testing the fingerprint first

There is no reason to send a browser at Akamai if its fingerprint falls over on the easy tests, so I scored it first.

We ran Obscura 0.1.10 headless with --stealth, driven through Playwright over CDP, against the #111 local probe and then against the public test pages people actually cite. The probe reads the browser’s own surface, and here Obscura looks the part. navigator.webdriver is false, on both the prototype and the instance. The user agent is Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... Chrome/145.0.0.0 Safari/537.36, a plausible Windows Chrome with no HeadlessChrome token. window.chrome is complete, with runtime, csi, loadTimes, and app all present. navigator.userAgentData reports Chrome 145 and Chromium 145. The WebGL unmasked renderer is a spoofed ANGLE (NVIDIA, NVIDIA GeForce RTX 30...). There are five plugins, a Europe/Berlin timezone, and no automation residue at all: no leaked CDP globals, no driver keys.

The score came out at 94 out of 100. That puts Obscura above every Chromium CDP tool in the bench, which sit at 70 because their user agent still carries a headless token, and well above vanilla Playwright and Selenium at 35. The one thing that costs it points is the native-code check. When you ask whether a Web API function is really native by inspecting its toString, a few of Obscura’s functions give themselves away as patched. That is the single tell, and the bench labels it “patched fns”.

The public suites are where it gets more interesting, and more honest. On Sannysoft, Obscura passes cleanly, 21 checks green and none red. But CreepJS never finishes. The page loads to about 8 KB and then stops, with no trust score ever computed. Pixelscan renders its landing chrome but never produces an automation verdict. iphey renders too, but the verdict banner it normally paints, trustworthy or suspicious, never populates. The two oracles from #111, BrowserScan and deviceandbrowserinfo, were inconclusive for the same reason. browserleaks is the exception that proves the point: its canvas and WebGL pages are light enough that Obscura runs them, and the WebGL page confirms the same spoofed NVIDIA GPU the probe reported.

The split is the interesting part. Obscura sails through the boolean checks, the ones that ask a single yes-or-no question about a property. It stalls on the heavy suites, the ones that run thousands of lines of fingerprinting JavaScript to completion. Those suites cannot flag it, but only because they never finish running inside the minimal engine. Hold that thought, because Akamai’s sensor is exactly that kind of heavy script.

Building the hybrid, Obscura mints and curl_cffi replays

The plan follows #100. Use the browser only to mint a valid Akamai session, then hand the cookies to curl_cffi and let it hammer the availability API for every date we care about. The code lives in our repository under obscura_radisson_deals.

The session layer is session.py. It launches Obscura as a CDP server with --stealth, connects Playwright, and loads the room-display page for the target hotel. Then it harvests the cookies and checks them by calling the content-api rooms endpoint. If that call returns 200, the session is good and we also get the room ids we need for scanning. If not, it reloads the page and tries again. This retry loop is not decoration. It is there because of what we found next.

def mint(self, attempts=6, settle_s=8.0):
    for attempt in range(attempts):
        if attempt == 0:
            self._page.goto(self.page_url, wait_until="load", timeout=45000)
        else:
            self._page.reload(wait_until="load", timeout=45000)
        time.sleep(settle_s)
        self.cookies = self._harvest()
        probe = resolve_rooms(self.cookies, self.hotel_code, self.currency, self.proxy)
        if probe["ok"]:
            self.room_ids = probe["room_ids"]
            return True
    return False

From a clean residential IP it works, and it works fast. On a good load, with bm_sv and bm_mi present in the jar, we took the harvested cookies straight to curl_cffi with a Chrome 131 TLS profile. The availability-api/stays-by-rooms endpoint answered HTTP 200 with hundreds of kilobytes of real rate JSON. Against the Munich airport hotel the first load minted a valid session and resolved its rooms on attempt zero. That is the whole thesis in one response. The 30 MB engine minted a session a fast HTTP client could reuse.

It is not always the first load. Sometimes the handshake does not complete, _abck stays in its unsolved state with a -1 in the second segment, and the API answers 403. Obscura harvests nine cookies on those loads too, but without bm_sv and bm_mi the session is not valid, so mint() reloads until one lands. On a trusted IP a reload or two is enough. The reason it can miss at all is the same trait we saw on CreepJS: the minimal engine runs the Akamai sensor part of the way and then not. Camoufox, a real Firefox, does not have that wobble.

There is a second signal worth recording. When we called the API from inside Obscura’s own page context instead of replaying externally, it did not return 403. It returned 500, an application error, which means the request cleared the Akamai wall and reached the backend. A 500 is not a bot block. So the in-browser path can get through too. It just needs the request shaped exactly the way the site’s own code shapes it, and on the room-display page the SPA kept navigating and tearing down our injected fetch before it settled.

You can find the full code used in this article in The Lab repository, available only to paying readers, in the folder “113.Obscura”.

The wall is the IP, not the fingerprint

Read more

Never imagined that someone would have read it. It was just an excuse for writing down my scraping notes in a more structured way.

Four years later, both TWSC and the industry look very different.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

Health of The Web Scraping Club

First of all, thank you.

Thank you to all the readers who think I’m saying something interesting enough to be subscribed to.

In this era, when we’re all fighting for attention, having 7,000+ people who choose to receive this newsletter and open it every week seems like a miracle.

This is despite no marketing efforts, 0 ads, and an issue with Google SEO that made the whole domain disappear from the results from one day to the next.

In the next few days, to solve this issue, there will be a new domain address that, according to my experiments, should fix this.

Let me also thank you, all the backers, paying readers, and companies that make this possible.

While still very lean, The Web Scraping Club in its current format could not run without sponsors and readers who choose to support it. And about the companies backing TWSC, let me spend a special word on the NetNut team, who are having a hard time now. I’ve met great people there, and they were among the first believers in this newsletter.

Behind the scenes, TWSC is far from perfect, but I’m doing my best to keep things running smoothly.
I’ve just launched a partner portal, where sponsors can monitor their campaign. For the readers, this year we’ve released the new version of the Proxy Price benchmark tool, which I promise will be updated more often in the future. Also, the Scraping Wiki, our repo with all the findings of The Web Scraping Club organized as a second brain, is a recent new entry. Please give it a look and share your feedback.

What’s next

Apart from fixing the Google SEO issue in the immediate term, I’ve got a couple of ideas that still need time to be implemented, but hopefully they’ll see the light before the end of the year.
Something I’ve been asked for a long time is a Job Board dedicated to open scraping positions and the great talent to fill them.
Is this something you’d like to have here?

Ideally, I’d like to revamp the YouTube channel. And last but not least, I’m working on a web-scraping event in Italy in October. More details soon, but be ready for it!

Check the TWSC YouTube Channel

How the industry changed

The industry over the last four years looks very different.

We got LLMs; we know it, so the coding part is way faster. But what they brought to the industry is the legitimacy of web scraping.

It seems the world has recognized the role of web data and browser automation in innovation. While web data was once seen as something shady for spying on your competitors, it is now the main moat for training LLMs, and browser automation is the key tool for giving your agents real-time data.

This changed the perception of the industry as a whole, and the recent funding rounds (Nimble, Oxylabs) are the proof.

Think about it: have you ever heard of a funding round for a proxy company before 2023?

Conclusion

It’s an exciting time for the industry, and I’m glad to work at The Web Scraping Club right now. I’ll be at conferences around Europe (Oxycon in September and the Extract Summit in Dublin), which will allow me to meet great professionals and learn about the latest trends.

While both TWSC and the industry have changed a lot over the years, the spirit of this newsletter hasn't: no marketing fluff, honest reviews, some hot takes, and my personal opinions. I know that in four years, not all articles can be a 10 out of 10, but I can assure you that every article written here was the result of sweat and hard work. And it will be the same for the future.

That’s where the MarkItDown Python library comes into play!

Here, I’ll show you everything you need to know about MarkItDown, from installation to practical examples, so you can convert HTML pages, PDFs, Word documents, Excel spreadsheets, and many other data sources into LLM-ready Markdown.

Why the Markdown Data Format Is More Relevant Than Ever

As I covered in a recent post, AI-ready web scrapers, RAG pipelines, and agent frameworks all have one thing in common: they rely heavily on Markdown!

That’s no coincidence... Mainstream LLMs (including OpenAI’s latest models) natively “speak” Markdown and often generate Markdown-formatted responses without being prompted.

Why? Because Markdown strikes the perfect balance between plain text and structured documents. It preserves headings, lists, tables, links, code blocks, and other semantic elements without the overhead and noise of HTML.

As a result, LLMs can understand and process Markdown efficiently while consuming far fewer tokens. This translates directly into lower inference costs and leaves more of the context window available for meaningful content.

In my experience (and as I demonstrated in a benchmark I conducted for Bright Data), Markdown has become the natural format for feeding documents, especially content extracted from the web, into LLMs.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

Why Markdown Conversion Matters in Web Scraping Pipelines

Web data is rarely the final destination. Quite the opposite, scraped content generally represents the starting point of a larger data analysis pipeline, where the goal is to extract insights, enrich datasets, or power downstream applications.

In these workflows, the information you need tends to go beyond traditional web pages. PDFs, Word documents, videos, presentations, and other files can provide additional context required for accurate analysis and processing. These resources may come from the web and be collected during scraping, or they may be added later.

Regardless of the original format, data is increasingly processed by AI systems. Since LLMs naturally work well with Markdown, converting different sources into a clean, structured Markdown representation can greatly simplify your data pipelines.

That’s why you need a multi-format Markdown conversion library in your toolkit, such as MarkItDown!

The new anti-bot solution - Toughest walls, lowest price. Claim your free 10 000 requests with coupon code WSCLUB

Claim your credits

MarkItDown: A Complete Overview

Let me now introduce you to MarkItDown, one of the most popular libraries for converting multiple input formats into Markdown.

What Is MarkItDown?

MarkItDown is an open-source Python library for converting a wide range of documents and other data sources into Markdown. The project is developed and maintained by Microsoft.

It’s quickly become one of the most popular tools in its category (more on this later in the post), with over 165k GitHub stars, over 12 million monthly downloads, and a thriving community.

MarkItDown supports dozens of input formats, ranging from Office documents and PDFs to images, audio files, web pages, YouTube videos, and structured data formats such as JSON, XML, and CSV.

Key Features

The main capabilities supported by MarkItDown are:

• Broad format support: Converts dozens of input formats, including PDFs, Office documents, HTML, images, audio, EPUBs, ZIP archives, YouTube URLs, and structured data files.

• Python API: Offers a clean, developer-friendly API for integrating document conversion into Python applications.

• Native CLI: Includes a rich CLI for converting files directly from the terminal, making it ideal for scripting, batch processing, and DevOps workflows.

• Plugin architecture: Supports third-party plugins, allowing the community to extend the library with custom converters and additional processing features.

• AI-powered processing: Integrates with LLMs to generate image descriptions, perform OCR, and transcribe audio, enriching the resulting Markdown.

• Flexible dependency management: Lets you install only the converters you need, reducing dependencies and keeping deployments lightweight.

• MCP server support: Provides a lightweight Model Context Protocol server that allows AI agents to call MarkItDown as a tool.

• Azure AI integration: Works with Azure Document Intelligence and Azure Content Understanding for higher-quality OCR, layout analysis, and structured field extraction.

Supported Input Formats

Installation and Setup

Follow the steps below to add MarkItDown to your system.

Prerequisites

MarkItDown requires Python 3.10 or higher. It’s also recommended to install it inside a Python virtual environment.

Installation via pip

The easiest way to install MarkItDown is through the PyPI markitdown package with:

pip install 'markitdown[all]'

Note that [all] option adds all optional dependencies required to support every available input format.

If you only need support for specific file types, you can install the specific dependencies individually for more control. For example:

pip install 'markitdown[pdf, docx]'

This adds only the extra dependencies required for PDF and Word file conversion.

Available optional dependencies include:

• [all]: Installs all optional dependencies.

• [pptx]: Adds support for PowerPoint files.

• [docx]: Adds support for Word documents.

• [xlsx]: Adds support for Excel files.

• [xls]: Adds support for older Excel files.

• [pdf]: Adds support for PDF documents.

• [outlook]: Adds support for Outlook messages.

• [audio-transcription]: Adds support for WAV and MP3 audio transcription.

• [youtube-transcription]: Adds support for YouTube video transcription.

• [az-doc-intel]: Enables integration with Azure Document Intelligence.

• [az-content-understanding]: Enables integration with Azure Content Understanding.

Note: For installation from source or Docker-based usage, refer to the official MarkItDown docs.

MarkItDown in Action: Full Examples

Time to explore some practical examples of converting different input data formats into Markdown.

HTML Document to Markdown

Suppose you have an HTML page stored locally (e.g., as part of an offline web scraping procedure). You can convert it to Markdown using the MarkItDown CLI:

markitdown example.html -o example.md

Alternatively, you can achieve the same result via the Python API:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("example.html")

with open("example.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

Now, assume the input is the following HTML page (i.e., the homepage from example.com):

The resulting example.md file will contain:

Notice how MarkItDown focuses on the meaningful HTML content, ignoring tags like <style>.

The library extracts the relevant elements and converts them into a structured Markdown representation, keeping important information such as headings, links, and text formatting while removing unnecessary HTML overhead.

Web Page to Markdown

A particularly useful feature for web scraping workflows is that MarkItDown can also fetch and convert web pages directly from their URLs, eliminating the need to manually save the HTML first.

Just pass the URL of the target page directly to the MarkItDown CLI:

markitdown 'https://example.com' -o example.md

Or to its Python API:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("https://example.com")

with open("example.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

Under the hood, MarkItDown performs an HTTP GET request to the specified URL using Requests, retrieves the page HTML, and converts the resulting content into Markdown. As usual, keep in mind that the request may fail if the target website uses anti-bot protections.

The output will be the same example.md Markdown file shown in the previous example.

Check the TWSC YouTube Channel

PDF to Markdown

Many websites store valuable information in downloadable PDF files (e.g., shopping flyers, business reports, whitepapers, product documentation, etc.).

While you can send these files directly to an AI model, doing so often consumes unnecessary context and tokens because PDFs may contain complex layouts and additional formatting overhead.

MarkItDown converts PDFs into clean Markdown while preserving important elements such as headings, document structure, lists, and tables, making the content easier for LLMs to process.

Assume your input is the sample PDF below:

Convert it to Markdown with the MarkItDown CLI:

markitdown sample.pdf -o sample.md

Or in Python:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("sample.pdf")

with open("sample.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

The result will be this sample.md file:

As you can see, MarkItDown preserves all the meaningful content from the PDF.

Excel to Markdown

Now, suppose you have relevant internal data stored in an Excel spreadsheet and want to pass it to an LLM for analysis or processing:

Convert it to Markdown through the MarkItDown CLI:

markitdown spreadsheet.xls -o spreadsheet.md

Or use the Python API:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("spreadsheet.xls")

with open("spreadsheet.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

The result will be a Markdown file containing a table with the same data as the original spreadsheet:

Or, for a better visual representation, through the Markdown viewer:

Need help with your scraping project?

MarkItDown Plugins

MarkItDown also supports third-party plugins that extend its built-in capabilities with additional converters and processing features. You can discover available plugins by searching GitHub for the #markitdown-plugin hashtag:

To list all installed plugins, run:

markitdown --list-plugins

Plugins are disabled by default. To enable them from the CLI, add the --use-plugins flag:

markitdown --use-plugins <SOURCE_FILE_PATH> -o <OUTPUT_FILE>

If you’re using the Python API, enable plugins by setting the enable_plugins parameter to True:

from markitdown import MarkItDown

md = MarkItDown(
  enable_plugins=True,
)

Once enabled, MarkItDown automatically loads all installed plugins.

MarkItDown Alternatives: Other Document-to-Markdown Libraries

MarkItDown isn’t the only library for converting documents to Markdown. Other popular libraries for converting different types of data into Markdown include:

Conclusion

Markdown is the language of LLMs and, as a result, the preferred format for AI-powered data processing and analysis pipelines. In this post, I covered how to use MarkItDown to convert web pages, HTML documents, PDFs, Word documents, Excel spreadsheets, PowerPoint presentations, YouTube videos, and many other file formats into clean, AI-friendly Markdown.

You now know how to get the most out of MarkItDown to prepare data for AI workflows, regardless of whether it comes from web scraping, downloaded files, or local documents.

I hope you found this guide useful. If you have any questions or comments, feel free to leave them below. Thanks for reading, and see you in the next one!

FAQ

How to set a proxy in MarkItDown?

MarkItDown doesn’t provide a dedicated proxy configuration option. Still, it relies on the Python requests library internally, which supports proxy configuration through special environment variables. Thus, you can set a proxy in MarkItDown with:

export HTTP_PROXY="http://user:password@your-proxy-server.com:port"
export HTTPS_PROXY="http://user:password@your-proxy-server.com:port"

Or, directly in the Python code with:

import os
from markitdown import MarkItDown

# Set your proxy addresses (replace with your proxy details)
os.environ["HTTP_PROXY"] = "http://user:password@your-proxy-server.com:port"
os.environ["HTTPS_PROXY"] = "http://user:password@your-proxy-server.com:port"

md = MarkItDown()
# ...

How to deal with 429 errors when converting YouTube videos to Markdown with MarkItDown?

Try to convert a YouTube video to Markdown, such as with:

markitdown "https://www.youtube.com/watch?v=1wtQAoBZLTg" -o video.md

The request is likely to fail with 429 Too Many Requests errors due to YouTube anti-bot protections. This can happen even with a fresh IP address. To improve reliability, try to route MarkItDown requests through a high-quality residential proxy.

How to use the MarkItDown MCP server?

The MarkItDown MCP server exposes a single tool that allows AI agents and MCP-compatible applications to convert different data sources into Markdown. The tool is convert_to_markdown, which accepts http:, https:, file:, and data: URIs.

Install the MarkItDown MCP server with:

pip install markitdown-mcp

Then, connect it to an MCP-compatible client, such as Claude Code, Claude Desktop, Codex, Gemini CLI, etc.

As AI usage is growing more and more every day, you’d like an AI-powered tool that can improve your scraping productivity, don’t you?! Especially when it comes to managing selectors, and DOM structure changes…

Well, there’s good news for you: this is the exact gap Bright Data Scraper Studio is designed to close!

This article discusses what Bright Data’s Scraper Studio is, why it exists, and how to use it with a step-by-step tutorial.

Let’s get into it!

What Scraper Studio Actually Is

Scraper Studio is a cloud-hosted, end-to-end scraping environment running entirely on top of Bright Data’s infrastructure. The tool is designed from the ground up with scraping primitives in mind, as it provides built-in proxy rotation, native JavaScript rendering, anti-bot handling, and direct integrations with data delivery targets like Amazon S3, Snowflake, Google Cloud, and Microsoft Azure.

The platform offers two build modes that sit on the same underlying scraper runtime:

• AI agent mode: You provide a target URL and a plain-English description of the data you want to extract. The AI agent visits the page, infers a structured output schema, and then generates the full scraper code for you. At the end of this automated process, you review and approve the schema before code generation begins. This gives you a checkpoint to catch misaligned fields before they propagate into your pipeline, allowing you to change them.

• Full IDE mode: A full browser-based JavaScript code editor with debugging tools for when you need direct, line-level control over the scraper logic. This is the mode you can use for when you’re dealing with multi-stage scrapers, custom pagination logic, complex state management between page navigations, or sites with particularly aggressive anti-bot stacks that need fine-tuned handling.

What’s more important is that both modes produce the same type of scraper. This means that you’re never locked into one approach: a scraper built in the AI Agent mode can be opened and edited in the IDE at any time, and any scraper can be updated using the Self-healing tool.

Scraper Studio Key Features

Let’s take a close look at Scraper Studio’s features:

• Built-in proxy infrastructure: Scraper Studio runs on Bright Data’s proxy network which, among other types, provides users with 400+ million residential IPs across 195 countries, with 99.99% uptime. For scraping professionals, this means no proxy pool management and, more importantly, no dealing with burned IPs.

• Anti-bot bypass, natively integrated: JavaScript rendering, fingerprinting evasion, and bot detection bypass are built into the runtime. This is significant because it means your scraper code doesn’t need to account for these concerns. You write the extraction logic, and the platform handles the unblocking automatically.

• AI-powered self-healing: This is arguably the most operationally relevant feature for running scrapers long-term. When a target site changes its structure, and your scraper breaks, the self-healing tool lets you describe the fix in plain English. The AI analyzes the current code, proposes changes with a diff view, and you approve or reject them. This means that you remain in control of the overall process, as changes aren’t auto-applied unless you explicitly enable auto-approval

• Scheduled runs and data delivery: Scraper Studio includes a scheduling layer, so you can configure periodic runs without external orchestration. Combine this with native delivery to S3, Snowflake, GCS, or Azure, and you have a complete data pipeline that starts from scraping and ends with structured output in your data warehouse, with no spaghetti code required.

• CLI and agent integration: If you prefer working outside the browser UI, scrapers can be built and triggered via the Bright Data CLI, and the AI Agent mode is accessible programmatically, currently compatible with tools like Claude Code, Cursor, and Codex. This means you can integrate Scraper Studio into existing developer workflows without being locked into the web interface.

• Wide pricing offer: The pricing offer may seem like a secondary theme, but it’s not. First and foremost, Bright Data offers you 5k page loads for free every month, backed by their expert support if you need it. The other pricing options are:

  • Pay-as-you-go: $1.5 per 1k page loads, and you pay only for successful requests.

  • Scale: $400/Mo that includes about 400k page loads per month and more.

  • Enterprise: A custom pricing tier for enterprises, based on their specific needs.

Scraper Studio Best Use Cases

If you’re already familiar with Bright Data products and offers, you may be asking why use Scraper Studio. Below is a list of practical examples when this is the right tool to use:

• No APIs or ready-to-use datasets: If you need to scrape a site that isn’t available on Bright Data’s pre-built Datasets Marketplace or on its web scraping APIs, and you want to own the scraper logic yourself but not the infrastructure, well, Scraper Studio is your best choice.

• Recurring pipelines: When you’re building a recurring data pipeline and don’t want to manage cron jobs, deployment infrastructure, or proxy pools separately.

• Quick prototypes: The AI Agent mode can go from a single URL to a working scraper in under 10 minutes. This makes it genuinely useful for validating whether a scraping approach is feasible before committing engineering time on a larger scale.

• You’re in a team with mixed technical levels: The AI Agent is accessible to non-engineers, and the IDE gives developers full control. They both produce the same type of scraper, deployable on the same infrastructure, and as it is interchangeable, your team can manage the same project with different technical levels.

How to Build a Web Scraper With Scraper Studio Using the AI Agent Mode

Time to take the theory into practice. In this section, you’ll learn how to create a web scraper with Scraper Studio using its AI agent mode. The target URL for this tutorial is from the Kappa’s website:

The scraper will retrieve the data from this target page, using the constraints via the prompt description.

Follow the next steps to build your scraper!

Step #1: Create a New AI Web Scraper Project

After creating a new Bright Data account, go to the Scrapers tab and click on Develop a web scraper under New:

You’ll be redirected to a page where you can insert the target URL and the prompt. For this tutorial, the prompt is the following:

Extract the product name, price, available variations / colors, sizes, image URLs, product description, product details.

Below is the visual result:

Note that this is also where you can immediately go for the IDE mode by clicking on Open IDE.

When you’re ready, click on Start scraping with AI, and Scraper Studio will make its magic!

Step #2: Analyze and Approve the Output Schema

The system will automatically produce the output below:

This is the output schema. It’s a structured list of data fields the scraper will extract, along with their data types. Compare this with the data in the target page, and click Approve if the output matches your expectations.

After your approval, Scraper Studio will start creating the web scraper. When the process is done, you’ll receive the following pop-up:

You’ll also get a notification email when the scraper is ready:

All right, your scraper is ready for data retrieval.

Check the TWSC YouTube Channel

Step #3: Manage Preferences and Retrieve The Data

When the scraper is ready, this doesn’t mean the data is scraped. Before scraping the data, you have to manage your preferences.

First of all, go into your scrapers and navigate to the Delivery preferences tab:

Here is where you can choose among:

• Data format for local download, like JSON, NDJSON, CSV, or XLS.

• Third-party connections like Amazon S3 bucket, AWS, Snowflake, and more.

• Get notified via email based on the status of the data delivery.

If you want a scraper that starts recurrently, navigate to the Subscriptions tab and set your schedule:

If you just want to test things around or if you need a one-time scraper, the Initiate manually tab is already sufficient. You just need to click on Start, and the scraper will begin retrieving the data:

Very well! You made it to retrieve data with your first scraper made with Scraper Studio.

Step #4: Check Runs and Go to IDE Mode

To check the status of each scraping run, go to the Runs section. This will display all the information you need on each data retrieval performed by the scraper:

The Actions section allows you to download the data in different formats. Below is the result in JSON:

[
  {
    "product_name": "Blue Men's Trackhouse Moto GP Team Motorcycle T-Shirt with Contrast Inserts",
    "product_subtitle": "AMIRILORE TRACKHOUSE Blue Diva-Grey Antrachite-NEON Yellow",
    "brand": "KAPPA",
    "price": {
      "value": 67,
      "currency": "USD",
      "symbol": "$"
    },
    "sku": "Style: 322J6HW - Color: A02",
    "available_sizes": [
      "XS",
      "S",
      "M",
      "L",
      "XL",
      "2XL",
      "3XL",
      "4XL",
      "5XL",
      "6XL"
    ],
    "image_urls": [
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02_5f5f425b-48d1-4181-af4c-ce863229cbf4.jpg>",
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02_aa19b0fa-ca41-4894-ad74-8d408cc31725.jpg>",
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02.jpg>",
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02_8bd76dc9-0550-48f9-a974-e3c38a4b707e.jpg>"
    ],
    "description": "Product Description GARA. T-shirt for man in polyester fabric. REGULAR FIT. Short sleeve, roundcollar, insert on shoulders in contrast color, sides and shoulders inserts. Omini logo, Team personalizations and sponsors sublimated. TRACKHOUSE.",
    "product_details": "Description Active Jerseys Gender Man Season Spring Composition Main Fabric: Recycled Polyester 100%",
    "input": {
      "url": "<https://www.kappa.com/products/active-jerseys-man-amirilore-trackhouse-blue-diva-grey-antrachite-neon-yellow-322j6hw-trh-a02>"
    }
  }
]

As you can see, the results are exactly how’d you expect.

To change the mode to the full IDE one, navigate to the Code tab:

This is where you can write your custom JavaScript to modify anything you need.

Terrific! You completed your first scraping project with Bright Data’s Scraper Studio.

Need help with your scraping project?

Comparison Table: Scraping via Custom Code Vs Using Scraper Studio

The table below summarizes the differences in scraping the same target using a scraper you coded yourself or using Scraper Studio:

Conclusion

In this article, you discovered what Bright Data’s Scraper Studio is, its features and best use cases, and how easy it is to use it.

As you’ve learned, Scraper Studio is an ideal solution for building a web scraper without any infrastructure overhead, but also without managing selectors manually. Thanks to it, you don’t have to deal with common challenges like infrastructure maintenance, proxy rotation, and anti-bot bypass, but also with DOM changes.

In a matter of a few minutes, you can go from a prompt written in natural language to a production-ready scraper, with automated data delivery to Amazon S3, Snowflake, and many other destinations, and with the possibility to schedule your scrapers and manage DOM changes without manual overhead.

So, let us know in the comments: have you ever tried Scraper Studio? What’s your experience with it?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Retrieval Augmented Generation (RAG) was designed to solve exactly such limitations. By decoupling what the model knows from what it can access at query time, RAG gives LLMs the ability to reason over fresh, domain-specific, and proprietary knowledge without any retraining. In theory, this closes the knowledge gap entirely. In practice, the quality of a RAG system is bound entirely by the quality of its retrieval layer, and this is where most implementations break down.

Indeed, many teams think that the solution is already offered by web search tools that, nowadays, are integrated in all commercial LLMs. This article makes the case for why that is the wrong call and why web scraping is the right retrieval layer that RAG actually requires.

Let’s get into it!

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

What is RAG?

Retrieval Augmented Generation (RAG) is an architectural pattern in natural language processing that augments the inference pipeline of a generative large language model (LLM). It does so by dynamically injecting externally retrieved, contextually relevant information into the prompt at query time. In simple terms, rather than relying exclusively on knowledge encoded in model weights during training, RAG decouples parametric memory (what the model learned) from non-parametric memory (an external, queryable knowledge store), combining both at runtime. This produces models’ outputs that are accurate and grounded.

The canonical RAG pipeline operates in two distinct phases:

1. Retrieval phase: Given a user query q, a retriever component encodes q into a vector embedding database and performs approximate nearest-neighbor (ANN) search against a pre-indexed corpus of document chunks.

2. Generation phase: The retrieved passages are concatenated with the original query and injected into the LLM’s context window as an augmented prompt. The model, then, conditions its output token generation on both its parametric weights and the retrieved evidence, producing a response that is grounded in the supplied documents.

Modern RAG implementations have evolved beyond this baseline. Advanced RAG patterns introduce query rewriting, hypothetical document embeddings (HyDE), re-ranking layers, and recursive or agentic retrieval loops. Modular RAG frameworks further decompose the pipeline into interchangeable components, enabling fine-grained control over each retrieval and generation step.

At the infrastructure level, modern RAG pipelines rely on three core components working in concert:

• A vector database for storing and querying document embeddings at scale.

• An embedding model responsible for encoding both documents and queries into a shared latent semantic space.

• A generative model (the LLM) that synthesizes retrieved evidence into a coherent, task-specific response.

Below is an image that shows the high-level architecture:

Why LLMs Need RAG

During the training phase, an LLM compresses knowledge from its training corpus into billions of parameters. This process is inherently lossy because the model doesn’t store facts verbatim: it learns statistical associations between tokens. As a result of this process:

• Factual recall is probabilistic, not deterministic: The model may confidently produce a plausible-sounding but incorrect answer on a user query, because it’s sampling from a learned distribution rather than retrieving from a verifiable source. The process of producing a plausible-sounding but incorrect response is called hallucination.

• Knowledge is frozen at the training cutoff: Any event, document, regulation, product update, or research finding that postdates the model’s training window is entirely invisible to the model. For enterprise applications where data changes frequently, this is a critical failure mode. This means that if you query a model on something that occurred after its training ended, it will surely hallucinate the response.

• Specific, private, or proprietary knowledge is absent by design: Internal documentation, CRM records, legal contracts, and domain-specific corpora are never part of a pre-training dataset.

RAG was designed precisely to address these constraints that are inherent to LLMs. Below is how RAG maps to concrete business scenarios that enterprises face every day.

Documentation-Aware Support Bots

One of the most immediate and impactful enterprise applications of RAG is the construction of intelligent support bots grounded in a product’s technical documentation. Consider a SaaS company with several pages of technical docs, API references, changelogs, and troubleshooting guides. An LLM cannot reliably answer user questions about that product. What happens is that it either lacks the knowledge entirely or, worse, generates plausible-but-wrong instructions that erode user trust.

With RAG, every user query triggers a retrieval step against the indexed documentation corpus. The bot responds based on what the docs actually say, not on what the model statistically associates with similar-sounding topics.

The same pattern extends naturally to internal knowledge bases like HR policy portals, IT helpdesks, onboarding assistants, and legal FAQ systems, where the source of truth is a controlled, company-owned corpus rather than the public web.

Legal and Regulatory Intelligence on Recent Developments

The legal industry represents one of the clearest demonstrations of RAG’s necessity. Why? Because court rulings, regulatory guidance, legislative amendments, and compliance frameworks are published continuously. Legal research on an LLM is inherently dangerous because its training cutoff means the model is blind to any case law or regulation published after that date. And, in law, a single overlooked precedent can have material consequences.

RAG enables a fundamentally different architecture: legal documents, newly published court opinions, and regulatory updates are scraped, processed, and indexed into the retrieval layer as soon as they are available. When a lawyer or compliance officer queries the system, the retriever fetches the latest relevant documents, and the LLM synthesizes a grounded answer.

Competitive Intelligence and Market Monitoring

Enterprises routinely need to track competitor activity, industry trends, and market signals across news sources, press releases, job postings, and public filings. This information is inherently ephemeral because a competitor’s product launch announcement published yesterday is invisible to any LLM. By building a RAG pipeline that continuously ingests and indexes targeted public web sources, organizations can deploy analysts-in-the-loop systems that answer strategic queries based on fresh, real-world evidence rather than stale parametric knowledge.

Why Web Search Integration Is Not Enough

On the side of the architecture, the retrieval part of RAG systems can be built in different ways. The common ones are two:

• Creating a knowledge base of documents (like a plethora of PDFs and similar).

• Retrieving the needed information from the live web.

The knowledge base of documents is generally something that companies have already built over the years. So, let’s focus on the web retrieval part—which is also the most interesting one for us, scraping professionals.

Today, every commercial LLM has a web search tool integrated into it. So, at first glance, it seems to be the right tool to use for creating RAG systems. After all, if the model can query the live web, doesn’t that solve the knowledge cutoff problem entirely? The answer is: partially, and only for a narrow subset of enterprise needs. This is because the web search integration and RAG are fundamentally different mechanisms that solve different problems. Below are the main differences between the two:

• Web search operates on public web content: First of all, a web search tool cannot reach your internal knowledge base, your proprietary documentation, your CRM history, or your company’s legal contracts. Sure, some companies put their internal knowledge on the web with restricted access, but this encompasses only a few cases. Also, the truth is that no company actually puts all its documentation on a website, even if with restricted access (think of legal contracts, for example).

• Web search is non-deterministic and uncontrolled by design: When an LLM issues a search query, the results it retrieves depend on the search engine’s ranking algorithm, which is commercially influenced and optimized for general web traffic, not for your specific domain. In a RAG system, instead, you control the corpus. In other words, you decide what documents are indexed, how they are chunked, how they are weighted, and when they are refreshed. This level of governance is not optional in regulated industries where compliance is a requirement. Indeed, an enterprise cannot defend an AI-generated legal or financial output that was grounded in whatever Google happened to rank first. Also, no one guarantees that the web search is actually able to read the content on a website that is highly protected. In other words, very often happens that the web search tool can’t overcome the anti-bot systems on a target website, and the model simply hallucinates the response to your query (and you don’t even notice!).

• The retrieval mechanism itself is categorically different: Web search tools rely on keyword-based and PageRank-style relevance signals. RAG retrieval, by contrast, operates in a dense semantic vector space. This means that it finds documents that are conceptually relevant to a query even when they share no exact keywords with it. For domain-specific queries, this distinction in retrieval precision directly impacts the quality of the generated response.

• Web search tools introduce latency, rate limits, and external dependencies: A RAG system backed by an internal vector store is a self-contained, auditable, scalable infrastructure component. A system that depends on a web search tool at every call is coupled to external services, subject to availability failures, and difficult to reproduce or audit

• Web search tools don’t solve the hallucination problem: Web search tools increase the probability that relevant information surfaces in the context window, but if the search query is poorly formed, if the right page isn’t indexed, or if the model misreads the retrieved snippet, hallucination remains entirely possible. RAG, when implemented with rigorous chunking strategies, high-quality embeddings, and re-ranking layers, provides a far more controlled and reliable grounding mechanism.

In short, web search tools make general-purpose LLMs more useful for casual queries. This means that an LLM equipped with a web search tool is useful to respond to basic queries that rely on data recently published on the web. However, they can’t replace RAG systems for any organization that requires precision, privacy, control, and auditability over wide and specific knowledge.

Check the TWSC YouTube Channel

Web Scraping as the Data Acquisition Layer for RAG

The limitations of web search tools presented in the previous paragraph are structural. The solution to overcome them is to use web scraping as the retrieval layer for your RAG pipelines. This is because by using web scraping, you are not querying an external search engine and accepting whatever it returns. You get direct, programmatic access to the content of any publicly accessible web page, on your own schedule, under your own control.

So, let’s see how scraping the web changes the dynamic of retrieving content from the web for RAG purposes.

Full Control Over the Knowledge Corpus

When you build a RAG pipeline fed by a web scraper, you are the one defining the boundaries of your knowledge base. You decide which domains, subdomains, and URL patterns are in scope. You decide how frequently each source is re-crawled, which pages are included, which are filtered out, and which are prioritized during indexing. None of this is possible with web search integration, where the retrieval scope is, by definition, the entire public web as ranked by an opaque external algorithm.

This level of corpus governance is a prerequisite for any enterprise deployment where the knowledge base must be auditable, bounded, and defensible.

Also, using web scraping as the retrieval layer doesn’t mean that you have to code and manage your own scraping infrastructure—which could cause overheads as well as a huge amount of investments. The scraping industry today is mature enough to offer you:

• Pre-scraped datasets whose data can be refreshed based on your needs.

• Scraping infrastructure as-a-service. In other words, you tell the provider the content to scrape, and their professionals do the job for you, providing you with ready-for-analytics data.

Accessing Content That Search Engines Cannot Deliver

Public search engines index a fraction of the publicly accessible web, and they do so on their own timeline and according to their own crawl priorities. Pages that were published recently, that exist on domains with low authority, that are dynamically rendered via JavaScript, or that are structured in ways that resist standard crawling are routinely underrepresented or absent from search engine indexes entirely.

A purpose-built web scraper is not subject to any of these constraints. It can be engineered to handle JavaScript-rendered single-page applications using headless browser automation tools such as Playwright or Puppeteer, capturing the fully hydrated DOM rather than the bare HTML shell that a naive HTTP request would return. It can navigate paginated content, follow internal links recursively to achieve full site coverage, and process non-HTML formats.

Now, you can also think that a plausible solution is to put a target URL into the prompt and ask the LLM to retrieve the content via the web search tool. But due to the nature of the web search tool, this is not a reliable solution, because, as discussed previously:

• The web search tool may not be able to retrieve the content if the domain is highly protected by anti-bots.

• The use of a web search tool is non-deterministic due to how LLMs work. This means that the LLM can respond without actually retrieving the content. Or, in some cases, the LLM can respond, but its response can be a mix of content retrieved from the target URL and other content that the web search tool has retrieved from the web. In other words, the LLM can autonomously decide to retrieve the content, for a given query, from the target URL you passed into the prompt, and from other content that the web search tool retrieves from the web, based on the query.

But for RAG pipelines that need comprehensive, deep coverage of a specific web domain, the right degree of control over the crawling behavior is a fundamental feature, often a non-negotiable one.

Need help with your scraping project?

Deterministic, Reproducible Retrieval

One of the most underappreciated properties of a scraping-fed RAG system is reproducibility. When a user queries a RAG system backed by a web-scraped and version-controlled vector store, the retrieval behavior is deterministic: the same query against the same index returns the same results. This makes the system testable, debuggable, and auditable in ways that a live-search-grounded system categorically is not.

In a search-integrated LLM, the retrieved content is ephemeral. Specifically speaking, the content exists only in the context window of a single inference call. This means that it cannot be logged in a structured way, and may differ entirely if the same query is issued five minutes later because the search ranking shifted. In a scraping-fed RAG system, instead, every document in the index has a known origin, a known ingestion timestamp, and a known embedding. If an output is questioned, you can trace it back to the exact chunk, the exact source page, and the exact crawl that produced it. This traceability is the operational backbone of trustworthy enterprise AI.

Freshness on Your Terms

The knowledge cutoff problem that cripples base LLMs does not disappear with a static RAG corpus. A vector store that was indexed six months ago and never refreshed is, from a knowledge currency standpoint, equivalent to a model with a six-month-old training cutoff. Web scraping solves this by making freshness a configurable, automated pipeline.

A production RAG scraping pipeline typically implements a tiered crawl schedule calibrated to the update frequency of each source: high-volatility sources like breaking news feeds, regulatory update pages, earnings release portals, and product changelog endpoints are re-crawled on a schedule ranging from hourly to daily. Lower-volatility sources may be re-crawled weekly or triggered by change detection signals. When new content is detected, only the affected chunks are re-embedded and re-indexed, making incremental updates computationally efficient even at scale.

This architecture gives enterprises something that neither a base LLM nor a search-integrated model can offer: a knowledge base with a freshness SLA. This is a contractual guarantee that the information grounding the system’s outputs is no older than a defined threshold.

Clean, Structured Text as a First-Class Output

Web scraping, when implemented correctly, does not simply dump raw HTML into the processing pipeline. A production-grade scraping pipeline extracts clean, structured, semantically meaningful text from the web.

This matters enormously for chunking quality, and by extension, for retrieval quality. Recall that the integrity of every chunk is bounded by the integrity of the raw text it was derived from. A scraper that faithfully extracts clean, well-structured prose gives the downstream chunker the structural signals it needs to produce semantically coherent chunks. A scraper that emits raw HTML soup, or that fails to distinguish between primary content and navigational boilerplate, contaminates the entire corpus with noise that no chunking strategy can fully recover from.

Furthermore, this is another point of distinction with web search tools. In another article here at the web scraping club, we already covered the importance of feeding LLMs in markdown, rather than in HTML. Based on this, consider that:

• Every scraping pipeline can be managed to return data in any format you prefer. It can be a structured JSON, markdown, or anything else.

• Web search tools go directly to “read” the HTML of the target web pages. This means that the LLM responds based also on footers, banners, and anything in between. And this can lead the model to confusion as it responds to users’ queries.

Conclusion

RAG is an architectural commitment and, as such, it must be preceded by a deliberate choice to build a system where outputs are grounded in a controlled and continuously refreshed knowledge base. But that commitment is only as strong as the pipeline feeding it.

Web search integration tools are not that pipeline. This is because they are optimized for general-purpose queries, not for the precision, governance, and reproducibility that enterprises demand.

Web scraping, on the other hand, gives you full ownership over what enters the knowledge base, how fresh it is, how it is structured, and how it can be traced back when an output is questioned.

The next step is to start thinking of your scraping infrastructure not as a standalone data collection tool, but as the retrieval backbone of an intelligent system.

So, let us know in the comments: are you using web scraping for RAG purposes?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Instead, the site loads a basic shell and then fetches the live content from an internal API, usually as JSON, which the frontend then renders. E-commerce sites rely on this approach the most, since stock and prices change all the time and nobody wants to redeploy a page for every update. In fact, this is how modern frontends work.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

This actually works in our favor. Internal APIs are much more stable targets than the HTML around them. If the site gets a redesign, all your selectors might break, but the JSON endpoint usually stays put because the frontend still needs it. They’re also easy to spot, since they use regular HTTP in the browser, so you can watch them and see exactly what comes back. The real challenge is finding the right request and figuring out how to reach it past any anti-bot protection.

I always start by hand, in the browser, because that’s the best way to build a mental model of the target and keep your skills sharp. After that, I move to code, since clicking around DevTools for every new site gets tedious, and a small interception script can turn a twenty-minute manual search into a quick automated run. For this example, I’ll use Harrods (https://www.harrods.com/en-it/) as the target. By the end, we’ll be pulling the full product catalog straight from the API the frontend uses.

Doing it by hand first

Open DevTools, go to the Network tab, and filter to Fetch/XHR. That filter drops the images, fonts, scripts, and CSS, and leaves you with the requests the page makes to move data around. On a well behaved site you reload the page, watch a handful of JSON calls show up, and one of them is obviously the product list.

From identifying internal API calls to analyzing structured responses, LLMs are transforming web research. anyIP powers the access layer with residential and mobile IPs built for modern web-data workflows.

Explore anyIP Proxies →

Harrods doesn’t make it that easy, and the reason is worth understanding because it’s common. When you load a category page like women-new-in, the products are already in the HTML. The page is server side rendered, so the first response comes back with the grid baked in and no separate product API call fires. If you only look at the initial load, you’ll conclude there’s no API and reach for HTML parsing.

The API is there. It just doesn’t fire until the page needs new products it doesn’t already have. So you paginate. Click to page two, and now the frontend has to go get sixty products it doesn’t have in memory, so it makes the call. This is the habit that pays off most when you hunt internal APIs. You interact with the page and watch what fires. Paginate, filter, sort, scroll. The data call shows up when the page needs data.

Do that on Harrods with the Network tab open and one request stands out from the analytics noise, a POST to /api/search/1/indexes/*/queries. That path is a giveaway. It’s the Algolia query API, and we’ve pulled product data out of Algolia before (https://substack.thewebscraping.club/p/scraping-algolia-endpoints). Harrods proxies it through its own domain instead of calling Algolia directly, which turns out to matter, but the shape is familiar.

That’s how I do it manually. It works, and honestly, it teaches you more about the target than any tool. But it’s slow, and if you need to check ten sites in an afternoon, it just doesn’t scale. So I automate the search instead.

What Harrods actually is

Before I run any tool, I like to map out the layers, since each one is a possible failure point.

The storefront is a Nuxt application, which we can see from the /_nuxt/builds/meta/*.json calls in the traffic. Nuxt server renders the first page and then hydrates into a single page app, which is exactly why the product grid is in the initial HTML but pagination is a client side fetch. Commerce runs on SCAYLE, visible from the hrd-prd.middleware.api.scayle.cloud and hrd-live.cdn.scayle.cloud hosts. Product search is Algolia, proxied through harrods.com/api/search. Currency and pricing for the Italian storefront are handled by Global-e, which shows up as the gepi.global-e.com calls.

On top of all that, there’s Akamai Bot Manager. This is the layer you have to get past just to load the page, and it decides if you can even use an automated browser. I’m not going to reverse engineer Akamai’s sensor here. That’s another story. Instead, I’ll sidestep it using a stealth browser and a warm-up navigation, then focus on the data. For this walkthrough, the real target is the JSON API layer. Akamai is just the wall I need to climb to get there.

Building the interceptor

The core of the tool is a handler attached to the browser’s response event. For every response, I check the content type, keep the JSON ones, drop the analytics noise, and save the body. Here’s the handler, from intercept.py:

def make_handler(captured, bodies_dir):
    seen = set()

    async def on_response(response):
        try:
            headers = await response.all_headers()
        except Exception:
            return
        ctype = headers.get("content-type", "").lower()
        if not any(m in ctype for m in JSON_MARKERS):
            return
        if is_noise(response.url):
            return

        request = response.request
        base = response.url.split("?")[0]
        key = (request.method, base)
        if key in seen:
            return
        seen.add(key)

        try:
            raw = await response.body()
        except Exception:
            raw = b""
        ...
        print(f"  [json] {request.method} {response.status} {base}  ({len(raw)} B)")

    return on_response

Check the TWSC YouTube Channel

For the content type filter, I use application/json, application/ld+json, text/json, and anything ending with +json. To cut out the noise, I block hosts that are almost always analytics or consent, Google, Facebook, OneTrust, Segment, and similar, so they don’t drown out the real endpoints. I deduplicate by method and path, since hitting a paginated endpoint five times is still just one endpoint. Every response I keep gets its body saved to disk, so I can check the payloads later.

Attaching it is one line, right after I open the page:

page.on("response", make_handler(captured, bodies_dir))

There are two details that make this work on a real site.

First, you need a warm up. If you go straight to a deep URL like women-new-in, Akamai will block you, since you’re asking for an inner page without the cookies a real browser would have collected. So I always load the homepage first, let Akamai set its _abck and bm_sz cookies, wait a bit, and only then go to the category page in the same session.

if warmup:
    print(f"  warmup -> {warmup}")
    await safe_goto(page, warmup, settle_ms * 2)
if await safe_goto(page, url):
    await drive(page, scrolls, settle_ms)

Second, there’s pagination. From the manual check, I know the product API call only happens when the page needs products it doesn’t already have. Just navigating to ?page=2 isn’t enough, since that triggers another server-rendered load. What you need is a client-side click, just like a real user would do. The most reliable way is to trigger the click through the DOM. Playwright’s normal click kept timing out on the pagination control.

async def click_through(page, selectors, clicks, settle_ms):
    for selector in selectors:
        for n in range(clicks):
            count = await page.locator(selector).count()
            if not count:
                print(f"  click skipped: {selector} not in DOM")
                break
            await page.evaluate("() => window.scrollTo(0, document.body.scrollHeight)")
            await page.wait_for_timeout(1000)
            try:
                await page.eval_on_selector(selector, "el => el.click()")
                print(f"  js-click [{n + 1}] {selector}  (matches={count})")
                await page.wait_for_timeout(settle_ms)
            except Exception as exc:
                print(f"  click failed: {selector} -> {type(exc).__name__}")
                break

Running it

Everything here runs from the shared venv, where Playwright (1.55.0), Camoufox (0.4.11), and curl_cffi (0.14.0) were already installed.

First the control. Plain Playwright, headless, straight at the category page:

python intercept.py \
  --url "https://www.harrods.com/en-it/women-new-in" \
  --engine playwright --headless --scrolls 3 --out runs/harrods_vanilla

It doesn’t even get a page back:

playwright._impl._errors.Error: Page.goto: net::ERR_HTTP2_PROTOCOL_ERROR
  at https://www.harrods.com/en-it/women-new-in

That’s Akamai dropping the connection at the HTTP/2 layer, before any HTML is served. It didn’t like the TLS and connection fingerprint of an unpatched automated browser. Worth noticing that this is a different failure than a block page. There’s no “Access Denied” body to read, the socket just closes. The control confirms plain Playwright is a non starter on this target, which is the whole reason the control exists.

Now Camoufox, with the homepage warm up and a click through to page two:

python intercept.py \
  --url "https://www.harrods.com/en-it/women-new-in" \
  --warmup "https://www.harrods.com/en-it/" \
  --click 'a[href="/en-it/women-new-in?page=2"]' --click-times 1 \
  --engine camoufox --scrolls 6 --settle-ms 3000 \
  --out runs/harrods_jsclick

This time the page loads, the title comes back as Designer Women New In | Harrods IT, and the harness prints the JSON calls as they land. Stripping the Akamai sensor beacon and the third party hosts, here’s what Harrods itself serves as JSON:

GET  200 /_i18n/10acb0d1/en-gb/messages.json            (54186 B)
POST 200 /api/rpc/getUser
POST 200 /api/rpc/getBasket                             -> basket
POST 200 /api/rpc/getWishlist                           -> key,items
POST 200 /api/rpc/getHashedString                       -> hash
POST 200 /api/rpc/getExternalIdpRedirect                -> harrods
GET  200 /_nuxt/builds/meta/d3043d4e-...-631f954ba023.json
POST 200 /api/search/1/indexes/*/queries               -> results  (488909 B)

The /api/rpc/* calls are the storefront’s own internal RPC endpoints for things like the basket, wishlist, and user session. These are interesting and worth mapping if you’re modeling the site, but they’re not what I was after. The last line is the one I wanted: a 488 KB POST to the Algolia proxy, keyed on results, and it only showed up after clicking to page two. On the first load, it wasn’t there, just like I saw in the manual check.

Reading the payoff

Here’s what came back in that response:

top keys: ['results']
result[0] keys: ['hits', 'nbHits', 'offset', 'length', 'facets', 'query', 'index', 'queryID', ...]
index: prod_harrods.com | nbHits: 1199
hits on this page: 60

Sixty products in one call, out of 1199 in the category, from the index prod_harrods.com. Each hit carries what you’d actually want:

- brand='Rodo'          name='Satin Dafne Slingback Heels 75'   price={'GBP': 79800, 'EUR': 100000, 'USD': 110000, ...}
- brand='Roksanda'      name='Wool Scarf-Tie Isabel Blazer'     price={'GBP': 139500, ...}
- brand='Alessandra Rich' name='Silk Polka-Dot Bow Dress'       price={'GBP': 160000, ...}

You get product id, reference key, name, brand, gender, URL, images, category hierarchy, and price as a map of currencies in minor units. For example, GBP: 79800 means £798.00. This is exactly the same data you see in the product grid, but now you can store it directly without having to parse the HTML.

The reusable part is the request itself. Here’s the POST body the frontend sent:

{"requests":[{"indexName":"prod_harrods.com","query":"",
  "params":"facets=%5B%22*%22%5D&clickAnalytics=true&ruleContexts=%5B%22EUR%22%5D&maxValuesPerFacet=1000&facetFilters=%5B%5B%22categoryPageIdentifier%3AWomen%20New%20In%22%5D%5D&offset=60&length=60"}]}

If you decode the params, it’s all clear. The facetFilters with categoryPageIdentifier:Women New In limits the query to this category. offset=60&length=60 handles pagination, so page two starts at offset sixty. Change the offset and you can walk through the whole category, sixty products at a time.

One thing is clearly missing here. There’s no x-algolia-application-id or x-algolia-api-key, which a direct Algolia call always carries. That’s because we’re not calling Algolia, we’re calling harrods.com/api/search, and the proxy injects the credentials server side. We don’t need to find or rotate an Algolia key, we just POST to the Harrods path with the cookies our browser already earned.

Replaying it to page the whole catalog

Read more

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

The rollercoaster inside the proxy industry

Residential proxy networks are facing a hard time. After the IPIDEA case, the FBI targeted NetNut, one of the biggest names in the industry, and seized their domain netnut.io, proxyjet.io (subsidiary company), and divinetworks.com (partner network for IP sourcing). The suspicion is that their residential proxy network was used for fraudulent operations.

I’ve worked closely with NetNut, and I don’t think they’re directly involved in anything: it’s a public company, there’s no need to put their network at a fraudster’s disposal. But some pieces of the KYC procedures and auditing were probably missing, given that they also had a network of resellers which are harder to control.

But the hard news is not ending here: following the Spur research on Smart TVs we shared last month, LG decided to ban the residential proxy option from their apps on the webOS platform.

This crusade against residential proxy networks probably will lead to major shifts in the next months (ready to be contradicted).

• Less supply, higher demand: higher prices and/or worse quality. At least for residential proxies, the price run to the bottom will stop. With demand growing and fueled by AI needs and less supply of IPs, more requests will transit through the same addresses, degrading quality. More providers will try to avoid this by charging more for premium-quality addresses (read: addresses with fewer requests on them).

• No more easy access to proxies: some companies have already restricted the usage of residential networks to customers who have already completed the KYC process.

  

  In this example, Bright Data is requiring KYC for using residential proxies. Until a few months ago, it was required only when targeting specific websites.

An interesting point of view about this situation can be read in an article from (Massive founder) on this Substack.

Jason's Substack

The Web Needs a Front Door

The web’s original breakthrough was radically low-friction access. CERN released its core technology royalty-free, allowing anyone with an internet connection to publish information, link to others, and build without asking a central platform for permission…

Read more

a month ago · 5 likes · Jason

A rising tide lifts all boats

Luckily, we don’t have just bad news from the scraping industry.

After the Nimble February announcement of a Series B funding round of 47M USD, Oxylabs received a 130M USD investment from private equity firm Warburg Pincus.

The need for data by AI, both for training and in real time, is fueling the industry revenues.
But it’s not just a matter of money; it’s about being recognized as a crucial technology layer for every business that needs web data.

Another piece of good news comes from the legal side: Federal Court Grants SerpApi’s Motion to Dismiss Google’s DMCA Lawsuit. This is a major win for the web scraping industry, since “the court rejected Google’s attempts to expand the DMCA to assert control over access to public pages”.

The Napster era for content

I’m not saying anything new when I state that AI overviews and LLMs’ answers are crunching visits to websites (and, in some cases, hammering them with millions of requests for scraping them).
Cloudflare recently announced that, by default, “for new customers and for new sites for existing customers, on September 15, 2026, the defaults will be set to allow for search but block training and agent use for pages with ads”.

The idea, which seems good on paper, is to make AI crawlers pay for data to compensate the publisher instead of scraping it. They already announced the Pay-per-crawl protocol some time ago, and this choice goes in the same direction.

We did the math some time ago, and it doesn’t make any sense. Those fees should be low enough to make it more convenient to pay instead of bypassing Cloudflare. This means that 99% of website owners won’t even notice the payment on their bank account.

I believe we’re entering the Napster era for online content. Premium content that we don’t want to be fully cannibalized by AI will be under a paywall. There will always be someone who pays for it and then shares it for free with others; it’s unavoidable, just like on Napster in the 2000s. Let’s call this unwanted marketing expenses.

Read and share more of this

Want to flag and discover more articles like the ones above? Scraping News (still in beta) is where the community surfaces them. Sign up and submit what you find worth reading.

Want a concept map of everything covered here, cross-referenced with our own work and other sources worth your time?

The Web Scraping Club Wiki lives on GitHub, and as an interactive site. It is plain Markdown, so you can also clone it into your own Obsidian vault and read it locally.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

An Introduction to SearchApi

Before exploring the new MCP offering, let me first introduce you to the world of SearchApi!

What Is SearchApi?

SearchApi is an online service that provides multiple scraping APIs. It allows you to get data from a wide range of websites, with a strong focus on major search engines.

SearchAPI endpoints provide programmatically access to real-time data from Google, Bing, and YouTube, as well as platforms like Amazon, Airbnb, Zillow, TikTok, and many others. It handles the underlying infrastructure for you, while also taking care of proxy rotation and bot protections.

Main APIs

Below is a table of some of the most relevant SearchApi endpoints:

To discover all endpoints, visit the official documentation. On each API page, you can also access the OpenAPI specification, as well as download the Postman collection.

Pricing Plans

SearchApi follows a pay-per-success pricing model built around monthly subscription plans. These range from developer-level access ($40/mo) to high-throughput enterprise tiers ($5,000/mo), each including monthly search quotas and 99.9% SLA guarantees.

Pricing decreases with scale, starting at around $4 per 1,000 searches and going down to about $1 per 1,000 at higher volumes. A free trial is also available, offering 100 free requests before committing to a paid plan.

Introduction to the SearchApi Web Search MCP Server

As a web scraping enthusiast, chances are you’re already familiar with scraping APIs or SearchApi itself. So, since the company has recently launched a new MCP server product (called Web Search MCP), let me focus on that!

How It Works

Web Search MCP enables AI models to connect to SearchApi’s endpoints via a unified interface.

It’s a remote-only MCP server that operates over Streamable HTTP. That means you can’t install it locally. Instead, it runs as a hosted service that your AI client connects via an MCP URL. The server also supports OAuth 2.0, enabling secure authentication.

Thanks to the OAuth support, you can connect SearchApi to your AI tools through a simple “Connect” flow. This way, the AI client can act on the user’s behalf without ever handling or storing the user’s API key or password. Access can be revoked at any time from either the SearchApi dashboard or the connected AI platform, giving you full control over permissions.

Your AI assistant sends requests to the MCP server, which processes them, executes the required actions, and returns structured results in real time. This setup removes the need to manage local infrastructure or dependencies while keeping integration straightforward.

SearchApi’s Web Search MCP server can be connected to any tool that supports remote MCP servers via Streamable HTTP, including Claude Desktop, Claude Code, Cursor, and many others. It’s also available as an official Claude connector.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Tools

The Web Search MCP server exposes over 80 tools. Some of the most useful ones include:

Discover all other tools in the official docs.

Pricing and Limitations

Pricing follows the same pay-per-success model as the regular SearchApi APIs, with each MCP tool invocation counting as a single API request. Web Search MCP supports up to 10 MCP integrations per account. Also, standard SearchApi rate limits apply.

How to Use the SearchApi Web Search MCP Server in a Real-World Scenario

To showcase the SearchApi Web Search MCP Server, I’ll configure it in Claude Code (though any other compatible MCP tool will work as well) and use it in a sentiment analysis workflow. The goal is to retrieve the latest reviews for a restaurant and let the AI analyze them.

Follow the instructions below!

Prerequisites

To follow along with this section, make sure you have:

• A SearchApi account (a free trial account is enough).

• Claude Code installed and configured locally.

• A basic understanding of how MCP works.

Step #1: Create Your MCP Integration

Start by logging in to your SearchApi account. From the dashboard, click “MCP” under the “INTEGRATIONS” section:

Next, select “Your MCPs” and click “New Integration”:

You’ll be taken to the page for creating a new SearchApi Web Search MCP integration. Start by giving your integration a name (e.g., “Review Bundle”).

Next, either select one of the recommended tool bundles or customize your own. To do so, scroll down, search for “review,” and add all review-related tools:

• Airbnb Property Reviews

• Facebook Business Page Reviews

• Google Maps Reviews

• Tripadvisor Reviews

• Walmart Reviews

Now, most review tools require a page URL, place ID, or another entity identifier. To retrieve those automatically, you’ll also need some search tools.

For that reason, search for “Google” and add:

• Google Maps Search

• Google Maps Reviews

The “Google Search Light” tool is fundamental. From the returned search results, AI finds the relevant page URL (e.g., the Tripadvisor page for a specific restaurant), which typically already contains the required identifier or can be used to derive it. This ID or URL can then be passed directly to the corresponding review tools to fetch detailed reviews and structured data.

Once you’ve selected all the required tools, click “Create Integration” to generate your SearchApi MCP integration. You’re now ready to connect it to your AI client. Cool!

Check the TWSC YouTube Channel

Step #2: Configure Your SearchApi MCP URL in Claude Code

After creating the MCP integration, you’ll be redirected to your SearchApi MCP page. Here, you can find your MCP URL along with setup instructions for Claude Code, Claude Desktop, and Cursor.

Copy the MCP URL, which follows this format:

https://www.searchapi.io/mcp?token=<YOUR_SEARCHAPI_MCP_TOKEN>

<YOUR_SEARCHAPI_MCP_TOKEN> is a unique token generated by SearchApi. It identifies your specific MCP integration and is used for authentication.

Note: For enterprise integrations and agent platforms that require delegated access, OAuth authentication is the recommended approach.

Next, add the MCP server called review_bundle to Claude Code using this command:

claude mcp add review_bundle "<YOUR_SEARCH_API_URL>" --transport http

Make sure to replace the <YOUR_SEARCH_API_URL> placeholder with the URL you copied from the dashboard. Amazing!

Step #3: Verify That the MCP Integration Works

Begin by launching Claude Code:

claude

Then run the command below:

/mcp

This will list all available MCP servers, including review_bundle:

Select the review_bundle server, then choose “View tools”. You’ll see all 7 configured tools:

This confirms that your Claude Code environment has successfully connected to the remote SearchApi Web Search MCP tools. Perfect!

Step #4: Test the Workflow

To verify that the MCP integration works for review scraping, try a sentiment analysis prompt like this:

Get the latest reviews for Sottocasa Pizzeria Harlem in New York from Google Maps and Tripadvisor. Then, generate a sentiment analysis report in Markdown that includes quotes from the retrieved reviews, and save it as reviews.md.

Execute the prompt, and you should see something like this:

This is what Claude Code did:

1. Understood that it first needed to retrieve place IDs.

2. Used google_maps_search and google_light_search to find the Google Maps ID and Tripadvisor ID for the restaurant, respectively.

3. Called tripadvisor_reviews to fetch the latest 15 reviews using the Tripadvisor ID, and google_maps_reviews to fetch 15 recent reviews using the Google Maps ID.

4. Aggregated the retrieved reviews in JSON format and generated a structured Markdown sentiment analysis report with supporting quotes.

5. Exported the final output to reviews.md.

Important: Claude’s built-in Fetch and Web Search tools may not reliably achieve this goal due to anti-bot protections on platforms like Google and Tripadvisor.

The final result is a reviews.md file containing a sentiment analysis based on the latest 15 Google Maps reviews and 15 Tripadvisor reviews for the selected restaurant:

Notice how the report includes direct quotes from relevant reviews and real-world insights extracted from them. Mission complete!

[Extra] Analyze the Output of the Web Search MCP Tools

Expand the MCP tool calls made by Claude Code, and you’ll see:

Notice the use of Google Maps Search to retrieve the Google Maps ID of the target restaurant.

Similarly, expand the Google Search Light tool call to inspect the JSON SERP returned by SearchApi for the “Sottocasa Pizzeria Harlem Tripadvisor” query in the US:

Next, focus on the Google Maps review scraping step:

Note how this returns results in a structured array field (reviews) containing all relevant review data. The same pattern applies to the Tripadvisor review tools:

This demonstrates how the SearchApi Web Search MCP enables Claude Code to complete sentiment analysis tasks by providing real-time scraped review data from multiple platforms, returned in a structured format that is easy to process and analyze.

Need help with your scraping project?

Other Supported Use Cases

Beyond review scraping and sentiment analysis, SearchApi’s Web Search MCP unlocks a wide range of use cases. These include:

• Travel planning and price optimization: Search flights, hotels, and vacation rentals while comparing prices and availability across dates. AI agents can recommend optimal travel windows and cost-efficient itineraries.

• Market and ad intelligence monitoring: Analyze ads across Meta, Google, TikTok, and LinkedIn to track competitor messaging, creative strategies, and campaign activity in real time.

• E-commerce product research and comparison: Retrieve product listings, pricing, and reviews across Amazon, Walmart, eBay, Best Buy, and Google Shopping to build product comparison or recommendation systems.

• Local business discovery and mapping agents: Use Google Maps and Apple Maps to find places, extract reviews, generate directions, and build location-aware assistants for restaurants, services, or POIs.

• Social media and brand intelligence tracking: Fetch public profiles from Instagram, TikTok, and Facebook to monitor influencers, competitors, or brand presence and engagement metrics.

• News and trend monitoring agents: Search Google News to track breaking news, industry updates, and topic trends for real-time intelligence dashboards or alerting systems.

• Educational and research assistants: Query Google Scholar to retrieve academic papers, legal opinions, and citations for research summarization and knowledge extraction.

Conclusion

I’ve personally tested MCP servers from various providers in the scraping industry, and the experience with SearchApi’s Web Search MCP has definitely been smooth.

I didn’t encounter any significant issues during setup, and the integration worked immediately. Across the MCP tools I tested, none of the requests failed, which is a strong indicator of reliability and overall quality.

I’d therefore recommend it as a solid solution for giving AI agents rich search capabilities across multiple platforms, especially for companies and developers with some budget (given the lack of a free plan).

I hope you find this post useful. If you have any questions, feel free to reach out in the comments below. Until next time!

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

I wanted a number I could actually trust, not just another marketing claim, and started to write this piece. Then I saw the great work on the Tyler Richards website (richards.foo/tools/stealth-benchmark) and took some inspiration from it. My idea was to take the open source Python automation stack, point each tool at the same signals a bot detector checks in JavaScript, and score them all the same way. In the past, I have written about anti-detect browsers and fingerprint test pages, but those were about commercial GUI browsers. This time, I focused on the libraries you actually pip install and use in code.

The whole test harness is in our GitHub repository reserved for paying users, inside the folder 111.STEALTH_BENCHMARK. You can run it yourself and get your own results.

The tools I put on the bench

I grouped the fifteen tools into families, so the scores are easier to compare.

My two baselines are plain Playwright and plain Selenium, with no stealth and no patches. These are the control group, and they are supposed to get caught. If a “stealth” tool scores the same as vanilla Playwright, that tells you a lot.

The Playwright stealth patches are patchright, rebrowser-playwright, and undetected-playwright. These keep the Playwright API and try to remove specific automation leaks underneath it.

The CDP-native group is pydoll, nodriver, and zendriver. They drop WebDriver entirely and talk to Chrome over the DevTools Protocol. nodriver is the official successor to undetected-chromedriver, written by the same author, and zendriver is a maintained fork of nodriver.

Undetectable browsers are getting impressively capable, but even the stealthiest one stalls when its connection lets it down. anyIP’s residential and mobile proxies pair real-user IPs with 99.9% uptime - so the network is never the weak link

Build with anyIP →

Then the browsers that rebuild the fingerprint instead of patching flags on top of a normal Chrome. camoufox and invisible_playwright are patched Firefox builds that rewrite the fingerprint at the C++ level. CloakBrowser does the same to Chromium, shipping its own patched binary with a drop-in Playwright API. RayoBrowse is a self-hosted stealth Chromium from Rayobyte that runs in a Docker container and serves a fingerprint picked from a database of real devices. Obscura is the odd one out, a headless browser engine written in Rust that embeds V8 and ships its own browser rather than driving Chrome at all. And scrapling through its StealthyFetcher wraps a stealth Chromium, while undetected-chromedriver is patched Selenium.

What the detector actually sees

A bot detector is not just one thing, so before testing, you have to decide which layer you are actually targeting. In practice, a real detector looks at four main areas: the JavaScript fingerprint surface, the transport (TLS and HTTP/2), request timing, and the proxy exit. For this benchmark, I focused only on the JavaScript fingerprint surface and ignored the rest on purpose.

This is not about being lazy. Scoping it this way is what keeps the results honest. Transport fingerprinting mostly depends on the browser’s own network stack, which is the same whether you use Chromium or Firefox, no matter which Python library you use. Timing and behavior are a different topic, more about mouse movement and dwell time. Proxy reputation is about your IP, not your tool. So the real question I can answer by swapping libraries is simple: when a detector’s JavaScript runs inside the page, what does it actually find?

The detector looks for a fixed list of tells. Is navigator.webdriver true. Does the user agent say HeadlessChrome. Is the WebGL renderer a software rasterizer like SwiftShader instead of a real GPU. Are there leftover automation globals on window, like the old chromedriver cdc_ variables or a framework’s init scripts. Does the chrome object have the right properties. Do the user agent, navigator.platform, and the Client Hints all agree on the OS. The knowledge base on richards.foo lists hundreds of these, but I pulled the highest severity ones into my own probe.

How I set up the test

At the core, I use a single HTML page called probe.html, served from a small local server. Each tool loads this page, the page reads its own fingerprint in JavaScript, and writes the result back into the DOM as base64 JSON. I use base64 so I do not run into any HTML-escaping issues when I pull the rendered page from each driver.

To keep the harness manageable, I wrap every tool, no matter how different its API, in a single method:

def render(url, wait_selector=None, timeout=30000, settle_ms=6000, screenshot_path=None) -> str:
    # launch in the tool's stealth config, goto(url), wait, return final HTML, close

The new anti-bot solution - Toughest walls, lowest price. Claim your free 10 000 requests with coupon code WSCLUB

Claim your credits

Whether it is Playwright sync, Selenium’s execute_script, pydoll and nodriver’s async CDP calls, scrapling’s StealthyFetcher.fetch, CloakBrowser’s launch(), RayoBrowse over a Docker CDP socket, or Obscura over its Rust CDP server, they all get wrapped by render(url) -> html. I use the same method for the probe and the two oracle pages, so I never have to special-case a driver. Here is the full vanilla Playwright adapter, and it does not get simpler than this:

 from playwright.sync_api import sync_playwright

def render(url, wait_selector=None, timeout=30000, settle_ms=6000, screenshot_path=None):
    with sync_playwright() as p:
        browser = p.chromium.launch(channel="chrome", headless=True)
        page = browser.new_page()
        try:
            page.goto(url, wait_until="load", timeout=timeout)
            if wait_selector:
                page.wait_for_selector(wait_selector, timeout=timeout)
            else:
                page.wait_for_timeout(settle_ms)
            if screenshot_path:
                page.screenshot(path=screenshot_path, full_page=True)
            return page.content()
        finally:
            browser.close()

Check the TWSC YouTube Channel

For every Chrome-based tool, I used channel="chrome" to drive the real installed Chrome, not Playwright’s bundled Chromium. This choice is important. Playwright’s bundled Chromium in headless mode reports a software WebGL renderer (SwiftShader), which is just a headless artifact, not something the tool did. When I point the same tools at real Chrome, the WebGL renderer shows the actual Apple M2 GPU. By using real Chrome everywhere, the tests I measure reflect the tool’s stealth, not which Chromium binary happened to ship. The engine-level browsers like Camoufox, CloakBrowser, and Obscura ship their own binary, so they use that instead.

I ran everything in headless mode on purpose, since that is both harder and more common for scraping. Headed mode hides some of these tells automatically, and I mention that where it is relevant.

I do not just trust my own score, so I use two outside referees. My probe gives a 0 to 100 score by subtracting a weight for each tell the tool leaks. Then I check two open bot-detection pages for an independent verdict: BrowserScan and deviceandbrowserinfo. deviceandbrowserinfo is more useful because it returns a JSON breakdown, an isBot flag, and about twenty individual detections like hasWebdriverTrue, isHeadlessChrome, and isAutomatedWithCDP. I tuned the weights in my probe so the ordering matches those verdicts, and that is the calibration step. The runner also saves a screenshot of the deviceandbrowserinfo verdict for every tool, so the human or bot call is on record, not just my parser’s output.

Running it, and what came back

Fifteen tools, three targets each, and here is the table the reporter produced.

My probe score and the deviceandbrowserinfo verdict matched on 12 out of 13 tools I could get a verdict for, so that is a 92 percent match. I saved the heatmap of every tell by every tool at 111.STEALTH_BENCHMARK/results/heatmap.png, and you can see the tiers at a glance.

Four tools passed the strict referee: camoufox, CloakBrowser, RayoBrowse, and scrapling. Three of them get there the same way, by rebuilding the browser itself instead of patching automation flags onto a normal Chrome, so there is no HeadlessChrome in the user agent and nothing that reads as CDP automation. scrapling is the fourth, and it passes for a reason worth its own section below, because it took a version bump to get there. Everything that still leaves navigator.webdriver true sits at the bottom of the table, and everything in the middle is one user agent string away from a clean pass.

Read more

Then you get access to a residential proxy pool, and the scraper works.

If you’ve been there, you probably moved on without asking why. The adage is simple: if it works, don’t touch it!

But that “why” matters more than you think because residential proxies aren’t a kind of black magic. They’re a specific technical architecture on top of which your scrapers work. And as with any architectural choice, you have to make decisions on purpose and based on specific parameters.

In this article, you’ll get a complete technical picture of residential proxies, starting from how IP addresses are classified, how the proxy infrastructure routes your traffic, what residential proxies are, and their common types. In the end, you’ll also discover the ethics behind residential IP sourcing and how to choose ethical providers with a practical decision framework.

Let’s get into it!

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

What Is an IP Address?

An IP address (Internet Protocol address) is a numerical label assigned to every device connected to a network that uses the Internet Protocol for communication. On the practical side, the IP protocol serves two core functions:

• Host or network interface identification.

• Location addressing.

The current dominant versions are IPv4 and IPv6:

• IPv4 addresses are 32-bit integers, expressed in dotted-decimal notation as four octets ranging from 0 to 255 (for example, 203.0.113.47), yielding a theoretical address space of ~4.3 billion unique addresses.

• Due to exhaustion of this space, IPv6 was introduced, using 128-bit addresses expressed in colon-separated hexadecimal groups (for instance, 2001:0db8:85a3:0000:0000:8a2e:0370:7334), offering a space of ~3.4 × 10³⁸ addresses.

IP addresses aren’t handed out randomly by providers: there’s a strict hierarchy that they must respect. Specifically, a global authority called IANA divides the total pool of IP addresses into large blocks and distributes them to regional bodies known as RIRs (Regional Internet Registries). These RIRs then assign smaller blocks to ISPs (your home internet provider) and cloud providers (like AWS or Google Cloud), who finally assign individual IPs to end users and servers.

Why does this matter for scraping? Because this entire chain is publicly documented, and anyone can look up who owns a given IP address using tools like WHOIS (a directory of IP ownership records). This means a target website can instantly tell whether your request is coming from a residential IP, a datacenter IP, or a known proxy or VPN provider.

This is precisely why residential proxies are so valuable in web scraping: their IPs are indistinguishable from those of regular home users, making them much harder for anti-bot systems to flag.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

What Is a Proxy Server?

A proxy server is an intermediary host that sits between your client and the target web server, forwarding requests and responses on your behalf:

Without a proxy, the TCP connection to the target server is established directly from your machine. With a proxy, your client opens a connection to the proxy, which opens a separate connection to the target, and the target sees the proxy’s IP address as the request origin. This way, your real IP never appears in the target’s access logs or network layer.

This IP substitution is the fundamental value proposition of a proxy in a scraping context. Every request your scraper sends carries a source IP and, as covered above, target servers actively use that IP to make trust decisions. By routing through a proxy, you externalize your network identity. This way, the anti-bots on target servers evaluate the reputation, geolocation, and classification of the proxy’s IP, not yours.

At the network level, a proxy is simply a process listening on a TCP port, capable of accepting inbound connections and initiating outbound ones. It maintains two separate TCP connections for each proxied request: one with your client and one with the target. These connections are independent: they can use different protocols, different keep-alive settings, and the proxy can pool or reuse the outbound connection across multiple client requests.

The proxy may also perform additional functions depending on its implementation. For example, it can manage authentication, logging, IP rotation (cycling through a pool of IPs across requests), and header manipulation.

Proxy Protocols: HTTP/HTTPS vs. SOCKS5

When routing traffic through a proxy, the protocol determines how the proxy handles your traffic at the application and transport layers. This has real implications for scraping purposes. The main protocols are the following:

• HTTP/HTTPS: Proxies that use these protocols operate at the application layer (OSI Layer 7). When your scraping client sends a request through an HTTP proxy, it constructs a full HTTP request and sends it to the proxy server, which forwards it to the target on your behalf. For plain HTTP traffic, the proxy reads the request in full. For HTTPS traffic over an HTTP proxy, the client issues a CONNECT method request, instructing the proxy to establish a raw TCP tunnel to the destination. Once the tunnel is open, the TLS handshake occurs directly between your client and the target server. This means that the proxy becomes a blind pipe and cannot decrypt the payload. This is critical to understand: an HTTP proxy handling HTTPS via CONNECT does not perform SSL termination (unless it’s explicitly a man-in-the-middle proxy with a custom CA cert injected).

• SOCKS5: Is a lower-level protocol operating at the transport layer (OSI Layer 5). Unlike HTTP proxies, SOCKS5 is protocol-agnostic. This means that it simply relays raw TCP (and optionally UDP) streams without any awareness of the application-layer protocol being tunneled. After a brief handshake, the proxy establishes a TCP connection to the destination and shuttles bytes in both directions without inspection. This makes SOCKS5 suitable for any TCP-based protocol, not just web traffic.

One operationally significant detail to consider is DNS resolution. With SOCKS5, the domain name is forwarded to the proxy for resolution, meaning DNS queries originate from the proxy’s network, not yours. This prevents DNS leaks, which can expose your real network location even when your traffic routes through a proxy. HTTP proxies also resolve DNS on the proxy side for CONNECT-tunneled requests, but behavior can vary by implementation.

Now, here’s a question for you: for scraping purposes, which protocol is the best? Well, for most scraping use cases, either protocol is functionally equivalent from the target server’s perspective, since both can tunnel TLS. SOCKS5 is generally preferred for its flexibility and slightly lower overhead, but HTTP proxies are often easier to drop into existing HTTP client configurations without additional dependencies.

Tired of stitching together proxies, browsers, and parsers just to scrape one page? Gaffa handles the whole stack for you, with the control to fine-tune every step when you need it.

Start With Gaffa

What Are Residential Proxies?

A residential proxy is a proxy server whose outbound IP address is assigned by a consumer Internet Service Provider (ISP) to a real, physical household device like a home router, laptop, smartphone, smart TV, or similar. The word “residential” refers directly to the ISP classification of the IP: these are addresses allocated to residential customers, and they appear as such in public IP intelligence databases.

How Residential Proxies Work

The operational model of a residential proxy network involves three distinct parties: the proxy provider (the company offering proxy servers on the market), the exit node (the residential device), and the scraping client (you).

The exit node is a real consumer device whose owner has installed software that, in exchange for some compensation, allows the provider to route third-party traffic through their internet connection. This software runs as a background process and registers the device with the provider’s orchestration infrastructure. The device’s residential IP is thus made available in the provider’s pool.

The provider’s infrastructure is the backbone that ties everything together. It consists of:

• A gateway layer: A set of proxy endpoints that your scraping client connects to. These endpoints are hosted on standard data center infrastructure and serve as the entry point into the residential network.

• An orchestration/routing layer: The internal system responsible for selecting which residential exit node to use for a given outbound request, based on criteria like target geolocation, session persistence requirements, and node availability.

• The exit node pool: The fleet of residential devices registered with the provider, each maintaining a persistent connection back to the orchestration layer.

Thanks to this infrastructure, the scraping requests flow like that:

1. Your scraping client connects to the provider’s gateway endpoint and authenticates.

2. The gateway receives your request and passes it to the orchestration layer, which selects an available residential exit node from the pool based on your session’s parameters.

3. The orchestration layer forwards the request to the selected exit node over the persistent internal tunnel.

4. The exit node’s software initiates the actual outbound TCP connection to the target server from the residential device’s IP.

5. The target server receives a request from what appears to be a residential IP, processes it, and returns the response.

6. The response travels back through the same path in reverse: exit node → orchestration layer → gateway → your client.

This is why, from the target server’s perspective, the only visible connection is the one from the residential exit node. Your client’s IP, the provider’s gateway IP, and the internal routing infrastructure are entirely invisible.

Check the TWSC YouTube Channel

Types of Residential Proxies

Residential proxies are segmented into distinct types based on two primary axes: IP persistence (how long you hold the same IP across requests) and targeting granularity (how precisely you can select the exit node’s location).

This section reports and describes the common residential proxy categories that are based on those two axes.

Rotating Residential Proxies

This is the most common type. With rotating proxies, the provider’s orchestration layer assigns a new exit node—and, therefore, a new IP—for each outbound request, or at a configurable interval. From your client’s perspective, you connect to the same gateway endpoint every time, but each request exits from a different residential IP. This is the default behavior on most residential proxy platforms and is the correct choice for large-scale scraping where you need to distribute requests across a wide IP footprint to avoid per-IP rate limits. Belos is a schema that shows, at a high level, how this process works:

The rotation can be triggered in two ways, depending on the provider:

• Per-request rotation: A new IP is selected for every single HTTP request.

• Timed rotation: The same IP is held for a configurable duration before rotating.

In the context of web scraping, the primary tradeoff is that rotating proxies make stateful interactions difficult. If a target website tracks session state via server-side session IDs or cookies tied to an IP, switching IPs mid-session will break the session. So, for purely stateless scraping cases like fetching product pages, search results, and public data, rotation is ideal.

Sticky Residential Proxies

Sticky proxies allow you to hold the same residential IP for an extended, configurable period. This typically ranges from a few minutes up to 30 minutes, depending on the provider, and is bounded by the exit node’s availability online. The session is maintained by passing a session identifier in your proxy credentials, which instructs the orchestration layer to pin your requests to the same exit node for the duration of the session.

This type is essential for scraping workflows that require statefulness, like multi-step form submissions, login-based scraping, checkout flows, or any target that correlates sequential requests by IP and invalidates sessions on IP change. The risk is that if the exit node goes offline mid-session (the residential device disconnects from the network, which happens regularly on consumer connections), the session is broken, and the provider will typically reassign you to a new IP. This requires your scraper to handle session recovery; otherwise, your pipelines will break.

Static Residential Proxies

Static residential proxies (often known as ISP proxies) occupy a middle ground. These IPs are registered to an ISP (giving them a residential-looking ASN and WHOIS classification) but are hosted on stable data center infrastructure, rather than actual consumer devices. The provider procures these IPs directly from ISPs and hosts them on servers they control.

The key advantage over standard rotating residential proxies is stability and speed. Since the IP is hosted on a server rather than a consumer device, it doesn’t go offline unpredictably, latency is lower, and bandwidth is not constrained by a household connection. You get a fixed, persistent residential-classified IP that behaves like a data center proxy in terms of reliability.

The tradeoffs are:

• Cost: These are among the most expensive proxy types.

• Increasing detectability: As anti-bot vendors are building databases of known static residential/ISP proxy IP ranges, this kind of proxy works as a cat-and-mouse race to escape anti-bots.

Mobile Proxies

Mobile proxies are a specialized subtype where the exit node is a mobile device connected via a cellular network, and the IP belongs to a mobile carrier’s ASN. These are distinct from standard residential proxies because mobile IPs carry a specific characteristic: mobile carriers use large-scale carrier-grade NAT (CGNAT), meaning a single public IP is shared among potentially thousands of concurrent mobile users.

This has a profound implication for anti-bot systems: blocking a mobile carrier IP risks blocking thousands of legitimate users simultaneously, making mobile IPs extremely difficult for targets to block without significant collateral damage. As a result, mobile proxies are the hardest proxy type to block and command the highest price per GB in the market. They are most valuable for targets with aggressive anti-bot systems, typically heavily protected e-commerce sites, social platforms, and ad verification use cases.

The tradeoff is latency and bandwidth variability. The exit node is on a cellular connection, which is subject to signal quality, network congestion, and higher round-trip times compared to a wired residential or data center connection.

Residential Proxies Types: A Summary Comparison

The table below summarizes the subtypes of residential proxies, each with its own specific characteristics:

Need help with your scraping project?

How Residential Proxies Are Sourced and Why It Matters

The technical sophistication of residential proxy networks is surely interesting for techies. But what’s also interesting is where the exit nodes actually come from. This is because understanding the sourcing model is not just an ethical exercise. It has direct operational implications for scraping professionals because the legality and reputational standing of the proxies you use can affect your liability exposure and the long-term reliability of your infrastructure. It also has implications for companies working in highly regulated markets, where specific certifications matter more than anything else. We’ve seen recently how bad things can go from one day to another if your proxy provider is not fully compliant on both IP sourcing side and customer onboarding.

As introduced previously, providers build their residential IP pools by recruiting real consumer devices to act as exit nodes. The mechanisms for doing this fall into a spectrum that is described in the subparagraphs below.

SDK Monetization

The most widespread IP sourcing method is SDK embedding. In this model, a residential proxy provider operates a B2B program where mobile app developers and software vendors integrate the provider’s SDK into their applications. In exchange, the developer receives a revenue share or a flat fee per active device. When an end user installs the app, the SDK runs in the background and enrolls the device’s internet connection into the proxy pool.

The provider’s SDK typically does the following:

• Establishes a persistent outbound connection to the provider’s orchestration infrastructure.

• Monitors device state to only proxy traffic when the device is charging and connected to Wi-Fi (to avoid draining battery or mobile data, which would generate user complaints).

• Throttles bandwidth consumption to stay below the threshold of user noticeability.

• Handles NAT traversal to make the device reachable for outbound proxy sessions.

The apps embedding these SDKs span a wide range. Just to name a few:

• Free VPN clients.

• File utility apps.

• Weather apps.

• Games.

So, the SDK is not the app’s core function: it’s the monetization layer that the developer has grafted on.

Opt-In Peer Networks

Some providers operate a peer model, where users knowingly and explicitly enroll their devices in exchange for a direct benefit. This typically is free access to a premium VPN or bandwidth credits. The consent flow clearly explains that the user’s connection will be used to route third-party traffic.

This is the most ethically defensible sourcing model, as the value exchange is explicit and the user retains full awareness of what their device is being used for.

Ethical Considerations

The ethical landscape of residential proxy sourcing is genuinely complex, and it deserves some treatment to help you discern the right proxy provider to choose.

The Consent Problem

The foundational ethical issue is informed consent. When a residential device is used as an exit node, its owner’s internet connection carries traffic for a third party. This means that their IP address becomes associated with requests they did not initiate, to targets they may never have interacted with. If a scraping client uses that exit node to send requests that the target considers abusive, the device owner’s IP is the one that appears in server logs. The potential consequences for an unknowing participant range from having their IP blacklisted by services they use personally to, in extreme cases, being flagged by automated fraud or abuse detection systems.

This is why true informed consent in this context is very important. Also, for deep ethical reasons, informed consent would require a clear, plain-language explanation that the user’s internet connection will relay unknown third-party traffic, that their IP will be the visible origin of that traffic, and that this may affect their IP’s reputation.

Bandwidth and Resource Consumption

Enrolling a device in a proxy pool consumes the device owner’s bandwidth. In many markets, this resource is metered and has a real financial cost. In mobile-heavy markets where Wi-Fi and cellular data are tightly managed household resources, this is a non-trivial extraction of value from the device owner. This is another reason why proxy providers offer rewards to IP owners.

The Downstream Use Problem

Even when sourcing is fully and clearly consensual, the ethical chain does not end at the exit node: it extends to what the proxy traffic is used for. This is because a residential proxy provider cannot fully control what its clients scrape. This means that the same infrastructure used by a legitimate price monitoring service can be used for credential stuffing attacks, ad fraud, scalping bots, or scraping protected personal data.

For this reason, responsible providers implement terms of service enforcement, traffic monitoring, and client vetting. As a scraping professional, you are a link in this chain and carry some responsibility for the downstream ethics of the traffic you generate.

Legal Exposure

The legal status of residential proxy usage varies by jurisdiction and use case. In the United States, the Computer Fraud and Abuse Act (CFAA) has been interpreted broadly to potentially cover unauthorized access to computer systems. And whether bypassing a website’s IP-based blocks via residential proxies constitutes “unauthorized access” has been the subject of significant litigation.

In the European Union, the GDPR introduces additional complexity when scraped data includes personal information. So, note that using residential proxies does not insulate you from these legal frameworks. It only changes the network-layer attribution of the requests.

What to Look For in a Residential Proxy Provider

If you’re new to the scraping industry and need to get residential proxies, you may be overwhelmed by the offers that are on the proxy market. There are a lot of providers out there, and choosing the best one for you may be hard.

If you want to evaluate providers not only based on budget, the suggestion for you is to primarily make ethical considerations. If you don’t know where to start, indicators of a responsible residential IP sourcing model include the following:

• Published transparency reports detailing how exit nodes are recruited.

• Clear opt-in documentation and publicly accessible SDK disclosure pages.

• Active terms of service enforcement against prohibited use cases.

• Direct compensation models for exit node participants.

• Compliance certifications. Some of the most ethical providers have pursued SOC 2 or ISO 27001 certifications, which indicate a structured approach to operational governance.

Conclusion

Residential proxies are easy-to-use tools, but may be hard to understand. The gap between “it works” and “I know why it works” is where most engineers live, and it’s fine until it isn’t.

Understanding the full stack changes how you make decisions for your scraping projects. Why? Because it changes how you pick a provider, how you evaluate proxy types for a given target, and how you think about the ethical and legal weight of the infrastructure you’re routing traffic through.

So, now that you know how this infrastructure actually works, let us know: does it change how you’d pick proxies for your next scraping project?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

I’m picking up where I left off in a previous investigation, so I won’t repeat all the details here. Back then, I pulled libshopeeaegis.so (one of Shopee’s native crypto libraries) into Ghidra and rebuilt it in Python, byte for byte. That was a straightforward win: just an integer switch over standard AES, SHA, and HMAC, once you get past the intentionally confusing symbol names. But as I mentioned at the end of that article, Aegis is just a general-purpose toolbox, not the actual library that signs API requests. The real signer is a different beast, and that’s what I’m looking at now.

What actually gates the API

Capture any Shopee API request, strip the anti-fraud headers, replay it, and the backend answers with an HTTP error and code 90309999. That code is the whole story of this article. It is the anti-fraud rejection, and by the end you will have seen it come back from the app, from a plain HTTP client, and from two different real browsers.

Two headers carry the protection. On the Java side, two interceptors in okhttp, the HTTP client library the Android app is built on, attach them: com.shopee.app.network.antifraud.b and .d, which name themselves SecurityNewSapInterceptor and SecurityNewSapPostInterceptor. The first header is af-ac-enc-sz-token, a constant for the session:

af-ac-enc-sz-token: ghHLT8yNv0Xd8t9C2/t6Sw==|uJEC5q...JfDCw4ImZS5+gWyOO5Dr...8bd|rsGw3AWMR7Ryh7Jo|08|1

The right foundation to keep a smooth scraping workflow. anyIP helps teams collect web data more consistently with high-quality residential and mobile IPs worldwide.

Explore anyIP Proxies →

This token is a pipe-delimited string: an encrypted device fingerprint, a 16-byte wrapped key, an 81-byte ciphertext, a 12-byte nonce, and two version markers. It’s computed once per session and then reused. The second header, x-sap-ri, comes with a set of signatures keyed by short hex strings, and these change every request since they’re computed over the request itself. Both headers are generated by libshpssdk.so, a hefty 4.7 MB native library. The tricky part is the per-request signature, since that’s the bit you can’t just precompute and reuse.

Finding the signer, and deciding not to port it

libshpssdk.so exports exactly one symbol, JNI_OnLoad. Everything else is hidden, which tells you the native methods are registered at runtime with RegisterNatives, the Java Native Interface (JNI) call an app uses to bind native functions to Java methods, rather than exported by name. So we catch the registration as it happens. On arm64, RegisterNatives is a slot in the JNIEnv function table at byte offset 0x6b8. We hook that slot spawn-gated, before the app’s own code runs, and read out every method the library registers. The agent is in /shopee_sap_oracle/agent_oracle.js.

We are after one JNI signature. A request signer takes the request and returns headers, which in JNI terms is two byte arrays in and a string out, ([B[B)Ljava/lang/String;. One method matches. It is vuwuuwvw, registered at libshpssdk+0xc7ebc, on the obfuscated class com.shopee.shpssdk.wvvvuwwu, a name we recovered from the jclass argument of the registration call through java.lang.Class.getName().

When I opened that address in Ghidra, I found the function itself barely does anything. It just marshals its arguments and passes them to another function, along with a 440-byte blob and a table of function pointers. That’s the classic shape of a bytecode interpreter: the blob is the program, the pointer table is the opcode handlers, and the real logic lives in the bytecode, not in any native function I can easily read. This is where my earlier approach hits a wall. Last time, I could just read the primitive bodies and match them to AES and SHA. Here, there are no such primitives. To reimplement the signer, I’d have to reverse and interpret the whole virtual machine (VM), a project that would take weeks and is on the same level as breaking DroidGuard, Google’s own anti-abuse VM. So instead of porting it, I decided to just call it directly.

The one thing that works: the oracle

The signer is a Java method on a class we already resolved, so we drive the app’s own code. The agent exposes one RPC (remote procedure call) export that takes a URL and a body, turns them into the two byte arrays the method expects, and returns whatever the signer returns. The driver (/shopee_sap_oracle/frida_oracle.py) spawns the app, waits for the signer to register, and signs requests we choose. Re-running it against the current app, the signer registers in three seconds and answers:

[SIGN] url=https://mall.shopee.ph/api/v4/native/homepage body={"test":1}
   3c7b6059 = LR4te3nzTKb0RcbDqGs1bsXb0pB=
   71356940 = uMaRB6hpxBCq1BwZ2bFqGXjGFow=
   a0e0dcb3 = WFiutLJdM2C0t5Lr4yaoPr9CLn4=
   eaab58ec = 3BJOkkGhS53kj19RA2QqVLfC1m8sZFLSYm8W8S+Kl3yHU+vuzvb+...   (~2 KB)
   x-sap-ri = d9e7576a26efc4c9896e6716017c0e63cf0cae2c4c560ed7dd03

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

That’s the win, and I want to be clear about what it actually means. I can now produce the exact headers the app would attach for any request I choose, without needing to understand how the signer works under the hood. But before I trust these headers, it’s worth looking closely at the output, because it gives you a sense of how tough it’ll be to reuse them. The structural checks I’m about to show run on the captured values using /shopee_sap_oracle/analyze_headers.py. No device needed.

The short values look like base64 of a 20-byte digest, which is the length of HMAC-SHA1, so the easy reading is that they are keyed hashes and that is where a first pass leaves them. The easy reading is wrong. Decode seven of them as standard base64, and five come back with non-zero bits in the final padded position, which a conforming base64 encoder never produces from raw bytes:

  6m6r94xGLKAjTMYNMepdMiLQpSB=  len=20  NON-canonical (low2=1)
  wLJO02yz2I+FAZETVwC0aTCflyB=  len=20  NON-canonical (low2=1)
  1iwMbNiaPma2XLT7ToRs4jgiasE=  len=20  canonical
  47WhCSbrTx1r5FdFFlgtpHPN0OO=  len=20  NON-canonical (low2=2)
  LzCPR2/LYpvqP/z1HBXtuzj+8dQ=  len=20  canonical
  vmPAmagMmOlffOGIWF8MwJEBuNT=  len=20  NON-canonical (low2=3)

Check the TWSC YouTube Channel

So these aren’t just plain base64-encoded digests. Either the alphabet is shuffled or there’s some transformation after encoding, and that’s exactly the kind of detail that trips up anyone trying to reproduce the signature with HMAC-SHA1 and a standard encoder. The keys themselves are interesting too: within a session, no two requests share a key, and those eight-hex identifiers aren’t the actual header names you see on the wire. Plus, endpoints with a body return four entries, while GETs without a body return three. There’s a whole mapping layer from the signer’s per-request keys to the real header names, and I never managed to reverse that part.

Then there is x-sap-ri itself. It is 26 bytes: a leading counter, a three-byte session ID, an eight-byte middle field, a 01 marker at byte twelve, and a thirteen-byte tail. The counter is the interesting part, because it is coarser than it looks. In the fresh capture, both requests came back with counter d9 and session e7576a in front, and only the middle and tail differed:

d9 e7576a 26efc4c9896e6716 01 7c0e63cf0cae2c4c560ed7dd03
d9 e7576a b4c3b40323380413 01 d56d3f5c7eaef45395425b9720

So the leading byte isn’t just a simple counter that increments with each request. What actually separates one request from another is hidden in the middle and tail fields, and the token is tightly bound to the specific request through those parts. That binding is the first reason why a replay fails, and it’s where the real obstacles begin.

App wall one: the replay is out of sequence

The obvious next step was to sign a real product request using the oracle and try replaying it from outside the app. I picked a phone case from Shopee’s public site (shop_id=20396998, item_id=5420033304), had the oracle sign its product detail URL, attached the session’s af-ac-enc-sz-token and a cookie I grabbed with a one-shot header hook, and replayed the request with curl-cffi. The driver for this is /shopee_sap_oracle/replay.py. Here’s what Shopee sent back:

[*] oracle signed 4 headers: ['5cfcca02', 'c382fc01', 'cf1e6149', 'x-sap-ri']
==== REPLAY RESULT ====
HTTP 403
{"5":false,"2":false,"0":2,"3":90309999,"error":90309999, ...}

The status code changes depending on the endpoint, but the error code is always the same. Having a valid signature is necessary, but it’s not enough. This replay was always going to fail for two reasons: first, the x-sap-ri binding ties the token to the device’s own request sequence, so any injected, out-of-sequence request gets rejected, even with a fresh signature. Second, I put the signer’s per-request-keyed values under those same eight-hex keys, but I never actually confirmed that’s how the app names them on the wire. For a clean external replay, you’d need both of those right, and I could only be sure about one.

App wall two: the RASP cancels everything

The way to remove both doubts is to stop replaying and make the request from inside the app, where the sequence, the device state, the TLS, and the real header names are already coherent. The app builds its API calls on okhttp, so we went looking for its OkHttpClient at runtime with Java.choose. The probe is in /shopee_sap_oracle/crash_trace.js. There are more than a dozen live instances, and you spot the API client by its interceptors. The antifraud layer is not among the application interceptors where we first looked. It sits lower, among the network interceptors, which run after okhttp has set up the real connection, so it signs the request that is actually about to go on the wire.

As always, the code that will be used for the python reimplementation we’re showing now can be found in our GitHub repository reserved for paying users, inside the folder 110.SHOPEE-PART2.

Read more

If you’ve run a scraper at any real scale recently, you’ve felt the shift. The block didn’t come from too many requests per minute, and rotating to a fresh IP didn’t fix it. The page just quietly refused to load the data you came for. That’s fingerprinting doing its job, usually before your first byte of useful content arrives.

We keep hitting the same pattern. A pipeline that ran fine for months starts returning empty pages or CAPTCHAs, the proxy pool looks healthy, the rate limits are nowhere near tripped, and nothing in the logs explains it. What changed is not how many requests you send. It is how much the target now knows about the client sending them.

What browser fingerprinting is, and why it matters for scraping

Fingerprinting identifies and tracks a client from the unique combination of characteristics its browser and device expose, with no cookies, no login, and no explicit identifier required. It doesn’t ask “who is this user?” It asks “does this look like the same client as before, and does this client look real?”

The signals are individually boring. Browser version, OS, screen resolution, installed fonts, timezone, locale, how the device renders graphics. None of them identify you alone. Put them together and you get a profile distinctive enough to survive across sessions, IP changes, and incognito mode. For the defense side that’s a bargain, because one collection step feeds fraud detection, bot detection, rate limiting, and access control at once.

From IP-based blocking to identity-based detection

The traditional model was simple. Send requests, rotate IPs, stay under the rate limit. As long as each request looked like it came from a different address, you stayed under the radar.

Fingerprinting breaks that model cleanly. Rotate all the IPs you want, and if your browser fingerprint stays identical across them, you’ve just told the anti-bot that one client is hiding behind a dozen addresses. No real user does that. It is more suspicious than a single sticky IP would have been. The rotation you relied on becomes the tell.

How fingerprinting works under the hood

Modern systems don’t lean on one identifier. They build a layered profile across browser properties, device capabilities, rendering behavior, and network characteristics, then score how coherent the whole thing looks. The more consistent and ordinary the profile, the more trust it earns. That last part is the one most scrapers get backwards, and we’ll come back to it.

Passive signals

These come for free, straight from the HTTP request or a basic JavaScript query, with no heavy lifting required:

• User-Agent string (browser, version, OS)

• Accept headers (language, encoding support)

• Screen resolution and color depth

• Timezone and locale

• Installed fonts and plugins

Active signals

Active fingerprinting runs JavaScript to pull things that are much harder to fake, because they depend on how the hardware actually behaves:

• Canvas fingerprinting renders hidden graphics. Tiny differences in font rasterization, anti-aliasing, and GPU instructions produce a distinctive output.

• WebGL fingerprinting probes the GPU and surfaces details about the graphics hardware and driver stack.

• AudioContext fingerprinting reads the audio processing pipeline, which varies slightly across systems and adds more entropy.

These matter because they measure behavior, not claims. A User-Agent is just a string you can set to anything. A Canvas hash is the device answering for itself. Spoof the User-Agent but leave the Canvas output screaming “headless Chromium on a Linux server,” and the two halves of your story don’t match.

Network-level signals

Detection doesn’t stop at the browser. The network layer sanity-checks the rest of the profile:

• IP address and ASN (who owns the network)

• Geolocation consistency, checking IP against timezone against language

• TLS fingerprinting (JA3, and increasingly JA4), the shape of your TLS handshake, which most HTTP clients get wrong in a recognizable way

• HTTP/2 fingerprinting, the frame and settings ordering that Akamai in particular leans on

• Connection reuse and timing patterns

A browser claiming to be mobile Safari in Spain, connecting from a datacenter ASN with a Python-default TLS handshake, has three independent signals contradicting each other. Any one is forgivable. Together they’re a verdict.

What makes a fingerprint suspicious

Most fingerprints aren’t obviously fake. They’re built from real values: common browsers, standard resolutions, familiar operating systems. What anti-bot systems hunt for is how those real values fit together.

The first giveaway is a mismatch between attributes. Real devices follow predictable patterns. Certain browsers cluster on certain operating systems, resolutions cluster around popular hardware, fonts and rendering align with the platform. A macOS Safari profile serving Windows fonts, or a mobile User-Agent reporting a 2560-wide desktop viewport, breaks that pattern. Each discrepancy is explainable on its own. Stacked together, they aren’t.

Then there’s rarity. A perfectly unique fingerprint feels like good privacy and is terrible for blending in. Real users fall into large, overlapping groups of near-identical configurations. A profile built from attributes that almost never co-occur is both easier to track and quicker to flag. This is the over-randomization trap, and it’s the most common self-inflicted wound we see. People randomize every parameter and end up with a device that has never existed on Earth.

Consistency over time counts just as much. Real users are stable, same device, same browser, broadly the same setup from one request to the next. When one fingerprint shows up on an IP in one region and then again on an IP in another, or when key attributes drift between sessions, it reads as synthetic. Several IPs sharing a single identity is a pattern normal traffic just doesn’t produce.

Automation leaves its own tells. Even a carefully configured environment leaks something: a missing or stubbed API, a navigator.webdriver flag left on, slightly different JS execution paths, rendering quirks that don’t match a real build. No single tell is fatal. Detectors collect the small ones until the picture tips toward automation.

Last comes behavior. A flawless fingerprint still falls apart under robotic behavior. Pages loaded instantly with no intermediate navigation, clicks landing with pixel precision, sessions that fire one request and vanish, all of it undercuts the identity. Online, your identity is as much how you move as how you look.

Why fingerprinting breaks the old playbook

For years the routine was simple. Rotate IPs, randomize headers, respect rate limits. It worked because defenses were built around request-level signals, mostly where traffic came from and how fast it arrived. Fingerprinting moves the decision from the isolated request to a persistent identity, and most of the old tricks stop paying off.

Stateless requests became stateful identities. The old model judged each request on its own. Block one IP, grab another, keep going. Fingerprinting adds continuity. Change the IP and the browser environment can still look identical, or inconsistent in ways that link the sessions together. Rotating IPs no longer resets who you are.

Header randomization stopped working too. Rotating User-Agents and tweaking Accept-Language used to add diversity. Fingerprinting doesn’t read those values in isolation. It checks whether they agree with each other. Spin up a new User-Agent on every request while your rendering and JS execution stay constant, and you don’t look like many users. You look like one unstable client lying about itself. Tools like Apify’s fingerprint-suite and BrowserForge exist because building a coherent set of headers and fingerprint values is harder than randomizing them.

Datacenter-only setups draw scrutiny. Datacenter proxies are fast, cheap, and easy to rotate, but once the network is part of the identity that stops being free. A datacenter ASN has a different shape than residential traffic, and pairing one with a fingerprint that claims to be a consumer laptop is a contradiction. It won’t always block you, but it raises the baseline scrutiny, and stacked with any other inconsistency it pushes you over the line. This is why residential and mobile proxies have become table stakes on harder targets, even though they only fix the network half of the problem.

Detection also moved earlier. Older systems often gave you partial access before a rate limit kicked in. Now the fingerprinting script usually runs the moment the page loads, sometimes before any real content is served. Fail the check and you never reach the data. You get a CAPTCHA, a soft block, or a clean-looking empty response.

How to build a setup that survives fingerprinting

The goal isn’t to be invisible. It’s to be unremarkable, an identity where every layer tells the same ordinary story.

Run a real browser engine

Lightweight HTTP clients can’t fake Canvas, WebGL, or AudioContext, because those depend on a real rendering pipeline. So most serious setups drive an actual browser with Playwright, Puppeteer, or Selenium.

Vanilla automation still leaks, which is where the patched ecosystem comes in. playwright-stealth and puppeteer-extra-plugin-stealth paper over the best-known tells. undetected-chromedriver and its async successor nodriver harden Selenium-style automation, and patchright is a drop-in patched Playwright aimed at the same goal. When you need full managed profiles, anti-detect browsers like Camoufox (open-source, Firefox-based, with fingerprint injection built in), Kameleo, GoLogin, Multilogin, AdsPower, and Dolphin Anty handle the consistency for you. None of these is a magic bullet, since anti-bots study them too, but they clear the low-level signals that plain Playwright fails on day one.

Keep each fingerprint internally consistent

This is where over-randomization kills more scrapers than under-randomization ever has. Don’t change every parameter on every request. Give each session one stable, coherent identity, where the User-Agent matches the OS, the resolution matches the device class, and fonts, language, and timezone all tell the same story. If you generate fingerprints programmatically, lean on something built for coherence, like BrowserForge or fingerprint-suite, rather than rolling your own random values.

Align the fingerprint with the network

A fingerprint is never judged alone. Claim to be a mobile user in Spain while connecting from a datacenter in another region, and the whole session weakens. Treat proxies as part of the identity layer, not just a source of IPs:

• Residential or mobile proxies to match real-user network conditions

• Geo-targeted IPs so the IP, timezone, and language reinforce each other

• Session-based (sticky) routing for continuity across a session

And don’t forget the handshake. If your browser fingerprint is flawless but your TLS fingerprint is a Python default, JA3/JA4 will give you away. curl_cffi, curl-impersonate, and tls-client exist to make that layer match the browser you’re pretending to be.

Treat sessions as isolated identities

Each session should behave like its own user, with its own fingerprint, its own IP, and its own cookies and local storage. Don’t reuse one fingerprint across several IPs at the same time, and don’t let session state bleed between identities. In practice that means pairing one fingerprint to one IP for the life of a session, then rotating both together, the way a real device and its connection move as a unit.

Make the behavior match the identity

A perfect fingerprint can’t save robotic behavior. Real users don’t teleport onto deep pages, strip the data, and disappear. They land somewhere shallow, navigate, pause, scroll. Adding delays that reflect reading time, and routing through intermediate pages instead of hammering endpoints directly, does more for credibility than another round of fingerprint tuning.

Know when to stop building it yourself

Maintaining all of this is real, ongoing work, and the targets keep moving. Browser hardening, fingerprint coherence, proxy alignment, session isolation, behavioral pacing, every layer needs upkeep. At some point the maintenance cost outweighs the control you get from it, and pushing these layers down into infrastructure starts to make sense.

That’s the niche a managed scraper API fills. Instead of hand-coordinating proxies, browser environments, and anti-detection logic, you hand off the identity layer and keep your code about the data. Infatica’s Scraper API is built for that hand-off, with fingerprinting, session management, and request routing handled underneath, so what you maintain is extraction logic rather than evasion logic. It isn’t the right call for every project, but when anti-detection upkeep starts eating your roadmap, it’s a trade worth weighing.

Where this actually bites

Fingerprinting decides what data you can reach, how trustworthy it is, and whether your pipeline survives at scale. Getting past IP filtering is step one. Holding a believable identity for the whole session is the real game, and it’s where most of the projects we see actually break.

Price intelligence and travel feel it first. Travel platforms are among the most aggressive adopters. Prices move by location, device, and browsing pattern, so a setup that trips the defenses sees prices real users never see, or different ones every session. A hotel-pricing comparison is only as reliable as the identities behind the requests.

Search results are another. Search engines read the context of a query, not only the query itself. Weak fingerprinting earns an instant CAPTCHA. To capture what real users in a given region actually see, the requests have to look like they came from real users in that region, on consistent devices.

E-commerce monitoring runs into the same wall, and we’ve watched it happen most often on the luxury retailers sitting behind Akamai. Retailers tune availability, pricing, and promotions by user profile, and they use fingerprinting to tell shoppers from bots. Get flagged and you see hidden or reordered listings, throttling, or temporary bans, small distortions that quietly poison competitor-pricing analysis over time.

Ad verification and market research depend on seeing exactly what a real user sees: ad placement, localized copy, device-specific variants. Fingerprinting interferes by serving fallback content, blocking certain placements, or masking geo-targeted campaigns. That’s fatal when you’re validating campaigns across regions.

Account-based platforms lean hardest on fingerprinting to spot unusual access. These setups tend to fail not on a single request but because the session itself stops looking trustworthy. A sudden device-profile shift, an IP-location jump, or off behavior can each trigger a security check or kill the session.

Conclusion

Fingerprinting changed the question. It used to be “where is this request coming from?” Now it’s “does this client hold together as a believable identity, request after request?” Browser attributes, network signals, and behavior all feed that judgment, and the old reflex of rotating IPs and randomizing headers actively works against you when the layers don’t agree.

The setups that hold up aren’t the ones that randomize the hardest. They’re the ones that look the most ordinary: one coherent fingerprint, matched to one plausible network, behaving like a real person, held steady for the length of a session. The closer your setup sits to how real users actually browse, the more reliable your data collection becomes, and the more boring, in the best way.

Fortunately, not all providers have gone in that direction (think of the simple one-click Cloudflare Turnstile). However, many others are designing more intricate mechanisms to counter AI-based automation.

Is this the end of CAPTCHA solvers? And does this impact your scraping script? And how should we interpret these changes overall? Let me introduce you to the new generation of CAPTCHA systems to address all of these questions!

Give your AI a web data layer – Decodo's Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

Why CAPTCHAs Have Become More Sophisticated

Before the rise of modern AI, CAPTCHAs could already be solved automatically using traditional machine learning techniques and browser automation. The approach was based on three steps:

1. Detect the challenge.

2. Recognize its content.

3. Simulate the required user interaction.

For example, image classification models can identify the correct images in reCAPTCHA v2 (”Select all images containing hydrants”):

(For more details on reCAPTCHA solving, see our previous articles: Part I and Part II.)

Similarly, OCR models could extract text from distorted image-based CAPTCHAs.

Combined with browser automation, these models were often accurate enough to solve many CAPTCHA challenges without human intervention.

A good example was Botright, an open-source browser automation library that integrated machine learning models to solve CAPTCHAs. The project has since been abandoned (its last commit was over two years ago), but it demonstrated how effective this approach could be.

I’ve personally tested similar techniques. While they were far from 100% reliable, they were still capable of passing many CAPTCHA challenges.

CAPTCHA providers have long been aware of these techniques. As automated solvers became more capable, they responded by making CAPTCHAs steadily more intricate. This led to the introduction of puzzle-based challenges, such as those used by AWS WAF CAPTCHA, hCaptcha, and other providers.

Yet, the rapid advancement of multimodal AI models has raised the bar even further…

Instead of training specialized ML models for each CAPTCHA type, LLMs can understand images, reason about what they see, and let the controlled browser interact with the page much like a human would. That has forced CAPTCHA providers to develop more sophisticated techniques to distinguish humans from bots.

This is where we stand today!

Are CAPTCHAs Dead Because of AI?

The short answer is: No!

While multimodal AI models have fundamentally changed how CAPTCHA providers think about distinguishing human users from automated bots, traditional CAPTCHAs are far from obsolete. There are at least three reasons why:

1. Not every bot uses AI: Many scraping bots still rely on traditional browser automation or simple scripts. They don’t have access to AI tools capable of understanding and solving CAPTCHA challenges.

2. LLMs are not deterministic: AI models can solve the same CAPTCHA challenge most of the time, but fail spectacularly on others. No wonder you can find examples on Reddit and X of AI-powered browsers repeatedly failing at CAPTCHA challenges.

3. Solving the challenge is only part of the verification process: CAPTCHA systems usually evaluate much more than just the answer to the test. They also analyze behavioral signals such as mouse movements, cursor acceleration, click timing, scrolling behavior, touch interactions, and many other indicators. That’s why one-click, press-and-hold, and slider CAPTCHAs exist. An AI model may correctly identify the solution, yet still exhibit interaction patterns that look “robotic” rather than “human,” causing the verification to fail.

That said, AI is undeniably making life harder for CAPTCHA providers. Numerous research papers have shown that, given sufficient training data and realistic browser automation techniques (such as human-like mouse movements, scrolling, and timing), AI systems can successfully solve many traditional CAPTCHA schemes.

The real question is no longer whether AI can solve older CAPTCHAs, as it clearly can. The more interesting question is: Can it reliably solve the new generation of AI-resistant CAPTCHAs?

New Approaches to AI-Resistant CAPTCHAs

More and more CAPTCHA providers are experimenting with advanced challenges specifically engineered to resist AI-powered solvers.

In this section, I’ll focus on three approaches to newer CAPTCHAs that I’ve personally seen challenge even some of the most popular CAPTCHA-solving solutions.

hCaptcha’s New Puzzle-Based Challenges

hCaptcha has always been one of the more difficult CAPTCHA providers to bypass, thanks to its creative human verification tests.

As AI capabilities have advanced, hCaptcha has responded by introducing elaborate puzzles purpose-built to thwart automated solving:

Notice how many of these challenges require much more than simply clicking on the correct images. They can involve drag-and-drop interactions, spatial reasoning, and specific knowledge.

Instead of fighting every CAPTCHA, focus on avoiding them. anyIP gives you access to high-quality residential and mobile proxies that help your traffic blend in more naturally.

Start with AnyIP

Microsoft’s Multi-Option CAPTCHAs

Explore the Microsoft forums and Reddit, and you’ll find plenty of users complaining about Microsoft’s newer human verification challenges (example 1, example 2, example 3). Here are a few examples:

These challenges can be somewhat ambiguous. They tend to present multiple plausible answers, require you to search through numerous options, and may involve several consecutive steps before the verification is complete. As a result, they take considerably longer to solve and can be frustrating for legitimate users (especially if a mistake forces them to restart the entire challenge).

Temu’s Proprietary CAPTCHAs

Some platforms are moving away from standard CAPTCHA providers and building their own verification systems in-house. A notable example is Temu, which has recently introduced a restrictive, custom CAPTCHA mechanism:

These challenges go beyond simple image selection and require very specific actions. For instance, they can involve multi-step interactions such as drag-and-drop tasks or slider-based tests that must be completed in a precise way.

What they generally have in common is the heavy use of AI-generated images. This creates an interesting dynamic: AI is being used to produce the very content to stop AI-powered bots.

The Future of CAPTCHA: What Comes Next

While these new CAPTCHA challenges are definitely more difficult for AI to solve, it’s still unclear whether AI will evolve quickly enough to keep up in the ongoing arms race between scraping developers and CAPTCHA-protected websites.

What’s clear is that providers will continue experimenting…

Hand-Gesture CAPTCHAs

Hand-gesture CAPTCHAs represent a shift from traditional image-based verification toward real-world interaction signals.

Google has recently begun testing a new reCAPTCHA system that asks users to prove they’re human by performing simple hand gestures in front of their device camera.

When enabled, the browser requests camera access and records a short video of the user’s hand movement. The system then analyzes the footage by extracting around 21 hand-landmark points (measurements of finger and joint positions) to identify specific gestures.

This is an attempt to improve “liveness detection,” making it harder for automated systems to pass as real users. Still, it’s not impossible that AI-driven tools could eventually simulate these gestures using AI-generated videos via camera injection.

Check the TWSC YouTube Channel

Stronger CAPTCHAs, Worse User Experience?

Newer CAPTCHAs are significantly harder for AI to pass, but they also introduce a new layer of friction for us, the everyday users. That’s the main trade-off in the current wave of anti-AI verification systems.

As models become more capable, CAPTCHAs are forced to evolve in parallel, often becoming more demanding not only for bots but also for legitimate users.

As a result, many users have been complaining on Reddit and X about frustration, increasingly time-consuming challenges, and privacy concerns (such as having to use a camera just to complete a CAPTCHA).

This has led to growing criticism that could eventually push providers to relax some of these verification mechanisms, either by making them easier or less frequent (which would be a win-win scenario for web scraping).

It’s quite clear that the real objective is finding a balance between effectively blocking bots and avoiding unnecessary friction for legitimate users.

Are These New CAPTCHAs Effective Against AI?

These newer CAPTCHAs aren’t becoming more elaborate simply for the sake of increasing difficulty. Providers are well aware that the probability of AI reliably passing these checks at scale remains low, though not zero.

The idea is to design challenges that are intricate enough to deter automation, while still incorporating sufficient interaction signals and behavioral data to assess whether the user is actually human.

In essence, beneath the surface, the core principle hasn’t fundamentally changed. What we are seeing is an evolution in execution, rather than a reinvention of the underlying idea.

Need help with your scraping project?

Avoiding CAPTCHAs Remains More Effective Than Solving Them

The final question is: “Should these new CAPTCHAs scare us, web scraping experts/enthusiasts?” In a sense, they might, but ultimately this is not really a revolution…

The truth is that in web scraping, the most effective strategy is rarely about solving CAPTCHAs. Instead, it’s about avoiding them altogether.

CAPTCHA systems are just one layer of a broader anti-bot ecosystem, and triggering them is a signal that the traffic is already considered low-trust or detectable.

For this reason, the focus should be on prevention rather than resolution. Best practices involve:

1. Routing traffic through high-quality residential proxy networks with strong reputation IPs.

2. Using stealth browser environments such as Kameleo Docker, Camoufox, or Rayobrowse.

3. Managing and rotating browser sessions in automated scraping workflows.

The end goal is to consistently emulate real, human-like browsing behavior to minimize the chances of detection. If human verification tests still appear, CAPTCHA solvers will remain the go-to option, as they must evolve to keep up with new challenges in order to stay in business.

The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Conclusion

In this post, you’ve understood why CAPTCHAs have become more sophisticated and the role AI has played in this evolution. You’ve also seen what providers like hCaptcha, platforms like Microsoft, and sites like Temu are doing to defend against AI-powered automated scrapers and bots, as well as what the future of CAPTCHAs might look like.

I hope you now have a clearer understanding of the current and future CAPTCHA landscape. If you have any questions, feel free to leave them in the comments below. Until next time!

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

On July 2nd, the FBI, together with several industry partners, seized hundreds of domains tied to NetNut. For those who don’t know, NetNut is a large residential proxy service run by Alarum Technologies, a company that’s even listed on NASDAQ. The seized domains included netnut.com, proxyjet.io, and divinetworks.com.

This didn’t come out of nowhere. Let me walk you through how we got here, because the timeline is important to understand what happened.

How the takedown happened

It all started on June 19th, when three security firms published reports showing that NetNut’s residential proxy network was actually powering a botnet called Popa. The software was running on everyday devices like smart TVs and streaming boxes. Qurium, one of the security firms, stumbled onto this while looking into some denial-of-service attacks and traced the control servers back to pirated streaming apps. Then Brian Krebs made the story public, and about two weeks later, the authorities stepped in.

The takedown was a team effort: Google, the FBI, Lumen Technologies, The Shadowserver Foundation, and others all played a part. This is where things get interesting from a technical perspective, because taking down a proxy network is nothing like just unplugging a server.

Residential proxy networks are built to be decentralized. The exit nodes are actually millions of consumer devices scattered all over the world. Google estimated that NetNut had at least 2 million devices in its network. Obviously, you can’t just go and seize someone’s smart TV. What you can do is target the things those devices rely on to function as part of a commercial proxy service.

First, there’s the command and control layer. Google shut down the accounts and services that NetNut was using to control the malware. What I found interesting is that this C2 infrastructure was actually running on Google’s own services. This technique, called ‘living off trusted services,’ lets malicious traffic blend in with normal cloud traffic. Without command and control, there’s no way to route customer traffic through those devices.

Next is the enrollment pipeline. Google Play Protect began warning users and disabling apps that included NetNut SDKs, and it will keep blocking any new installs. If a proxy network can’t add new devices, it starts to shrink over time, since devices get reset, replaced, or just go offline.

Then there’s the commercial front door. By seizing the main domains, the FBI basically shut down NetNut’s storefront and made it impossible to bring in new customers at scale.

All of this together, according to Google, cut down the number of available devices by millions and seriously hurt NetNut’s business.

What makes this case different

I’ve seen proxy networks get taken down before, but NetNut is different. This wasn’t some anonymous group hiding on the dark web. NetNut was run by a company listed on NASDAQ, with investor relations, quarterly reports, and even a legal team that put out a statement saying they were aware of the seizure and cooperating with investigators.

It must be said that Alarum has rejected the accusations, disputing their accuracy. On the commercial and official side, NetNut has always presented itself exclusively as a legitimate proxy provider for enterprise data collection, without ever declaring any kinship with the clandestine Popa botnet. In its communications following the seizure, the company stated that neither Alarum nor NetNut had been formally contacted by the FBI or any other governmental authority in connection with these matters, while temporarily suspending data traffic through the affected network services in order to investigate the incident and assess the infrastructure involved. Markets, however, delivered their own verdict quickly: Alarum’s stock crashed by more than 70% at the end of last week, bringing its market cap down roughly 96% from its 2024 peak, and the company itself warned that prolonged disruptions are expected to have a material adverse effect on its operations and financial results.

It’s important to be clear about which part of NetNut’s network is involved here, because NetNut wasn’t just selling one type of proxy. They offered rotating residential proxies, static ISP proxies, mobile proxies, and regular datacenter proxies. Datacenter IPs are the least controversial, since they come from cloud and hosting providers with clear ownership. The takedown focused on the residential side, where NetNut actually used two very different sourcing models.

On one side there are the ISP proxies, sourced through DiviNetworks, the company NetNut originally grew out of. This is an above-board arrangement: as Spur documented in its research on how proxy providers co-opt entire networks, DiviNetworks recruits ISP partners with a straightforward revenue-stream pitch, terminates a GRE tunnel on the partner’s edge router, installs software on a VM inside the partner network, and uses policy-based routing to steer a slice of traffic through that VM and back out through NetNut’s infrastructure. From the partner’s perspective, a small amount of traffic is diverted; from NetNut’s perspective, selected egress from the partner’s public IP space becomes a commercial proxy product. The ISPs sign an agreement and get paid. You can argue about whether their subscribers know, but at least the contractual chain is in daylight.

The other side is the residential proxy pool, and this is where the Popa botnet comes into play. Researchers found that NetNut built this network by distributing SDKs for devices like smart TVs and streaming boxes. Synthient looked at over 20 real Popa publishers and didn’t see any of them asking users for consent. This is the part of the network that led to the takedown. When you have two million devices enrolled without real consent, that’s not a legitimate proxy network; it’s a botnet with a billing system, at least from a law enforcement point of view.

But the fact that divinetworks.com was seized along with the other domains tells me that investigators aren’t making a clear distinction between the different parts of the business. Even providers who think their sourcing is clean should take note.

This isn’t just a one-off event; it’s part of a bigger pattern

If this all sounds familiar, it’s because we’ve seen it before, and not long ago. The NetNut takedown follows right after Google’s disruption of the IPIDEA proxy network back in January 2026. At the time, IPIDEA was one of the biggest residential proxy operations out there, and NetNut’s main competitor.

Google’s post-mortem on IPIDEA included what I think is the key point here: when their own botnet starts to fall apart, proxy operators just buy capacity from competitors and become resellers. The proxy industry is tightly connected, and Google is pretty confident that a lot of popular residential proxy brands were actually just reselling NetNut’s botnet.

If you buy residential proxies from a brand you haven’t really checked out, pay attention. The IP pool behind your provider’s logo might be NetNut’s, or IPIDEA’s, or whoever is next in line.

The timing here is no accident. Bot traffic is exploding, especially with AI agents and LLM-based crawlers on the rise. In all my years doing this work, I’ve never seen web scraping infrastructure get this much attention from regulators and platforms. We’re in a real bind: we need more scraping and automation than ever, which means more proxies, but at the same time, there’s a real push for transparency about how these networks are built and used, so we can tell the good actors from the bad.

The lesson from NetNut is that just being transparent about where your IPs come from isn’t enough. You also need to know who is actually using your network. Alarum said they did KYC and due diligence on NetNut’s customers, but Spur found that NetNut didn’t really require corporate verification or real know-your-customer checks before selling proxy access. Down the line, resellers and white labelers usually didn’t do any KYC at all. Anyone who knew where to look could buy access with just a burner email and a few dollars in crypto. The result? In just one week in June 2026, Google saw 316 different threat groups using NetNut exit nodes, including cybercriminals and espionage actors.

What this means if you were a customer

Let’s be honest: losing a proxy provider isn’t really a technical problem. If you’re running a serious scraping setup, switching providers just means changing your endpoint. And if you’ve set up any kind of automatic fallback system, which you really should have, your pipelines probably won’t even notice the downtime.

The real headache is legal and compliance risk, especially if you’re working in a highly regulated industry.

Take financial markets, for example: I’ve worked in this space myself. Web scraping is a key part of what’s called alternative data: things like scraped prices, app usage stats, satellite images, and transaction data. Investment funds use these to get an edge on listed companies, alongside the usual financial statements and market data.

Anyone who has tried to sell web-scraped data to an investment manager knows what comes next: the due diligence questionnaire. I’ve been through this process, and it’s not a formality. The standard DDQ used in the industry, published by FISD’s Alternative Data Council, dedicates an entire appendix to web scraping and explicitly asks vendors whether they use proxies, whether those proxies are anonymizing, whether the proxy providers include information sufficient for scraped websites to contact the company, whether data is collected from pages behind logins or CAPTCHAs, and whether the vendor or its service providers have received cease and desist orders, subpoenas, or been subject to investigations in the past five years.

Now imagine you’re a data vendor facing that last question, and your proxy provider’s homepage is showing an FBI seizure banner. Telling a hedge fund that your infrastructure supplier is under federal investigation is not going to help you close any deals. In this business, your compliance is only as strong as the weakest link in your supply chain. Proxies are definitely part of that chain, whether your legal team knows it or not.

How I Think About Choosing Proxy Providers After NetNut

The reality is that residential and mobile proxies have always been the most controversial part of the proxy world. Unlike datacenter IPs or ISP deals, residential proxies need real people to agree to run a proxy on their devices in exchange for something, maybe money if they’re in a bandwidth-sharing program, or a free app, since developers get paid by proxy providers for every install.

Most people don’t realize how widespread this is. Spur scanned over 6,000 LG and Samsung smart TV apps and found that more than 2,000 of them had residential proxy SDKs inside. That means about a third of smart TV apps are quietly acting as proxy infrastructure. The real difference between legitimate and abusive setups comes down to consent and disclosure. Some providers, like Bright Data, Massive, and Oxylabs, at least spell out the arrangement in their terms of service, as Spur pointed out. For example, you can check Bright Data's peer program directly on bright-sdk.com, where you can find the certifications and compliance that they obtained.

Some providers have shown they’re willing to take a hit to keep their networks clean. For example, Bright Data has publicly shared that they dropped SDK partners who wouldn’t digitally sign their apps or who served ads leading to shady websites. These partners made up over 10% of their residential peer network, so it wasn’t a small decision. On the You might be skeptical about companies promoting their own good behavior, but I think this is the right direction: sourcing based on real standards, not just growth at any cost. They’re also founding members of the Alliance for Responsible Data Collection, together with other players like Sequentum and Common Crawl.

So here’s what I take away from all this. After what happened with IPIDEA in January and NetNut in July, picking a proxy provider based only on price isn’t just about quality anymore; it’s about risk. You need to ask the same questions the security researchers asked NetNut: How do you source your IPs, and can you prove consent? Who can actually buy access to your network, and what does your KYC process look like, even for your resellers? And what happens, both in contracts and technically, when abuse is found?

If your provider can’t answer these questions, someone else will, maybe with a seizure banner.

You buy proxies from a provider with the biggest IP pool you can find. You rotate IPs for every request. You randomize User-Agent strings. You connect to proxy servers from various regions. You add delays between requests. You expect to smoothly scrape the necessary data in no time, as you have made everything so your scraper does not leave any traces behind and is impossible to get caught. The next thing you see usually happens around request 150-300 - this is blocks. CAPTCHAs, error 403, redirects to honeypot pages serving you wrong data. The whole pipeline crumples down.

If this situation is painfully familiar, you have probably let session stickiness out of sight while building a proxy rotation layer - a common mistake. This article explains in detail how to rotate proxies to mimic real users so your scraper remains a reliable data source even at scale.

Why careless IP rotation may speed up bot detection

When you realize a website has blocked you out, the first instinct is to add more IPs - maybe buy proxies that perform better and are more adapted for dealing with detection systems. You try to add more geo-diversity. In other words, you use the same approach as before, just scale it. Sometimes it helps a bit; however, sometimes, blocks are back even faster.

The mistake here is that you are looking for a problem in the wrong place. It is not the lack of IP diversity, but the lack of session consistency.

Modern anti-bot systems like Cloudflare, Akamai Bot Manager, DataDome, or PerimeterX do not rely on the IP address as a primary method of bot detection. Instead, they consider a lot of signals and model session behavior. In such circumstances, your rotating efforts, if not strategic, can only make things worse and produce behavioral signals that are bot-like, not user-like. More IPs are not a fix for that problem. Modeling behavioral signals of genuine users is.

The anatomy of bot detection

In order to build a truly stealthy scraper with a rotation layer that gets the job done, it is important to understand first how detection happens - what exactly can give you away.

So, there are four layers of different signals that anti-bot systems analyze simultaneously and make a decision whether to let you be or deny access.

• Layer 1 - Network

IP reputation, ASN classification, type of IP (residential, mobile, or datacenter), and provider fingerprints belong to that layer. Also, if a proxy or VPN IP is known, it would also be shown here.

This layer is the one developers usually optimize the most for. Ironically, this is the least important layer for security systems. That is why you fail when following scraping tutorials that offer to simply set as big an IP pool as possible and rotate addresses for every request. It is not because you have done something wrong or your proxies are bad, but because this approach does not work for scraping at scale.

• Layer 2 - TLS/HTTP fingerprints

HTTP clients’ fingerprints are nearly as unique as an IP address. TLS handshake leaves quite a trace with its cipher suite order, list of supported extensions, and elliptic curves. HTTP/2 creates another signal with header prioritization, window size, and frame ordering. The combination of TLS and HTTP/2 fingerprints is often enough for security systems to figure out whether it’s human-generated or bot-generated traffic, even without considering signals from other layers.

Widely used scraping libraries, for example, curl, requests, httpx, aiohttp produce distinct, detectable fingerprints. Python is a popular choice for building scrapers now; still, if you use its requests library, a calibrated WAF would spot it from TLS handshake alone, User-Agent headers be forgotten, whatever they say.

• Layer 3 - Behavioral

Requests cadence, referrer chain integrity, cookie handling, navigation patterns, and timing distributions are evaluated here. Even the delays between requests, if they are too precise and equal, can fuel suspicions, as humans do not make requests with a machine’s precision. Real users’ referrers have logic to them. Human users’ cookies are consistent and show that they came from a search engine page. They do not disappear mid-session or mid-navigation.

• Layer 4 - Session Graph

This layer evaluates whether the sequence of cookies, tokens, page flow, and state transitions within the session makes sense for a human user. That is a layer most scrapes fail to pass, as not so many developers pay attention to coherence. Cause even if stand-alone signals look normal, it is important that they make sense when evaluated together. For example, a user visits an e-commerce website. Their session token would show that. Their viewport size would stay the same throughout the visit. They will go between pages with different time intervals, reflecting reading and decision time.

Coherence is the key that opens or closes a door for your scraper. And one proxy rotation at the wrong time can reveal your scraper’s true identity.

Session stickiness: a true solution

In the context of a proxy architecture, session stickiness is a condition when all requests that belong to the same logical session are routed via the same IP address. At the very least, it must be the same subnet, geo location, and ASN cluster.

Now, let’s define what a session may mean, cause in terms of scraping, there are several options possible.

• Authentication session

The most obvious case. You log in, receive a session token, and proceed to the data. If your IP changes somewhere between the login request and the first data request after authentication, to the target server, it looks like a user logged in from Paris and then suddenly that same user is asking for data from Seoul. Session invalidation or re-verification is guaranteed.

• Stateful pagination

This type is subtler and catches even sophisticated scrapers off guard. The thing is that many modern applications track your pagination state on a server’s side - in other words, a server “remembers” a page you are currently at and ties it to your IP. If the IP address changes between pages 3 and 4, the server loses this track. It may either block you or return incorrect data.

• Multi-step forms

Cases when you need to simulate checkouts, survey scrapers, or lead generation cases are the most session-sensitive. The state that was established at the beginning is carried forward across the rest of the steps. A mid-process IP change is a red flag in such a scenario; detection systems will react immediately.

To sum up, it is not random IPs you need, but sessions and a rotation strategy that considers session coherence.

Building the rotation layer

Time to get to work. Remember: rotating session objects, not IP addresses.

The session object model

A session object, in simple words, is a set of signals that make a set of requests look like they originated from a real human user.

@dataclass
class ProxySession:
    session_id: str
    proxy_endpoint: str          # host:port or provider session token
    cookies: dict = field(default_factory=dict)
    headers_profile: dict = field(default_factory=dict)
    geo: str = ""                # e.g. "pl,us,en"
    asn: str = ""                # e.g. "12345"
    created_at: float = field(default_factory=time.time)
    last_used: float = field(default_factory=time.time)
    request_count: int = 0
    flagged: bool = False

    def is_expired(self, ttl_seconds: int = 600, max_requests: int = 200) -> bool:
        age = time.time() - self.created_at
        return self.flagged or age > ttl_seconds or self.request_count >= max_requests

The headers_profile is not just a User-Agent string. It is a set of HTTP headers that are used for each and every request of one session: User-Agent, Accept-Language, Accept-Encoding, Accept, sec-ch-ua, sec-ch-ua-platform, sec-ch-ua-mobile. They must stay the same for every request. If one request is coming as if from Chrome browser version 108, and another one from version 124, detection systems will notice a mismatch. Even without knowing exactly that you are a scraper, such a mismatch is enough to block you.

Sticky session assignment

Any scraper must use the same session object within one session - this is a rule. So the session assignment must not be random, but based on that rule to prevent any mix-ups.

import hashlib
import threading
from typing import Dict

class SessionPool:
    def __init__(self, proxy_provider):
        self.provider = proxy_provider
        self._sessions: Dict[str, ProxySession] = {}
        self._lock = threading.Lock()

    def _make_key(self, job_id: str, target_domain: str) -> str:
        raw = f"{job_id}:{target_domain}"
        return hashlib.md5(raw.encode()).hexdigest()[:12]

    def get_or_create(self, job_id: str, target_domain: str) -> ProxySession:
        key = self._make_key(job_id, target_domain)
        with self._lock:
            session = self._sessions.get(key)
            if session is None or session.is_expired():
                session = self.provider.new_session(geo=self._infer_geo(target_domain))
                self._sessions[key] = session
            session.last_used = time.time()
            session.request_count += 1
            return session

    def _infer_geo(self, domain: str) -> str:
        # Simplified — in production, map TLDs/domains to geo requirements
        tld_map = {".pl": "PL", ".de": "DE", ".co.uk": "GB", ".fr": "FR"}
        for tld, geo in tld_map.items():
            if domain.endswith(tld):
                return geo
        return "US"  # Default

Notice that the session key is determined by job_id + target_domain, so the scraper sticks to the same session until it expires. The session expiration is also based on 2 rules - TTL (sessions older than 10 minutes automatically get refreshed) and request count (after 200 requests, the session gets refreshed to prevent statistical fingerprinting from long-lived sessions).

Geo-tied rotation

Another life hack is to rotate IPs within the same geo or ASN cluster even when a session has expired, and you are spinning to the new one. Even between sessions, sudden changes (like jumping from Tokyo residential IP to Toronto residential IP) are noticeable and suspicious. When that happens, it looks more like a VPN switch, not a user returning.

Result-oriented proxy providers offer sticky sessions. In such a case, a session token has a location backed into it, and the provider guarantees that the traffic will originate as if from that location during a session’s lifetime.

Header profile coherence

Not only must IP remain the same throughout the session. If you rely on the same address, but the header profile changes from request to request, it is like a “Hello, I’m a scraper” sign for detection systems.

HEADER_PROFILES = [
    {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
        "Accept-Encoding": "gzip, deflate, br",
        "sec-ch-ua": '"Chromium";v="124", "Google Chrome";v="124", "Not-A.Brand";v="99"',
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": '"Windows"',
    },
    # Add more profiles reflecting real browser market share distributions
]

One session - one profile. No changes mid-session.

Retry logic

Retry logic is another thing that can blow your cover. When a request fails due to some reason and the target server instantly receives another one from the same origin, it most certainly means bots are in the game, as humans do not act with such light speed. An even less obvious trap is retrying with a different session object when your current session simply needs a brief backoff.

The better approach is to retry while relying on the same session object and a slight delay. If you receive a clear block signal, like 407, 429 errors with specific headers, or CAPTCHA, switch to the new session.

def fetch_with_retry(
    url: str,
    pool: "SessionPool",
    job_id: str,
    target_domain: str,
    max_retries: int = 3,
) -> requests.Response:
    failures: list[str] = []

    for attempt in range(max_retries):
        session = pool.get_or_create(job_id, target_domain)
        try:
            response = make_request(url, session)
            if response.status_code in (407, 429) or is_captcha(response):
                session.flagged = True
                failures.append(
                    f"attempt {attempt}: session {session.session_id} "
                    f"blocked (status={response.status_code})"
                )
                if attempt < max_retries - 1:
                    base = min(2 ** attempt * 2, 30)
                    time.sleep(base * (0.8 + 0.4 * random.random()))
                continue
            return response
        except (requests.Timeout, requests.ConnectionError) as e:
            failures.append(f"attempt {attempt}: {type(e).__name__}: {e}")
            if attempt < max_retries - 1:
                base = min(2 ** attempt * 2, 30)
                time.sleep(base * (0.8 + 0.4 * random.random()))

    raise MaxRetriesExceeded("All attempts failed:\n  " + "\n  ".join(failures))

There is an important difference: TransientError (which includes network hiccup and timeout) retries with the current session. BlockedError marks the session as flagged, which in turn triggers is_expired() and forces a new session creation.

Scaling concurrent sessions

At a small scale, the session pool model works just fine. However, if your scraper runs on 50+ machines simultaneously, you need all of them not to create new sessions each, but to stick to the one.

A Redis-backed session pool is a solution. Basically, Redis is a shared notepad that lives on a server and that every machine can read and write to, so that there is no chaos in sessions. Store each ProxySession object (serialized as JSON) with a TTL key:

import redis
import json

r = redis.Redis(host='localhost', port=6379, db=0)

def store_session(session: ProxySession, ttl: int = 600):
    key = f"session:{session.session_id}"
    r.setex(key, ttl, json.dumps(session.__dict__))

def load_session(session_id: str) -> Optional[ProxySession]:
    data = r.get(f"session:{session_id}")
    if data:
        return ProxySession(**json.loads(data))
    return None

If a proxy provider offers sticky sessions, your Redis only needs to track the session token, not the exit node state. A provider handling IP consistency and you dealing with cookies and headers is the preferred architecture for effective scaling because building and maintaining a distributed session pool is good 2-4 weeks of work. You need developers to do it - and add ongoing maintenance as websites update their detection systems.

Testing rotation layer

To be sure everything works fine, verify your rotation layer. Send 10 requests to see whether all of them really go via the same IP address.

import requests

def verify_session_stickiness(session_token: str, n: int = 10):
    proxy = f"http://user__sessid.{session_token}:pass@gw.dataimpulse.com:823"
    seen_ips = set()

    for i in range(n):
        r = requests.get("https://httpbin.org/ip", proxies={"https": proxy})
        ip = r.json()["origin"]
        seen_ips.add(ip)
        print(f"Request {i+1}: {ip}")

    print(f"\nUnique IPs: {len(seen_ips)}")
    assert len(seen_ips) == 1, f"FAIL: Session leaked across {len(seen_ips)} IPs"
    print("PASS: Session sticky across all requests")

verify_session_stickiness("your_session_token_here")

To verify headers, use https://httpbin.org/headers and compare headers across requests. Any differences in User-Agent, Accept-Language, or sec-ch-ua between requests of the same session are a problem.

If you want a deeper test, use bot.sannysoft.com or pixelscan.net. Those sites show what detection systems see - WebDriver flags, inconsistent browser APIs, JA3 anomalies, timezone/language mismatches. You see what exactly is wrong and at which layer. This is an easier, faster, and cheaper way than constant tries and errors.

Final thoughts

Scrapers that remain undetected are not the ones with the largest IP pools. They are the ones that mimic real users’ behavior the most persuasively - when the IP, the TLS fingerprint, the header profile, the cookie state, and the navigation flow all together create traffic patterns that are typical for a genuine human visitor.

IP rotation is necessary, but it is a tactic, while session architecture is a strategy. When you adopt that approach, you feel the difference - in lower block rates and higher data accuracy, in how much less firefighting your team goes through every time security systems update. Scalability is another benefit.

Also, there is an undeniable tendency for detection systems to develop, and so must your scrapers do. Rotation that is based on sessions, not single IPs, allows for that development.

If you want to skip the “build infrastructure from scratch” part, DataImpulse is an ethical proxy provider that offers both rotating and sticky sessions. You can set the duration of a session manually - up to 120 minutes is possible. DataImpulse (dataimpulse.com) provides 90M+ of residential, mobile, and datacenter enterprise-grade addresses in 195 locations with advanced targeting options (extra costs) to help maintain session coherence. Web scraping, SERP tracking, price monitoring, and other sensitive scenarios are among its use cases. Prices start from $1/GB - for non-expiring traffic and 24/7 human support.

Among the many tools a scraping engineer should master are headless browsers. But if you want to be completely honest, in the scraping industry there’s some confusion about the concepts of “headless browsers” and “browser automation tools”.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

---

In this article, you’ll get your ideas clear on headless browsers and browser automation tools. You’ll also discover the Oxylabs Headless Browser and learn how to use it in practice against common anti-bots. In particular, you’ll see the difference in servers’ responses when hitting the same target URLs with and without the Oxylabs solution.

Let’s get into it!

What is a Headless Browser?

A headless browser is a fully functional web browser that operates without a graphical user interface (GUI). To understand what this means in practice, it helps to discuss what a browser actually does under the hood.

When you open a browser like Chrome and navigate to a URL, the browser performs a precise sequence of operations:

• It sends an HTTP request to a server.

• It receives an HTML document.

• It parses that HTML into a DOM (Document Object Model) tree.

• It fetches and applies CSS stylesheets to compute the visual layout, downloads and executes JavaScript files, and processes any asynchronous network requests (XHR/Fetch API calls) triggered by that JavaScript.

• It renders everything into visible pixels on your screen.

Below is an image that graphically shows the whole process:

A headless browser performs every single one of those steps, except painting pixels to a physical display. It still builds the DOM, executes JavaScript, processes AJAX calls, and maintains a fully accurate representation of the page state in memory. But it does it without needing a graphical rendering environment.

It’s also important to understand the difference between a headless browser and a basic HTTP client. Tools like Python’s requests library or curl are pure HTTP clients. This means that they send a request and return whatever raw response the server provides. They have no concept of a DOM, no JavaScript engine, and no rendering pipeline.

The most widely used headless browsers are the following:

• PhantomJS: One of the first dedicated headless browsers, built on an older version of WebKit. It was widely used before 2017 but has since been officially abandoned and is no longer maintained. You may still encounter it in legacy codebases, but it’s better not to use it for new projects.

• Headless Chrome/Chromium: Google officially introduced headless mode in Chrome 59 (2017), effectively making PhantomJS obsolete overnight. It exposes the Chrome DevTools Protocol (CDP) as its programmatic interface, which automation frameworks connect to to send instructions and receive data.

• Headless Firefox: Mozilla introduced headless support in Firefox 55. Powered by the Gecko rendering engine and the SpiderMonkey JavaScript engine, it’s a strong alternative to Chromium, particularly useful when you need to scrape or test behavior tied to the Gecko engine specifically.

• WebKit: WebKit is the browser engine that powers Apple’s Safari. Playwright bundles a version of WebKit that can be run in headless mode on non-Apple platforms, giving you the ability to simulate Safari’s rendering behavior on Linux or Windows servers (something that would be otherwise impossible).

• Splash: A lightweight, scriptable headless browser built specifically for web scraping, developed by the Scrapy team. It is written in Python and Lua, exposes an HTTP API, and integrates natively with the Scrapy framework, making it a beginner-friendly option for Python-based scraping pipelines.

Your scraping workflows deserve a proxy infrastructure that just works. With Swiftproxy on your side, consistency is built-in.

Try Swiftproxy today

Headless Browsers vs. Automation Frameworks: A Critical Distinction

At this point, it’s essential to draw a clear architectural boundary between two categories that are frequently, but incorrectly, used interchangeably: headless browsers and browser automation frameworks.

As said in the previous paragraph, a headless browser is the actual browser engine itself. So, it’s the software responsible for all the core browser operations described above.

A browser automation framework, on the other hand, is a programming library or tool that sits on top of a headless (or full) browser and provides a developer-friendly API to control it programmatically. The framework itself does not render pages, execute JavaScript, or manage network traffic: it delegates all of that to the underlying browser engine. What it provides is an abstraction layer to navigate to URLs, click elements, fill in forms, wait for conditions, intercept network requests, take screenshots, and to perform any similar activity.

Think of it this way: the headless browser is the engine of a car, and the automation framework is the steering wheel, pedals, and dashboard. You need both to drive, but they are fundamentally different components with different responsibilities.

This distinction also has a practical implication. If a website employs bot detection based on browser fingerprinting, it’s the underlying browser engine’s characteristics that matter, not the automation framework on top of it. This means that switching from Puppeteer to Playwright doesn’t change your browser fingerprint if both are driving the same Chromium engine with the same configuration.

The list below reports the programming libraries and tools used to control headless (or full) browsers programmatically:

• Puppeteer: A Node.js library developed and maintained by Google that provides a high-level API to control Headless Chrome/Chromium over CDP. It was the first major modern automation framework to replace PhantomJS and remains widely used in the JavaScript/Node.js ecosystem.

• Playwright: Developed by Microsoft, Playwright is a more modern and feature-rich automation framework that supports Chromium, Firefox, and WebKit under a single unified API. It has become the preferred choice for many scraping engineers due to its superior handling of async operations, automatic waiting strategies, built-in network interception, and robust multi-browser support. On the side of programming languages, it’s available for Node.js, Python, Java, and .NET.

• Selenium: The oldest and most widely adopted browser automation framework in existence, originally released in 2004. Unlike Puppeteer and Playwright, Selenium communicates with browsers via the W3C WebDriver protocol. This means it supports a broader range of browsers and is the de facto standard in enterprise testing environments. Selenium supports multiple languages, including Python, Java, C#, Ruby, and JavaScript, which gives it an exceptionally large community and an extensive ecosystem of documentation and tooling. For engineers working in Python, Selenium with a headless Chrome or Firefox driver is still a very common scraping approach. Its main trade-off is that it can be slightly slower and less ergonomic for modern async-heavy scraping scenarios.

Why Do Engineers Need a Headless Browser?

When you start defining how to scrape a website, your first instinct is often to reach for a lightweight HTTP client. And, in many cases, that is the right call. Simple, server-rendered websites that return complete HTML in their initial response are perfectly scrapable with tools like requests coupled with BeautifulSoup.

However, as you begin working with modern websites, you will quickly encounter scenarios where an HTTP client falls flat. Understanding why this happens (and when a headless browser becomes necessary) is one of the most fundamental skills you need to master for your career.

This section discusses the most common scenarios when you need to use a headless browser for scraping target websites.

JavaScript-Rendered and SPA Content

The most common reason to turn to headless browsers is client-side JavaScript rendering. A large proportion of modern websites, in fact, are built as Single Page Applications (SPAs) using frameworks like React, Vue.js, Angular, or Next.js. In these architectures, the server does not send a fully populated HTML page. Instead, it sends a minimal HTML shell along with a bundle of JavaScript files. The browser then executes that JavaScript, which makes additional API calls to fetch the actual data and dynamically constructs the DOM.

If you scrape the raw HTML with an HTTP client, you will receive that empty shell and find none of the content you are looking for. A headless browser, instead, loads the page as a real user’s browser would, waits for the JavaScript to finish executing, and exposes the fully populated DOM for you to extract data from.

AJAX and Asynchronous Data Loading

Closely related to SPAs is the broader pattern of asynchronous data fetching. Even on websites that are not full SPAs, it’s extremely common for certain page sections to be populated via AJAX requests that fire after the initial page load. These requests hit internal or external APIs, and the responses are injected into the page dynamically.

With a headless browser and a tool like Playwright, you can intercept these network requests directly. Also, this is a powerful web scraping technique. Rather than scraping the rendered HTML, you capture the raw API response (usually as JSON), which gives you clean and structured data to work with.

Infinite Scroll and Pagination

Many websites like social media platforms, job boards, and e-commerce product listings implement infinite scroll. This means that new content is loaded dynamically as the user scrolls down the page. On the practical side, this means that there is no “Page 2” URL to follow. The content doesn’t exist in the DOM until a scroll event triggers a new batch of data to load.

In this scenario, a headless browser allows you to simulate scroll events, wait for new content to be injected, and continue extracting data until a stopping condition is met. This is fundamentally impossible to replicate with a plain HTTP client.

User Interaction Simulation

Some content is only accessible after specific user interactions, like clicking on a button, selecting a value from a dropdown menu, submitting a search form, switching between tabs, and similar. A headless browser gives you a full interaction API that allows you to click on elements, type into input fields, select options, trigger keyboard shortcuts, and simulate virtually any user action.

This is particularly important when scraping search-driven websites like travel booking sites, real estate portals, and any similar websites where results are generated after the user performs actions.

Bot Detection and Browser Fingerprinting Evasion

Many high-value websites deploy anti-bot systems that analyze incoming requests to determine whether they originate from a real browser or an automated script. These systems are trained using machine learning models to check for dozens of signals. A headless browser provides a much more convincing browser environment than a raw HTTP client.

In particular, with additional configurations and stealth patches, you can significantly reduce the likelihood of your scraper being detected and blocked. Of course, this is an arms race: anti-bot technology continues to evolve, and a headless browser alone is not a silver bullet.

Check the TWSC YouTube Channel

What Is Oxylabs Headless Browser?

Oxylabs Headless Browser is a cloud-based browser automation solution that allows engineers to run and control remote browser sessions without the overhead of managing browser infrastructure locally or on their own servers. Rather than spinning up a local Chromium or Firefox instance on your own machine or cloud VM, Oxylabs Headless Browser offloads all of that complexity to dedicated remote servers maintained by Oxylabs.

The result is a production-ready and fully managed browsing environment that you connect to via a standard WebSocket (WSS) connection, and control using any automation framework that supports the Chrome DevTools Protocol (CDP) exactly as you would a local browser instance.

The image below shows, at a high level, the architectural difference between traditional browser automation and using Oxylabs’ solution:

The currently available browser instance is a Chrome-based environment (ubc.oxylabs.io). This runs high-performance Chromium instances with advanced stealth configurations on dedicated servers with integrated residential proxies.

Key Features

What differentiates Oxylabs Headless Browser from a self-managed setup is the set of capabilities that are baked directly into the infrastructure. Rather than requiring you to implement them yourself, it provides you with the following:

• Integrated residential proxies: Every browser session is automatically routed through Oxylabs’ residential proxy network. This eliminates the need to manage a separate proxy integration and ensures that outgoing requests carry legitimate residential IP addresses, reducing the risk of IP-level blocks.

• Automatic CAPTCHA solving: The platform detects and solves CAPTCHAs by default on page load. For CAPTCHAs that appear mid-session, it exposes an event-based API via the window object that your script can subscribe to in order to pause and resume automation around the solving process.

• Geolocation targeting: You can pin your browser session to a specific country, state, or city by appending query parameters directly to the WebSocket connection URL. This is essential for scraping geo-targeted content, verifying localized pricing, or accessing region-restricted pages without maintaining a geographically distributed proxy infrastructure yourself.

• Device type emulation: The p_device parameter allows you to emulate desktop, mobile, or tablet environments. This gives you access to device-specific content, responsive layouts, and mobile-only UI elements or CAPTCHA variants.

• Session inspection: This is a debugging feature that uses VNC (Virtual Network Computing) to give you real-time visual access to a running browser session. This is invaluable when diagnosing issues that are not apparent from logs alone. For example, for unexpected page rendering, broken interaction flows, or timing-related problems in your automation script.

• MCP integration: Oxylabs Headless Browser supports connection via the Model Context Protocol (MCP), enabling AI systems such as Claude Desktop or Cursor to drive browser automation sessions autonomously, which opens the door to AI-driven scraping and web navigation workflows.

How Oxylabs Headless Compares to Self-Managed Headless Browsers

The practical significance of Oxylabs Headless Browser is best understood by contrasting it with the self-managed alternative. When you run Playwright or Puppeteer against a local Chromium instance, you are responsible for the entire stack:

• Installing and maintaining the browser binary.

• Managing proxy rotation and authentication.

• Implementing CAPTCHA-solving integrations.

• Applying stealth patches to avoid fingerprinting.

• Handling concurrency limits.

• Ensuring your infrastructure scales with your scraping workload.

Oxylabs Headless Browser, instead, abstracts all of that away behind a single WSS endpoint. In practice, this means that your code remains identical to what you would write against a local browser. The connection string is the only thing that changes.

This makes Oxylabs’ solution particularly suited for teams that need to scale browser-based scraping quickly without investing in dedicated infrastructure engineering.

As a final note, it’s worth considering that billing is based on traffic consumed (per GB). This means traffic optimization becomes an important cost management practice when operating at scale.

Need help with your scraping project?

Testing Common Anti-bots The Hard Way: Step-by-step Tutorial on How to Hit Them Headless Mode

In this section, you will learn how to hit common anti-bots in headless mode using Playwright. The idea is to compare the results obtained in headless mode with the ones obtained using Oxylabs Headless Browser.

The target websites with their anti-bots are:

• Ssense → Cloudflare

• Gucci → Akamai

• Loccitane → DataDome

• Empire-cat→ Kasada

Let’s get into the tutorial!

Prerequisites and Requirements

To replicate this tutorial, you need at least Python 3.8 installed on your machine.

Create a new folder for your project:

mkdir test-antibots

At the end of this step, the test-antibots/ folder will have the following structure:

test-antibots/
   |
   |-- tester.py
   |
   |__ venv/

Where:

• tester.py is the Python file that contains the scraping logic to test the antibots.

• venv/ is the folder that contains the virtual environment.

All the files can also be downloaded from our GitHub repository, available to anyone.

In the activated virtual environment, install the needed libraries:

pip install playwright chromium

Very well. You are ready to code your tester!

Step #1: Code The Scraper to Test the Anti-bots

To test the antibots, write the following in tester.py:

from playwright.sync_api import sync_playwright

# Target pages with their anti-bots
targets = [
    ("<https://www.ssense.com/en-us/men/designers/acne-studios>", "Cloudflare"),
    ("<https://www.gucci.com/us/en/pr/women/shoes-for-women/sandals-for-women/flat-sandals-for-women/womens-slide-sandal-with-horsebit-p-834427AAE046273>", "Akamai"),
    ("<https://loccitane.com/en-us/olio-jeunesse-divine-30ml-27DH030I22.html>", "DataDome"),
    ("<https://www.empire-cat.com/equipment/used/catalog/aa/j1s03815>", "Kasada")
]

results = []

# List of titles or substrings that indicate common blocks/challenges
BLOCK_INDICATORS = [
    "just a moment", "access denied", "attention required", 
    "cloudflare", "ddos protection", "captcha", "verify you are human"
]

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context()

    for url, antibot in targets:
        page = context.new_page()
        try:
            # We capture the response to check the status code
            response = page.goto(url, wait_until="domcontentloaded")
            page.wait_for_timeout(5000)

            title = page.title() or ""
            status_code = response.status if response else 0
            
            # Validation Logic
            is_blocked = False
            error_reason = ""

            # 1. Check HTTP Status (403 Forbidden is a classic block)
            if status_code >= 400:
                is_blocked = True
                error_reason = f"HTTP {status_code}"
            
            # 2. Check for "Waiting Room" or "Block Page" titles
            elif any(indicator in title.lower() for indicator in BLOCK_INDICATORS):
                is_blocked = True
                error_reason = "Challenge/Block Page detected"
            
            # 3. Check for empty titles (Common with Kasada/DataDome blocks)
            elif not title or title.strip() == "":
                is_blocked = True
                error_reason = "Empty/N/A Title (Hidden Block)"

            if is_blocked:
                results.append({"url": url, "antibot": antibot, "title": title, "status": "fail", "error": error_reason})
                print(f"❌ [{antibot}] Blocked: {error_reason}")
            else:
                results.append({"url": url, "antibot": antibot, "title": title, "status": "ok"})
                print(f"✅ [{antibot}] Success: {title}")

        except Exception as e:
            error_msg = str(e).split('\n')[0] # Keep it brief
            print(f"❌ [{antibot}] Exception: {error_msg}")
            results.append({"url": url, "antibot": antibot, "title": None, "status": "fail", "error": error_msg})

        finally:
            page.close()

    browser.close()

# Print summary table
passed = sum(1 for r in results if r.get("status") == "ok")
failed = len(results) - passed

print("\n" + "=" * 105)
print(f"{'STATUS':<10} {'ANTIBOT':<12} {'RESULT/REASON':<30} {'URL'}")
print("=" * 105)
for r in results:
    status = "✅ PASS" if r.get("status") == "ok" else "❌ FAIL"
    antibot = r.get("antibot", "N/A")
    # Show the reason if it failed, otherwise show the page title
    display_info = r.get("error") if r.get("status") == "fail" else r.get("title", "N/A")
    print(f"{status:<10} {antibot:<12} {str(display_info)[:28]:<30} {r['url']}")
print("=" * 105)
print(f"✅ Passes: {passed}  |  ❌ Fails: {failed}  |  Total: {len(results)}")
print("=" * 105)

This snippet does the following:

• Maps a list of target URLs, each associated with their anti bot.

• Uses Playwright to launch a Chromium browser in “headless” mode.

• For each target URL, the script:

  • Opens a new browser page.

  • Attempts to navigate to the site and waits for the initial content to load.

  • Pauses for 5 seconds to see if the page successfully renders the correct content or if it gets blocked/redirected by the site’s security.

  • Attempts to extract the page title as proof of a successful load.

• Avoids the common trap of assuming a successful page load implies successful data access by checking the HTTP status code, title semantics, and structural integrity (line “N/A” titles)-

• Prints summarized results.

Perfect. The code is ready for launch.

Step #2: Launch The Tester And Retrieve Results

After launching the scraper, the results are the following:

STATUS     ANTIBOT      RESULT/REASON                  URL
==========================================================================================
❌ FAIL     Cloudflare   HTTP 403                       <https://www.ssense.com/en-us/>...
❌ FAIL     Akamai       Page.goto: net::ERR_HTTP2_PR   <https://www.gucci.com/us/en/pr/>...
❌ FAIL     DataDome     Empty/N/A Title (Hidden Bloc   <https://loccitane.com/en-us/olio->...
❌ FAIL     Kasada       HTTP 429                       <https://www.empire-cat.com/equi>...
===========================================================================================
✅ Passes: 0  |  ❌ Fails: 4  |  Total: 4
===========================================================================================

As you can see, the anti-bots successfully blocked 4 requests on 4:

• Cloudflare returned a 403 (forbidden).

• Akamai returns a Page.goto: net::ERR_HTTP2_PR error, which is often related to fingerprinting.

• DataDome reported an empty result, which suggests it may have activated a captcha.

• Kasada returns a 429 error. Generally speaking, a 429 error is a “too many requests” error, so the server blocks your scraper because you’re hitting it too fast. But specifically for Kasada, this often indicates a block due to fingerprinting.

Terrific! You made it to the end of this tutorial. Now it’s time to see what happens when implementing the Oxylabs solution.

Testing Common Anti-bots With Oxylabs Headless Browser: Step-by-step Tutorial

In this tutorial section, you will learn how to hit the same websites as in the previous paragraph using Oxylabs Headless Browser.

Prerequisites and requirements

Other than the requirements previously met, you need an active Oxylabs account in order to create your username and password for using its headless browser. Also, make sure to match all the requirements needed by the Oxylabs Headless Browser.

In the test-antibots/ folder, create a new Python file that will contain the scraping logic using the Oxylabs product:

test-antibots/
   |
   |-- tester.py
   |
   |-- tester-oxylabs.py
   |
   |__ venv/

Good. You are ready to proceed with the actual tutorial.

Step #1: Code The Scraper Using Oxylabs Headless Browser

Write the following code in tester-oxylabs.py:

import json
import random
import string
import os
from playwright.sync_api import sync_playwright

# Define params for connecting to Oxylabs servers
username = "<your-oxylabs-username>"
password = "<your-oxylabs-password>"
endpoint = "ubc.oxylabs.io"

# Create folder for screenshots
if not os.path.exists("screenshots"):
    os.makedirs("screenshots")

def get_session():
    return ''.join(random.choices(string.ascii_lowercase + string.digits, k=10))

# Define Chrome arguments
chrome_args = json.dumps([
    "--disable-blink-features=AutomationControlled", 
    "--enable-features=NetworkServiceInProcess"
])

# Define params for Oxylabs Headless Browser
params = f"p_device=mobile&p_cc=US&p_session_id={get_session()}&args={chrome_args}"
browser_url = f"wss://{username}:{password}@{endpoint}?{params}"

targets = [
    ("<https://www.empire-cat.com/equipment/used/catalog/aa/j1s03815>", "Kasada"),
    ("<https://www.ssense.com/en-us/men/designers/acne-studios>", "Cloudflare"),
    ("<https://loccitane.com/en-us/olio-jeunesse-divine-30ml-27DH030I22.html>", "DataDome"),
    ("<https://www.gucci.com/us/en/pr/women/shoes-for-women/sandals-for-women/flat-sandals-for-women/womens-slide-sandal-with-horsebit-p-834427AAE046273>", "Akamai")
]

# Define semantics to intercept in responses
BLOCK_TITLES = ["just a moment", "access denied", "attention required", "security measure", "not acceptable", "403 forbidden"]
results = []

def run_benchmark():
    with sync_playwright() as p:
        for url, antibot in targets:
            print(f"---\nTesting {antibot}...")
            try:
                browser = p.chromium.connect_over_cdp(browser_url)
                context = browser.new_context()
                page = context.new_page()

                captcha_detected = False
                def handle_log(msg):
                    nonlocal captcha_detected
                    if "oxylabs-captcha-solve-start" in msg.text:
                        print(f"   [Solver] {antibot} challenge detected...")
                        captcha_detected = True
                    elif "oxylabs-captcha-solve-end" in msg.text:
                        print(f"   [Solver] {antibot} resolved!")
                        captcha_detected = False

                page.on("console", handle_log)
                page.goto(url, wait_until="domcontentloaded", timeout=60000)
                
                # Wait for the page to stabilize
                if captcha_detected:
                    page.wait_for_timeout(15000)
                else:
                    page.wait_for_timeout(8000)

                # Make and save screenshots
                screenshot_path = f"screenshots/{antibot.lower()}_result.png"
                page.screenshot(path=screenshot_path, full_page=False)
                print(f"   [Camera] Screenshot saved: {screenshot_path}")

                title = page.title() or ""
                low_title = title.lower()
                is_blocked = any(bt in low_title for bt in BLOCK_TITLES)
                
                if len(title) > 5 and not is_blocked:
                    print(f"✅ {antibot} PASS: {title[:30]}...")
                    results.append({"url": url, "antibot": antibot, "status": "ok", "title": title})
                else:
                    print(f"❌ {antibot} FAIL: {title}")
                    results.append({"url": url, "antibot": antibot, "status": "fail", "error": title})

                browser.close()

            except Exception as e:
                print(f"❌ {antibot} Exception: {str(e)[:50]}")

run_benchmark()

Below is what changes with respect to the version in the previous paragraph:

• It defines the credentials for connecting to Oxylabs servers and creates a folder where it will later store screenshots.

• With chrome_args(), it defines some parameters for setting the automation in Chrome.

• In the string params, the p_mobile and *p_cc* respectively manage the perceived device type and IP.

• It manages CAPTCHAs with oxylabs-captcha-solve-start and oxylabs-captcha-solve-end. And since it needs to manage CAPTCHAs at each request, the code now has a timeout that helps the pages stabilize before the scraper passes on to the next target URL.

• It uses page.screenshot() to make screenshots after the server sends the request and saves them into the previously created folder. This is important because the exceptions can’t all be handled via intercepting HTTP responses or via semantics. By making a screenshot before the connection is closed, you can actually see if the scraper has passed the anti-bot or if it failed at it.

Good. You’re now ready to run the code and see the results.

Step #2: Launch The Script

After launching the scraper, the results are the following:

---
Testing Kasada...
   [Camera] Screenshot saved: screenshots/kasada_result.png
✅ Kasada PASS: 2021 CATERPILLAR 950M | Empire...
---
Testing Cloudflare...
   [Camera] Screenshot saved: screenshots/cloudflare_result.png
✅ Cloudflare PASS: Latest Acne Studios Collection...
---
Testing DataDome...
   [Camera] Screenshot saved: screenshots/datadome_result.png
✅ DataDome PASS: Immortelle Anti-Aging Face Oil...
---
Testing Akamai...
   [Camera] Screenshot saved: screenshots/akamai_result.png
✅ Akamai PASS: Slides for Women | Designer Sl...

As you can see, using the Oxylabs Headless Browser, the scraper overcomes all the anti-bots!

As a final note, consider that, based on the configuration and settings you will actually implement, DataDome (but also other anti-bots) may activate sliding captchas:

In this case, the Oxylabs Headless Browser is of no help as it only supports hcaptcha, recaptcha, and turnstile dynamic captchas.

Hooray! You successfully made it to your first scraping project using the Oxylabs Headless Browser.

Conclusion

In this article, you learned the actual distinction between headless browsers and browser automation tools. You also discovered the Oxylabs Headless Browser which, despite the name, it’s cloud-based browser automation. The difference between this and the browser automation tools you already know is that it’s cloud-based. This means that when it’s called via the code, it manages all the infrastructure on its own without overhead on your side.

As you learned in the tutorial section, it’s also pretty easy to configure and to use: you just need to manage some parameters and call the Oxylabs endpoint.

So, let us know: were you already using the Oxylabs Headless Browser?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

---

The first is the UDP we want, HTTP/3 for our requests. The second is the UDP we never asked for, the WebRTC traffic that a browser generates on its own, which can leak around the proxy and expose our real IP. But supporting UDP and actually getting a client to route its UDP through the proxy are two different problems, and the result changes a lot depending on the client. An HTTP client, a browser doing HTTP/3, and a browser doing WebRTC all behave differently, so we tested all three, end to end, to see where a UDP proxy does the work for us and where we still have to step in on the client side.

A quick recap of HTTP/2 and HTTP/3

We already covered the request side of HTTP/3 in Faster Web Scraping with HTTP/3, so let’s just recap what matters for proxies. HTTP/1.1 and HTTP/2 run on TCP. HTTP/3 drops TCP completely and runs on QUIC, which is built on UDP and merges the transport and TLS handshakes into one. The part that matters for us is the transport: a proxy built to relay TCP has nothing to grab onto in a QUIC connection, since there’s no TCP stream to relay.

Adoption is real, but it’s easy to overstate. If we look at traffic, HTTP/3 is around a fifth of all requests, while HTTP/2 is still the majority at almost half, and this split has stayed roughly the same through 2026 (Cloudflare Radar). If we look at site support instead, the number goes up: around 38% of sites advertise HTTP/3 via Alt-Svc (W3Techs). Italy, where our test machine sits, is one of the higher HTTP/3 markets at about 30%. So HTTP/3 is a serious, well-supported protocol, not some niche thing, but it hasn’t replaced HTTP/2 yet, and most real traffic is still on TCP.

HTTP/3 and QUIC run on UDP, so the proxy layer has to carry it too. anyIP gives compatible clients production-ready SOCKS5 UDP ASSOCIATE across its residential and mobile network.

Explore anyIP UDP Proxies

The sales pitch follows naturally from here. If real browsers increasingly speak HTTP/3 and our scraper can’t, that difference becomes a signal, so we’d want to speak HTTP/3 too, and to do that through a proxy, the proxy has to carry UDP.

This is exactly what some providers are now selling. They offer UDP support, usually as a beta over SOCKS5, and they pitch it as a stealth feature: the idea is that most residential traffic is QUIC these days, so a client stuck on TCP stands out. There are two parts of this claim worth checking before relying on it, the stealth side and the tooling side, so we tested both, using a provider that carries UDP and one that doesn’t.

UDP support and the stealth question

Switching to HTTP/3 doesn’t remove protocol fingerprinting, it just moves it somewhere else. Detection happens at three layers, and moving from TCP to QUIC affects each one in a different way.

The TCP/IP stack fingerprint, meaning the SYN options, window sizes and TTL that give away the OS, really does disappear, because there’s no TCP handshake anymore. That’s the only layer that goes away, though. The TLS fingerprint stays exactly as it was, because QUIC still does a TLS 1.3 handshake, just wrapped inside QUIC instead of running over TCP. The ClientHello with its cipher suites and extensions is still there, and JA4 was designed to fingerprint the QUIC variant on purpose (Scrapfly QUIC/HTTP3 fingerprint). The HTTP/2 fingerprint, the SETTINGS frames, window updates and pseudo-header order, doesn’t get erased either. It just gets replaced by an HTTP/3 fingerprint built from QUIC transport parameters.

And those QUIC transport parameters are basically the same thing we were trying to get away from. Scrapfly says it clearly: they “work like HTTP/2 SETTINGS frames but at the transport layer.” Values like max_udp_payload_size and initial_max_data, and the order they come in, are different between Chrome’s quiche, Firefox, Go’s quic-go and ngtcp2, and that difference is itself a fingerprint (Scrapfly HTTP/2 and HTTP/3 fingerprinting). QUIC even adds new signals that have no TCP equivalent at all, like 0-RTT behavior and connection migration support.

So the layer that actually gets scrapers blocked today is TLS and HTTP/2 fingerprinting, which Cloudflare, Akamai and DataDome score together as JA4 and JA4H. That’s a TCP-side problem, and it’s one we already solve with a client that imitates a browser’s handshake. HTTP/3 fingerprinting is growing but still immature. Scrapfly is pretty direct about it: it “adds signals on top of” HTTP/2 instead of replacing it, real Chrome on HTTP/2 is still accepted on the vast majority of sites, and the scraper-on-HTTP/2 versus user-on-HTTP/3 gap is only “starting to become a signal on Cloudflare’s premium tier.” There’s also a catch worth keeping in mind: a scraper that speaks HTTP/3 through a non-browser QUIC stack shows a QUIC fingerprint that matches no real browser, so it can stand out just as easily as it blends in. In other words, a UDP proxy is what gets us onto HTTP/3 in the first place, but the stealth benefit only pays off if the client presents a QUIC fingerprint that a real browser would. The proxy just carries the traffic; making it look right is still our job.

Check the TWSC YouTube Channel

The tooling has not caught up yet

The other half of the pitch is the tooling. The most common client for browserless scraping is curl_cffi, so the real question is whether it can actually do HTTP/3 in the setup a scraper normally runs in. We tested it (run_curl_cffi_http3.py), and you can find the code, as always, on our GitHub repository available for our paid readers, under the folder 109.UDP

The library is compiled with the QUIC stack, ngtcp2 and nghttp3, and on a plain direct request it does reach HTTP/3. It even reaches HTTP/3 with a full Chrome fingerprint if we use the low-level interface, so impersonation and HTTP/3 aren’t really in conflict, even if the first error makes it look that way.

high-level requests API:
  plain, no impersonation, V3ONLY        -> HTTP/3
  Chrome impersonation, V3ONLY           -> ERROR: QUIC needs at least TLS version 1.3
  Chrome impersonation, HTTP proxy, V3   -> ERROR: HTTP/3 is not supported over an HTTP proxy
  Chrome impersonation, SOCKS5, V3       -> ERROR: HTTP/3 is not supported over a SOCKS proxy
low-level Curl API (same impersonation):
  impersonate chrome + V3ONLY            -> HTTP/3

Need help with your scraping project?

That middle error in the high-level API is just a TLS-version conflict inside the requests wrapper, not a real limitation, since the low-level call combines the Chrome fingerprint with HTTP/3 and gets there fine. The lines that really matter are the proxy ones. curl refuses HTTP/3 over a proxy completely, both HTTP and SOCKS, and it refuses before it even connects, so this is curl saying no, not the proxy. It happens even with a proxy that could carry the UDP. And since a scraper always runs through a proxy, in practice curl_cffi ends up being HTTP/2 over TCP, and a UDP-capable proxy doesn’t change that at all.

So this is the honest picture on the request side. A proxy that carries UDP does its job for a client that tunnels its own UDP, but the popular HTTP clients refuse HTTP/3 over a proxy right now, so the benefit is stuck waiting for the tooling to catch up. The proxy capability is arriving before the clients that could use it, which is good to know before expecting HTTP/3 to flow through a scraper just because the proxy supports UDP.

The WebRTC leak, where UDP really bites

Read more

However, web scraping can also serve as a foundation for building the data infrastructure needed to better understand national markets and societal conditions.

In this blog post, I’ll show real-world examples of how web scraping is already used in the public sector, and how institutions could leverage it to improve policies and deliver higher-quality services to citizens.

Why Web Scraping Matters for Public Institutions

Public institutions have access to statistical datasets from official sources. These are surely relevant, but they tend to provide a picture of the market only after events have already happened, sometimes months or even a year later.

That delay creates a challenge. After all, institutions need timely information to understand citizens’ current needs and respond effectively. Another issue is that relevant data is often scattered across many sources, such as different websites, web portals, and marketplaces.

This is where web scraping can help! By collecting large volumes of publicly available data from multiple sites, institutions can complement and enrich official statistics.

Data collection is only the first step, though. The real value comes from building a complete data pipeline that includes cleaning, deduplication, aggregation, geolocation, statistical modeling, and interpretation.

At Prague Crawl 2026, Bayernheimerová Klára presented a compelling example of this approach. Her talk showed how the Czech Ministry of Finance relies on rental listing data collected by an external provider from several real estate portals and processes it through an end-to-end pipeline based on a dedicated statistical methodology.

As I’ll present shortly, the result is a reliable source of insights that supports housing policy decisions and affordable housing programs across the country, while also providing a valuable service to citizens.

What’s important to understand is that the goal isn’t simply to collect raw data, but to power services such as interactive maps, calculators, and dashboards that are useful to both public institutions and citizens.

Case Study: Czech Rental Market Intelligence System

To see an actual application of web scraping in the public sector, I’ll now present the initiative developed by the Czech Ministry of Finance for rental price analysis across Czechia.

For your scraping needs, having a reliable proxy provider like Decodo on your side improves the chances of success.

Try Decodo Now

The Motivation Behind the Project

Just like in many other European countries, housing affordability has become a major social and economic challenge in the Czech Republic. Housing supply has struggled to keep up with demand, causing both property prices and rents to rise steadily across the country.

To confront this issue, the Czech Ministry of Finance needed detailed and up-to-date information on local rental markets. The initiative was launched specifically to fill that gap, creating a web data analysis process that gives a continuously updated view of rental prices across the entire country.

The objective is to provide a reliable foundation for housing policy, affordable housing programs, and public-facing tools that help citizens better resonate with local rental markets.

Web Data Sourcing

The data for this initiative has been collected using Apify, one of the largest marketplaces of ready-made tools for web scraping, automation, and AI.

If you aren’t familiar with Apify, it provides over 41,000 ready-made serverless cloud programs (called Actors) to automate a wide range of tasks, including web scraping across thousands of different domains.

These Actors are developed and maintained by the community (and in some cases by Apify itself) and run on Apify’s cloud infrastructure. You can use them directly through Apify Console via a no-code interface, or call them programmatically via API. They can also be integrated into workflows such as n8n, Make, Zapier, or AI agents via MCP.

Two main reasons influenced the choice of Apify:

1. It’s a Czech-based company.

2. It enables the Ministry of Finance to gather large-scale, up-to-date information on rental listings without maintaining scraping infrastructure in-house.

In general, a solution like Apify lowers the barrier for public-sector teams, as these may have limited engineering capacity.

Downstream Data Pipeline

At a high level, this is the downstream data pipeline implemented by the Czech Ministry of Finance:

1. Data ingestion: Collects rental listings via Apify-based web scraping from multiple Czech real estate portals.

2. Data cleaning: Sorts data and removes duplicates, incorrect entries, and inconsistencies to ensure it is up to date, accurate, and free from deviations or distortions, following the procedure defined in Decree No. 456/2024 Coll.

3. Geolocation and aggregation: Standardizes addresses, assigns cadastral units, and enriches listings with attributes such as size, amenities, and building type.

4. Statistical modelling: Uses hedonic regression with spatial and temporal weighting to estimate underlying rental price levels.

5. Output generation: Presents results as interactive price maps and calculators.

Note: The pipeline also includes data update and recycling via a rolling window approach. This process adds new listings while retiring outdated observations to keep the dataset current.

To make rental prices comparable across different locations, the methodology is based on a reference apartment. That represents a standardized apartment profile with predefined characteristics, such as size, type, furnishing level, etc.

This approach reduces differences caused by individual property features and enables more consistent comparisons of rental price levels across municipalities and cadastral areas.

Produced Output and Tools

The Czech Rental Housing Price Map consists of two main solutions:

• Interactive rent price map: Provides estimated rental prices at the level of municipalities and cadastral areas across the Czech Republic.

• Market rent calculator: A practical tool that calculates a statistical estimate of rental price levels based on a standardized apartment profile and selected property characteristics.

Important: The two solutions aren’t intended to determine the market rent of a specific apartment. Instead, they return statistical estimates based on a standardized reference apartment. So, they’re intended primarily for market monitoring, regional comparisons, and housing policy analysis.

Interactive rent price map

The interactive rent price map shows minimum, maximum, and median rents for each area based on market listings, adjusted per square meter for a standard unfurnished reference apartment. It covers four size categories from 1+kk/1+1 to 4+kk/4+1.

Note that the map is interactive, and you can zoom in and out to explore individual cadastral areas. It’s also updated four times per year to ensure current data.

Market rent calculator

The market rent calculator is available as a form on the Czech government website. It works as follows:

1. Select the territorial unit where you want to estimate the rent.

2. Choose the size category of the apartment.

3. Enter the floor area of the apartment (if you don’t know it, the calculator will automatically use a default value for the selected location).

4. Indicate whether the apartment is in a new building.

5. Optionally specify whether the building uses non-standard construction materials (e.g., other than brick or panel).

6. Optionally select additional features such as a terrace, furniture, or an assigned parking/garage space.

7. Click “Calculate rent” to obtain an estimate of the monthly market rent (CZK) and the corresponding price level, which represents the final estimated rent for the selected apartment profile.

Impact

The project provides a systematic and regularly updated overview of market rental prices across the entire Czech Republic, supporting consistent monitoring of housing market developments over time and across regions.

Beyond market observation, the outputs serve as a key evidence base for housing policy design and housing programmes within public administration. For example, the State Investment Support Fund (SFPI) relies on these results when working with its affordable rental housing schemes.

Web Scraping in the Public Sector: Extending the Model

The project carried out by the Czech Ministry of Finance is just one example of a much broader pattern: how web scraping can be used as a foundation for modern public-sector data systems.

In particular, the same pipeline logic can be applied across many other domains. For example, in labor markets, web scraping can be used to collect job postings, salary ranges, and skill requirements from recruitment platforms. This can help governments identify regional skill shortages and design more targeted education or reskilling programmes.

Similarly, in consumer price monitoring, scraping can track grocery prices, housing costs, and essential goods across regions, enabling better inflation tracking and cost-of-living analysis. Other potential public domains that can benefit from this include energy consumption and transport accessibility.

When combined, those datasets become even more powerful. For instance, housing data, job market data, and grocery price data could be combined into a broader “liveability” index. That would help citizens and policymakers assess the overall affordability of living in a given region.

Check the TWSC YouTube Channel

Building a Job Market Choropleth for Czech Districts

In this section, I’ll guide you through building a job market choropleth across Czech districts. The idea is to show how web scraping, combined with a complete data pipeline, can be utilized for other scenarios beyond the previous example.

Important: The project below isn’t related to the initiative from the Czech Ministry of Finance and has been created purely for illustrative purposes.

Prerequisites

To follow this tutorial section, make sure you have:

• An Apify account (a free plan is sufficient).

• Python 3.11+ installed locally.

I’ll also assume you have a Python project set up locally with the following libraries installed:

pip install pandas geopandas geopy matplotlib

These are the required dependencies for this project and will be used as follows:

• pandas: Load, clean, and manipulate the scraped job posting data before geospatial analysis.

• geopandas: Handle geographic data, perform spatial joins, and create district-level maps.

• geopy: Convert job location addresses into geographic coordinates through geocoding.

• matplotlib: Visualize the results by generating a choropleth map showing job density by district.

Step #1: Access the Indeed Jobs Scraper Apify Actor

Just like in the Czech Ministry of Finance example, you can use Apify to collect the source data. This saves you from building and maintaining a complete job scraping pipeline from scratch.

For this project, we’ll use Indeed as the data source. Indeed has a Czech version, and most listings include the full office address where employees are expected to work.

In detail, I recommend using the Indeed Jobs Scraper [PPR] Actor. This automates the extraction of job titles, salaries, locations, company information, and job descriptions from Indeed.

To get started, log in to your Apify account and select the “Apify Store” option from the left-hand navigation menu. Then, search for “Indeed Jobs Scraper [PPR]”:

Click on the Actor card to reach its page. Great!

Step #2: Run the Job Scraping Task

On the Actor page, you’ll find an input form that lets you configure the scraper before running it in the cloud. Select “Czech Republic” as the “Country” and enter “Software engineer” in the “Query” field:

Note: In a real-world scenario, configure the Actor’s inputs according to your needs.

Next, toggle the option to avoid duplicates so that the Actor takes care of data deduplication for you:

Click the “Save & Start” button to launch the scraping task. The Actor will start running directly in Apify Console. As the scraper progresses, you’ll see the extracted job postings appear in real time. Be patient, as the process may take a few minutes depending on the number of matching listings.

Step #3: Explore the Output and Export the Results

Once the run completes, you’ll be able to explore the scraped dataset directly in Apify Console. As you can see, the Actor returns structured Indeed job data:

In this case, the Actor retrieved 63 job postings. That may seem like a small number, but keep in mind that the scraper targets only jobs published during the last 14 days. This helps ensure that the dataset reflects the current state of the job market.

Next, switch to the JSON view and select “All fields”. You’ll notice that each job posting includes a location object containing address information for the position:

That location data is exactly what you need to visualize job openings on a map and analyze their geographic distribution across Czech districts.

Finally, open the “Storage” tab, select the “JSON” export format, and click “Download” to export the scraped dataset:

A file with a name similar to dataset_indeed-scraper_2026-06-18_11-54-00-555.json will be downloaded. Rename it to jobs.json and place it in your Python project’s root directory.

Step #4: Download the Required Czech GeoJSON Data

To visualize the distribution of job openings across Czech districts, you’ll need a GeoJSON dataset containing the geographic boundaries of those districts.

Generally, that type of data is open and publicly available. For example, one possible source is the siwekm/czech-geojson repository, which provides GeoJSON files for various Czech administrative divisions:

Download the okresy.json file, which contains the geographic boundaries of Czech districts (okresy in Czech).

Note: Although the okresy administrative system was officially abolished in 2003, these districts are still used for statistical analysis. This makes them a good choice for visualizing the distribution of job opportunities across the country.

Once downloaded, add the okresy.json file to your Python project. At this point, your project structure should look similar to this:

├── jobs.json
├── okresy.json
└── main.py

Step #5: Visualize the Job Data on a Map

In your Python file, define the complete pipeline with the following code:

# pip install pandas geopandas geopy matplotlib

import json
from geopy.geocoders import Nominatim
from geopy.extra.rate_limiter import RateLimiter
import pandas as pd
import geopandas as gpd
import matplotlib.pyplot as plt

# Load the scraped Indeed job openings
with open("jobs.json", "r", encoding="utf-8") as f:
    jobs = json.load(f)

# Load the list of addresses from the job postings
addresses = []
for job in jobs:
    loc = job.get("location", {})

    # Prefer fullAddress, fallback to formattedAddressLong or formattedAddressShort
    address = (
        loc.get("fullAddress")
        or loc.get("formattedAddressLong")
        or loc.get("formattedAddressShort")
    )

    # Filter out full-remote jobs with no address
    if address and address.strip().lower() not in ["home office", "remote"]:
        addresses.append(address)

addresses = list(addresses)

# Initialize geocoder for Czech job locations
geolocator = Nominatim(user_agent="cz_job_map")
geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1)

# Geocode all job addresses into latitude/longitude points
rows = []
for address in addresses:
    loc = geocode(f"{address}, Czech Republic")

    # Keep only successfully geocoded results
    if loc:
        rows.append({
            "address": address,
            "lat": loc.latitude,
            "lon": loc.longitude
        })

# Convert geocoded results into a DataFrame
df = pd.DataFrame(rows)

# Convert DataFrame into GeoDataFrame with point geometries
points = gpd.GeoDataFrame(
    df,
    geometry=gpd.points_from_xy(df.lon, df.lat),
    crs="EPSG:4326"
)

# Load Czech administrative districts (okresy) from a local GeoJSON file
okresy = gpd.read_file("okresy.json")

# Align coordinate reference systems between datasets
if okresy.crs != points.crs:
    okresy = okresy.to_crs(points.crs)

# Assign each job point to its corresponding district
joined = gpd.sjoin(
    points,
    okresy,
    predicate="within",
    how="left"
)

# Count number of jobs per district
counts = (
    joined["id"]
    .value_counts()
    .reset_index()
)

counts.columns = ["id", "count"]


# Merge job counts back into district geometries
okresy = okresy.merge(
    counts,
    on="id",
    how="left"
)

# Replace missing values with 0 (districts with no jobs)
okresy["count"] = okresy["count"].fillna(0)

# Plot choropleth map with job counts per district
fig, ax = plt.subplots(figsize=(10, 8))

okresy.plot(
    column="count",
    cmap="Reds",
    edgecolor="black",
    linewidth=0.3,
    legend=True,
    vmin=0,
    ax=ax
)

# Add labels for districts with job activity
top = okresy[okresy["count"] > 0].sort_values("count", ascending=False)

for idx, row in top.iterrows():
    centroid = row.geometry.centroid

    ax.text(
        centroid.x,
        centroid.y,
        str(int(row["count"])),
        fontsize=9,
        ha="center",
        va="center",
        color=(lambda v: "white" if v > okresy["count"].max() * 0.5 else "black")(row["count"]),
        fontweight="bold"
    )

ax.set_title(
    "Software engineering job density on Indeed in the Czech Republic (last 14 days, by district)"
)
ax.axis("off")

plt.show() 

Need help with your scraping project?

That’s what the above script does:

1. Loads scraped job data from jobs.json and extracts valid physical addresses, filtering out remote or missing location entries.

2. Initializes the Nominatim geocoder with a rate limiter to safely convert addresses into latitude and longitude without exceeding request limits.

3. Geocodes each address into coordinates and keeps only successful results, storing them in a structured pandas DataFrame.

4. Converts the DataFrame into a GeoDataFrame with point geometries so the job data can be used in spatial analysis workflows.

5. Loads Czech district boundaries from the okresy.json file and ensures both datasets use the same coordinate reference system.

6. Performs a spatial join to assign each job point to a district, then aggregates job counts per district and merges results back into the map.

7. Visualizes the final dataset as a choropleth map, coloring districts by job density and adding readable labels for high-activity areas.

Step #6: Run the Script

Execute the script, and you’ll get a result like this:

Notice how most fresh software engineering openings are concentrated in districts surrounding Prague, Brno, and Ostrava. These are the three largest cities in the Czech Republic, so the result clearly makes sense!

Now, this was just a simple example, but you can use the same approach to build an interactive map or add additional features, such as calculating the average salary for each position by district, and more advanced analytics.

Value and Playbook for Other Governments

The approach presented in this blog post is highly adaptable to any public dataset with a spatial dimension, where location plays a key role in understanding patterns and inequalities. Still, a reusable country-independent playbook emerges:

1. Collect data via web scraping (or other administrative, public sources).

2. Clean and validate the data.

3. Enrich and aggregate it with contextual and geographic features.

4. Apply statistical or analytical models.

5. Present the results through maps, dashboards, or other interactive tools.

As you can tell, this pipeline isn’t domain-specific and can be reused across many different policy areas. Finally, it’s crucial that the adopted methodology remains open and clearly explained to the public. This fosters transparency, reproducibility, and trust in data-driven decision-making.

Conclusion

Here, I’ve shown how the Czech Ministry of Finance uses a web scraping service as part of a full data pipeline to turn rental listings into actionable insights for housing policy and public tools.

By accessing data collected from multiple real estate portals and processing it through cleaning, geolocation, and statistical modelling, they transform raw web data into interactive maps and rent calculators that support decision-making.

In this article, I built a similar pipeline for a different use case: a job market choropleth across Czech districts. I started from scraped Indeed listings, geocoded job locations, mapped them to districts, and visualized the results on a map.

I hope this example was useful and inspiring. If you have questions or ideas, feel free to share them in the comments below.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

You can read that episode two ways. Either the guardrails were the only thing standing between a public model and a stream of attack-ready output, or the shutdown was a tidier story than the truth. We are not going to settle that here. For our corner of the world, the lesson is simpler. A capable model, stripped of guardrails or jailbroken around them, automates the expensive part of an attacker’s job. Deobfuscation, code analysis, and figuring out how an unfamiliar system behaves under pressure. Those tasks have always gated the scraper-versus-vendor arms race because they cost human reverse-engineering hours, and that gate is now moving.

The cat-and-mouse game we cover every week runs on a hidden economy. A vendor ships an obfuscated sensor. Someone on the scraping side spends days or weeks reading minified JavaScript, renaming variables, and rebuilding the logic until the parameters make sense. The cost of that labor is what keeps most protections standing. If a model can do a real share of it on its own, the math behind the whole arms race changes.

Manus AI, and why not Claude

Manus is not a chat assistant. It is an agent that pairs a language model with a sandboxed virtual machine, and inside that machine, it runs code, fetches pages, beautifies scripts, and iterates on its own output. Meta acquired it at the end of December 2025 for around two billion dollars, though by June 2026, Chinese regulators were already forcing parts of that deal to unwind. What that architecture buys is verification. Manus not only describes what a script might do. It runs the script and checks.

That difference shows up the moment you set guardrails next to it. Ask Claude or ChatGPT to deobfuscate a live anti-bot sensor and explain which signals feed its scoring, and you tend to hit a wall. Those products keep tight cybersecurity guardrails, and the Fable shutdown is the clearest sign yet of how hard that line is enforced. We handed the same class of task to Manus, and it got to work. We have looked at agentic browsers and AI scraping assistants before, but those were aids for a human operator who still did the thinking. Here, the model does the reverse engineering.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Mapping Akamai on Net-a-Porter

We have written about Akamai more than any other anti-bot system. For the bypass-oriented background, THE LAB #30, THE LAB #85, and THE LAB #100 cover the recovery-when-nothing-works case, proxy chaining, and the browser-login-plus-HTTP hybrid. We will not repeat any of that.

What we wanted this time was not a bypass. It was a map. Akamai Bot Manager protects net-a-porter.com, and it defends in stacked layers that each fail differently:

• A TLS fingerprint check at the edge, evaluated before a single line of the client JavaScript runs.

• A sec-cpt cryptographic challenge, whose state lives in the bm_sc cookie.

• A behavioral sensor that serializes a binary telemetry payload and posts it to a collect endpoint, validating the _abck cookie.

• A client-side anti-hook layer that checks the browser environment for tampering and reports integrity separately.

The model we asked Manus to build covers the client sensor and the challenge flow, plus whatever it could observe about the TLS gate from the outside. A working end-to-end bypass was out of scope. We are not shipping code with this article, and we did not ask Manus to produce a runnable solver, even though our tests suggest we are really close to a working solution. What we’ll share here is a summary of the analysis Manus did; in the code repository, you will find the full details.

Check the TWSC YouTube Channel

What we handed the agent

The task was deliberately cold. We did not give Manus our own Akamai notes, our wiki, or any of the prior Lab articles. We pointed it at net-a-porter.com and asked it to study how Akamai's protection works, deobfuscate the client script, and explain which parameters feed into the sensor. Nothing more.

We chose this target for three reasons. It runs a real, current Akamai deployment on a high-value luxury store. The sensor was observable live, so the agent could fetch and instrument the actual script rather than reason about a stale sample. And we already understand Akamai well enough internally to judge whether the output was accurate or confident nonsense.

The reason the task went to Manus rather than a chat assistant is the guardrail gap from the previous section. Deobfuscating a production anti-bot sensor and enumerating the signals it scores is exactly the kind of request the heavily guarded models decline. Manus took it, ran the analysis in its VM, and came back with the two documents below.

What came back

The output was more detailed than we expected from a single task run. We are sharing it in two parts and deliberately not sharing it all.

The two documents below are Manus AI’s output. We reproduce them as reportage, to show what an agent derived on its own. We have not independently re-run every claim, re-derived every value, or confirmed that each detail still holds against the live site. Treat the specifics as the agent’s findings, not as TWSC-verified ground truth. The sensor version Manus reports, 746702557, was compiled on 2025-05-09, so some details may already have rotated. Where the agent surfaced operational specifics that amount to a turnkey recipe (exact module filenames, the full endpoint table, the CSS selector ids, the challenge request names), we have redacted them.

This is a schema of how Akamai Bot Protection according to Manus AI

Part one, reproduced in full, the anti-hooking and TLS summary

This first document is the safe exhibit. It explains how Akamai detects a tampered browser and how the TLS gate at the edge decides who even gets a challenge. We reproduce it verbatim.

<iframe height="0" width="0" style="display: none; visibility: hidden;" sc="true" tabindex="-1"></iframe>

Part two, reproduced with redactions, the deep technical analysis

The second document goes much further. It walks the bootstrapping sequence, the module system, the fingerprinting surface, and the telemetry transport. This is the part that starts to read like a blueprint, so we keep the architecture and the reasoning and we strip the operational specifics. Redaction markers below are ours, not Manus’s.

If you want to see the full analysis, you can find it in our GitHub repository reserved for paying users, inside the folder 108.AKAMAI-MANUS.

Read more

IP sourcing in the proxy industry

Two articles written by Spur Intelligence caught my attention in the past week.
It started with a webinar I’ve attended, where they explained what ISP proxies are and how proxy companies get those IPs.

The analysis is great, and I’ve understood better how the proxy industry works.
They also mentioned how NetNut gets high-quality ISP IP addresses, and described that in detail in this article.

How Proxy Providers Co-opt Entire Networks

Spur reverse-engineers how Netnut sources its ISP proxies through a partner called DiviNetworks, which installs GRE tunnels and policy-based routing directly on the border routers of real ISPs. The result is genuine ISP IP space sold as proxy inventory, with actual subscribers still living on the same addresses. The piece is worth reading for the detection detail: outbound connections sit in the TCP source port range 40,000 to 40,200, partner routers expose an rtr-<isp>.divinetworks.com passive-DNS pattern, and the Howard University case shows an entire /16 (AS919, around 17,000 exit IPs) co-opted at the network edge. They even quote DiviNetworks’ own figure of $13,208 a month for a US /16.
Full article here

Smart TV Apps and Residential Proxy SDKs

A more recent article, instead, highlighted how proxy companies get residential proxies. No surprises, at least on my side, in reading that Smart TVs are used as exit nodes for proxies. In fact, just like with mobile apps, developers can use SDKs from proxy providers to monetize their app installations. In some cases, it’s quite clear from the consent screen what’s happening under the hood.

All the industries that rely on web-scraped data (AI in primis, but not only) need residential proxies, so proxy providers, who are the gatekeepers for these tools, should be very careful about onboarding users with legitimate use cases and keeping fraudsters away. In my career, I’ve worked with almost every big name out there, and I’ve found this is the case. As scraping professionals, we should always remember that we’re guests (sometimes unwanted) on both the target website and the proxy infrastructure, so we should be as respectful as possible and not be driven by greed to collect data.
Full article here

Scraping Infrastructure

A great article by the engineering team at Browser Use shows how they’re spawning browsers cheaply and quickly.

How We Made Cloud Browsers 3x Cheaper and Faster

Browser Use walks through rebuilding its cloud so every session is its own Firecracker microVM, and the interesting twist is that they run Firecracker on plain EC2 instead of bare metal, accepting nested virtualization to get faster scale-up and lower cost.

The numbers are great: $0.02 per browser hour down from $0.06, VM cold start under 400ms, and create latency of 825ms at p50 across a 10,000-session test.

The engineering details that are making the difference are multiple, from mapping memory in 2MB pages plus a userfaultfd handler that preloads hot pages (resume-to-ready drops from 9.8s to 3.1s, roughly 91x fewer page-fault stalls) to two-phase vCPU pinning that took a 1,000-browser launch from 17% failed sessions to zero. They also make the case for running fully headless rather than headful, since their low-level Chromium fork pushes block-avoidance to 81% on their own benchmark.

Full article here

Read and share more of this

Want to flag and discover more articles like the ones above? Scraping News (still in beta) is where the community surfaces them. Sign up and submit what you find worth reading.

Want a concept map of everything covered here, cross-referenced with our own work and other sources worth your time?

The Web Scraping Club Wiki lives on GitHub, and as an interactive site. It is plain Markdown, so you can also clone it into your own Obsidian vault and read it locally.