In this post, I’ll explain what the tool is, how it works, and which features it offers. I’ll also show you how to use it, evaluate its performance against anti-bot detection systems, and compare it with other similar libraries.

What Is invisible_playwright?

invisible_playwright is a Python library created by Federico Elia. It patches Firefox at the C++ source level to help bypass bot detection systems.

The project is open source and free to use, and is designed to help avoid anti-bot systems and CAPTCHA challenges such as reCAPTCHA v3, Cloudflare Turnstile, DataDome, Kasada, Akamai Bot Manager, PerimeterX, hCaptcha, and Imperva.

The library addresses two questions that anti-bot systems typically ask:

1. Is this a real browser? Yes, as the tool employs Firefox patched at the native level.

2. Is a real person using it? It simulates human-like interactions to make automated activity look more natural.

At its core, invisible_playwright works as a drop-in replacement for Playwright. The core difference is that it combines a modified browser engine, configurable fingerprints, and built-in human-like interactions. As of this writing, the project boasts more than 2k GitHub stars!

Further reading:

• Official documentation

• Official wiki

• Official guides

• invisible_playwright TWSC wiki page

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

Main Features

Now that you know what invisible_playwright is, let me introduce what it brings to the table.

Drop-In Playwright Replacement

invisible_playwright is 100% compatible with Playwright. That means you don’t need to rewrite your code. Both sync and async work, and every method remains the same. So, there’s no learning curve.

You can switch from standard Playwright like this:

That’s the only change needed. Now, each Playwright browser automation session will automatically get a distinct fingerprint. Note that the browser fingerprint generated includes GPU, audio, fonts, screen, and roughly 200 other fields.

Human-like movement and interaction

invisible_playwright simulates more natural mouse interactions. As opposed to instantly moving the pointer to a target, the cursor travels toward the target along a human-like path. This behavior is controlled through the humanize option, which applies to actions such as clicking, hovering, and dragging.

Enabled by default (humanize=True), you can disable it by setting it to False:

with InvisiblePlaywright(humanize=True) as browser:
    page = browser.new_page()
    # Browser automation logic....

Fingerprint Seeds for Reproducibility

To make runs reproducible, you can pin a specific fingerprint. This helps with debugging, as if a scrape fails with a specific seed, you can replay it exactly.

You can use this feature like this:

# Initialize the InvisiblePlaywright engine
sf = InvisiblePlaywright()

# Create a new Playwright-like browser instance
with sf as browser:
    # Access the randomly assigned fingerprint for each run.
    print("seed =", sf.seed)

    page = browser.new_page()
    # Browser automation logic....

This will produce an output like this:

Then, you can reuse the same fingerprint with:

# Same fingerprint every time
with InvisiblePlaywright(seed=1561645783) as browser:
    page = browser.new_page()
    # Browser automation logic....

The seed-based fingerprint generation adopted by invisible_playwright is deterministic. In other words, you are not rolling the dice on every run. Instead, the fingerprint is derived from a known seed value, making the same fingerprint reproducible across runs. This makes bugs easier to reproduce and debug, which is particularly useful when scraping at scale.

Proxy Support

invisible_playwright natively supports SOCKS5, SOCKS4, HTTP, and HTTPS proxies. You can configure a proxy as follows:

proxy = {
    "server": "http://your-proxy-domain.com:1234",
    "username": "my_wonderful_username",
    "password": "my_very_secret_password",
}
with InvisiblePlaywright(proxy=proxy) as browser:
    page = browser.new_page()
    page.goto("https://example.com")

DNS traffic is routed through the proxy by default, helping prevent local DNS leaks. The timezone is automatically derived from the proxy’s exit IP, keeping the browser configuration consistent with the proxy location.

If needed, you can override it by specifying a custom timezone:

with InvisiblePlaywright(proxy=proxy, timezone="Europe/Rome") as browser:
    #  Browser automation logic....

Reliable access is the foundation of any scraping project. IPRoyal offers 64M+ residential IPs to help you reach any website without blocks or bans.

Visit IPRoyal

Custom Fingerprint Fields

invisible_playwright lets you pin specific fingerprint fields while keeping the remaining fields derived from the seed. You can do that through the pin parameter:

with InvisiblePlaywright(
    seed=1561645783,
    pin={
        "gpu.renderer": "ANGLE (AMD, AMD Radeon RX 7900 XTX Direct3D11)",
        "gpu.vendor": "Google Inc. (AMD)",
        "screen.width": 1920,
        "screen.height": 1080,
        "hardware.concurrency": 8,
    },
) as browser:
    page = browser.new_page()
    # Browser automation logic...

In this example, the GPU, screen resolution, and CPU core count remain fixed, while other fingerprint fields continue to be generated from the seed. That’s useful when you want to target a specific device configuration without manually defining the entire fingerprint.

CLI Usage

The PyIP invisible-playwright package installs an invisible-playwright CLI command. The main supported commands are:

• invisible-playwright fetch: Downloads the patched browser engine if it’s not already installed. The download is approximately 240–265 MB, depending on your platform. The engine is cached locally after the first download, and the command also verifies its SHA-256 hash.

• invisible-playwright version: Displays version information for the patched browser engine, the invisible_playwright library, and other information.

MCP Server

invisible_playwright can also run as an MCP server, giving Claude Code, Claude Desktop, Cursor, Codex, and all other MCP-compatible clients access to a real stealth Firefox browser.

The exposed MCP server supports navigation, page reading, clicking, typing, screenshots, and JavaScript-based data extraction. You can register it in Claude Code with:

claude mcp add stealth -- uvx aihawk

The server is bundled via AIHawk, which also provides a UI and OpenRouter support for users who prefer not to work directly from the command line.

Multiple Integrations

invisible_playwright works with a long list of Python frameworks:

• Scrapy, via the scrapy-playwright middleware.

• Crawlee for Python, with a plugin subclass or single-line setup.

• Playwright MCP, with two flags on the Microsoft server.

• CodeceptJS, through the firefox block configuration.

• Robot Framework Browser, with named keyword arguments.

• And more…

How invisible_playwright Avoids Anti-Bot Triggering

Anti-bot systems generally rely on two main signals to detect automated browsers:

1. Browser identity, via fingerprinting signals.

2. Interaction behavior, such as whether mouse movements and clicks resemble those of a real user.

invisible_playwright addresses both areas by patching Firefox at the source level and supporting human-like interactions. Discover how!

Browser Identity and C++ Level Patching

Vanilla Playwright relies on the host machine’s native browser fingerprint and the default configuration of a fresh browser session. Signals like GPU, installed fonts, screen resolution, canvas rendering, and other browser properties can make automated sessions easier to identify.

invisible_playwright addresses this by patching Firefox at the source-code level. Instead of injecting spoofed values through page-level JavaScript, it sets fingerprint data inside the browser engine before it becomes accessible to scripts.

Rather than simply spoofing navigator.webdriver or injecting modified canvas data, the library configures several fingerprint surfaces, including:

• GPU and WebGL values that are coherent and matching.

• Canvas and WebGL rendering that’s deterministic per seed.

• Fonts bundled with the browser for consistency across platforms.

• Audio context data from the profile instead of the host device.

• WebRTC that offers a synthetic ICE candidate.

• Timezone and DNS routed through the proxy.

• Input events with proper isTrusted values.

These properties are derived from a single seed, helping keep different fingerprint surfaces consistent with one another. Anti-bot systems can often identify spoofed environments when individual signals contradict each other, such as a browser reporting Windows-related properties alongside macOS-specific characteristics.

Human-Like Input and Event Fields

Most humanization libraries rely on Bézier curves to generate smooth mouse movements from point A to point B instead of instantly teleporting the cursor. That’s an improvement, but it may not be enough on its own. After all, a perfectly smooth curve with uniform timing can still look artificial.

invisible_playwright goes further by generating Bézier-curve mouse paths from the selected seed. This makes movement patterns vary between browser identities while remaining reproducible for debugging.

It also patches the browser’s native input handling so pointer events contain more realistic values, including:

• Variable timing: Movement timing adapts to the device’s refresh rate instead of following a uniform schedule.

• Appropriate pressure: Pressure is set to 0 while hovering and to a non-zero value when clicking.

• Correct pointer type: pointerType is consistently set to mouse.

• Accurate movement deltas: movementX and movementY reflect the actual pointer displacement.

• Sub-pixel coordinates: Pointer positions can account for fractional coordinates on scaled displays.

These properties are determined by the browser’s native input-handling code, rather than being added or modified by the automation driver. Such a level of control requires patching the browser binary.

Check the TWSC YouTube Channel

invisible_playwright Against Popular Bot Detection Tests

In this section, I’ll show you how to set up invisible_playwright and use it to score its performance against well-known bot detection tests:

• AntCpt.com’s Score Detector for reCAPTCHA v3

• APIVoid’s Bot Detection Test

• CleanTalk’s Am I a Bot?

• Pixelscan’s Bot Detection Test

Prerequisites: Project Setup and Script Overview

To use invisible_playwright, you need:

• Python 3.11 or newer.

• Windows x86_64 or Linux x86_64/ARM64.

In your Python project, install invisible_playwright with:

pip install invisible-playwright

Then, download the patched browser engine using the CLI:

invisible-playwright fetch

You can now use invisible_playwright much like standard Playwright. For example, the following script opens a bot detection test page, waits 10 seconds for the test to complete, and takes a screenshot of the result:

from invisible_playwright import InvisiblePlaywright

# Create a new InvisiblePlaywright instance
with InvisiblePlaywright() as browser:
    page = browser.new_page()

    # Visit the bot detection test and wait for 10 seconds
    page.goto("<BOT_DETECTION_PAGE_TEST_URL>")
    page.wait_for_timeout(10000)

    # Take a screenshot of the entire page
    page.screenshot(path="antibot-challenge.png")

Vs Fingerprint

The result was:

Note the “Confidence Score” of 1 from Fingerprint, one of the most complete and popular device intelligence and visitor identification platforms.

Vs AntCpt.com’s Score Detector for reCAPTCHA v3

The result was:

A score of 0.9 is excellent and matches the score I achieved in my regular browser session.

Vs APIVoid’s Bot Detection Test

The result was:

The detected risk score was 0, indicating a very low likelihood that the session came from a bot.

Vs CleanTalk’s Am I a Bot?

The result was:

The test returned a 100/100 score with a clear “Looks human” result.

Vs Pixelscan’s Bot Detection Test

The result was:

The “You’re finally a Human” message is a strong indication that the tool successfully passed the test.

Need help with your scraping project?

Anti-Bot Performance Benchmarks

To evaluate invisible_playwright, I ran a simple script against a chosen website protected by each of the major anti-bot systems. The results are summarized in the following table:

Note: All tests were performed locally using my ISP-provided residential IP address.

In this limited experiment, invisible_playwright successfully loaded all five test sites, achieving a 100% success rate. By comparison, vanilla Playwright consistently failed in headless mode and also struggled with Kasada in headful mode.

Comparisons with Other Stealth Browser Automation Tools

This is how invisible_playwright compares with other anti-detect browser automation tools:

Each approach involves trade-offs. Firefox has a smaller market share than Chromium, which may itself be a detection signal. At the same time, it’s currently the only browser engine that can be patched at the native binary level.

Chromium-based tools typically patch the browser driver or inject JavaScript into the page, leaving more detectable surface area exposed.

playwright-stealth is lightweight and works with multiple browsers, but it relies on injected JavaScript, which anti-bot systems may detect. Camoufox uses a fingerprint database in place of seed-derived values, so it doesn’t provide the same reproducibility across runs.

invisible_playwright, by contrast, focuses on reproducible fingerprints and deep, native-level browser patching.

Final Comment

Let me get straight to the point: invisible_playwright looks like a very solid choice for web scraping projects where bot detection is a concern. The library not only performed well in my tests and delivered on its promises, but is also exceptionally well documented, with a long list of guides, examples, and low-level explanations of how it works.

There is one potential downside, though. Because of that, and the fact that the project is fully open source, anti-bot providers could potentially reverse-engineer its implementation as it gains adoption within the community.

That’s probably my biggest concern with the library, along with its reliance on a single main maintainer. However, both are challenges that apply to many open-source stealth browser automation tools.

I also want to add a personal note and give a shout-out to Federico Elia, the author of the library. In the era of AI, where almost anyone can vibe-code a product or library in a matter of hours, or even minutes, it’s genuinely refreshing to see this level of robustness and documentation. In my experience, that combination is still far from common. Well done!

The biggest advantages of invisible_playwright are definitely its full Playwright compatibility and straightforward setup. The seed-based fingerprinting is another particularly interesting feature that not all anti-detect browser automation libraries offer.

Keep in mind that the tool requires Firefox and Python. If you need Chromium or Node.js support, you’ll have to look elsewhere. Otherwise, I strongly recommend giving this project a try and seeing how it performs for yourself!

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Most authenticated scrapers I’ve seen don’t fail at login. They fail when one piece of state stops matching everything else, even while the login still seems valid.

That causes problems if you’re scraping account-gated dashboards, portals, or other authenticated applications over long periods. Saving a cookie can work well enough for a short-lived job. But if you run the same job for weeks, the login “state” becomes something you have to manage as well.

In my experience, you really have two options: save authenticated state and replay it into new browsers, or keep the same browser session alive. Both still end in re-authentication eventually.

So, what do you need to preserve, and what can safely be rebuilt?

What does being logged in actually look like?

It’s tempting to think of an authenticated session as one cookie, even though sometimes it is just that.

But more often, the browser ends up carrying authentication-related “signals” in several other places as well. You might have an HttpOnly session cookie, a token in localStorage, another value in IndexedDB, and a temporary token associated with an OAuth flow someplace else.

The issue is that all these technologies don’t share the same eviction timeframe, and sometimes those timeframes don’t align.

• A cookie could still exist locally after the server has invalidated the session behind it.

• An access token may expire while a refresh token remains usable.

• A refresh token may be rotated, meaning the old value stops working as soon as the new one is issued.

• There’s another interim state that has changed: maybe an IP address or even geolocation changes.

Some applications also make risk decisions using signals outside the state you saved, such as network location or characteristics of the browser making the request. This is the part that trips people up most: a person logs in by hand, and then a script picks up where they left off. Anti-bot detection loops run passively all the time, and if something looks suspicious, they’re quick to revoke access.

I’ve stopped thinking of it as logged in or logged out. It’s closer to three states: valid, stale, and recovering. The middle one is where I’ve lost the most time.

Why a saved storageState snapshot always breaks eventually

For most scraping jobs, replaying auth state into a fresh browser beats logging in on every run. Here’s what that looks like with Playwright:

const context = await browser.newContext({
  storageState: “auth.json”,
});

const page = await context.newPage();
await page.goto(PROTECTED_URL);

You authenticate once, persist the relevant state, then use it to bootstrap later browser contexts. Re-applying prior auth is much faster than repeating the login flow, for obvious reasons, and puts less load on the target. This means you don’t have to automate passwords or manage multi-factor authentication unnecessarily.

But there are a few gotchas, the biggest being: the state file only contains IndexedDB if you asked for it when you saved it.

await context.storageState({ path: “auth.json”, indexedDB: true });

The option was introduced in Playwright v1.51. On older versions, apps that keep tokens in IndexedDB (like Firebase Authentication) will silently lose them.

I think of auth.json as a photo of your login on the day you took it, not the login itself.

Imagine the scraper starts on Monday with a working token. On Wednesday, the application rotates that token during normal use. Your live browser now contains the new token, but the state file you created on Monday doesn’t.

Restart the worker on Thursday, and you restore Monday’s version of reality. You can start to see how this problem plays out, and why its nondeterministic state can cause a lot of confusion. To make matters worse, this “invalidation” can happen out of nowhere. If you’re paying per run, that isn’t just downtime. It’s money spent on scrapes that bring back a login page.

Cookies run into the same kind of trouble. Their local expiry tells you when the browser considers a cookie expired. It doesn’t tell you whether the server still accepts the session behind that cookie. In a perfect world, they’d always be in sync, but the web is anything but perfect!

Temporary state is another failure point. sessionStorage, for example, is deliberately scoped to a browser tab. Turn every piece of temporary browser state into something permanently replayable, and you can make authentication less reliable, particularly around redirects and CSRF protection.

My rule: assume any saved auth state is already going stale. Authenticated scraping is fundamentally a high-fault workflow, so take great care to ensure the system behaves as expected in production.

Why not just keep the browser running?

If replaying a snapshot loses state, the obvious alternative is to keep the original browser alive. For some workflows, that’s exactly what you need, but it can really depend on how long you need this transient state for.

A running browser preserves far more than a cookie file. Open tabs and JavaScript state survive, and storage updates happen naturally. If the application refreshes a token in the background, the browser keeps the new value rather than reverting to an older snapshot on the next run.

While trading off persistence for long-running processes, you’ve traded one problem for another, and you still haven’t solved it! Instead of asking whether your authentication state can survive for three weeks, you’re asking whether a Chrome process can. I wouldn’t bet on the Chrome process.

Long-running browser processes fail in ordinary ways:

• Pages leak memory.

• Connections disappear.

• Hosts restart.

• Deployments happen.

• Chrome crashes.

• Your automation code crashes even when Chrome doesn’t.

• ... and all these problems compound with one another!

You’re also limited when you need to scale. One persistent browser is one mutable environment. If ten workers need the same account, letting all ten operate against one live browser introduces coordination problems. With a saved state snapshot, each worker gets its own isolated copy instead.

A long-lived browser is useful when you need continuity inside the browser, but it isn’t a general replacement for saved state.

How can you detect an expired session?

The worst recovery strategy is waiting for the scraper to throw an exception, because authentication failures often don’t throw one.

A protected URL might respond with a 302 to /login. Another site returns 200, but the HTML is now a sign-in screen. An API request might start returning 401. Some applications send you to an account verification flow instead.

I normally want an authenticated scraper to validate application state explicitly after restoring a session. Here’s a quick example:

await page.goto(PROTECTED_URL, {
  waitUntil: “domcontentloaded”,
});

const authenticated =
  !page.url().includes(”/login”) &&
  (await page.locator(”[data-account-id]”).count()) > 0;

if (!authenticated) {
  await reauthenticate(page);
  await saveFreshState(page);
}

Here we’re using other signals on the page to determine if it’s properly authenticated or not. We don’t ask whether the cookie exists; instead, we ask whether the application still considers the browser authenticated. You could script up a cookie check, but from what we’ve seen running this at Browserless, cookie checks are brittle and throw a lot of false positives.

In production, I’d also distinguish auth failure from ordinary scraping failure. A changed selector shouldn’t trigger a login. Neither should a 500, a network timeout, or a missing asset. Otherwise, a minor site change becomes hundreds of unnecessary login attempts, which is a much worse failure mode. That’s a mistake you only need to make once. You have to pick one or two of the strongest signals you can see on the page and use those for your authentication condition check.

When is re-authentication the right answer?

Re-authentication tends to get treated as something a good persistence system should eliminate. I’d argue the opposite: it’s a recovery path you should design deliberately. I’ve never run a system that works 100% of the time, even ones we own end to end. I don’t expect it from a script running against someone else’s site.

If login is cheap and stable, periodically rebuilding state can be simpler than maintaining a browser indefinitely. Preserving authenticated state becomes much more valuable when login involves MFA, email links, or aggressive rate limits. Either way, you want re-authentication to happen only after you’ve positively identified expiry.

Your choice also hinges on concurrency. If many workers need the same authenticated starting point, a reusable snapshot lets each one launch its own browser from it. Keeping the browser itself alive makes more sense when a workflow relies on in-progress UI state.

In practice, reliable systems often use both. Start sessions from a known authenticated snapshot, let each browser mutate normally while it is alive, and validate authentication before doing expensive work. When the saved snapshot is no longer accepted, run the login flow again and replace it.

Saved state and live sessions solve different needs

We ran into this distinction while building authenticated session support at Browserless.

Our Authenticated Profiles feature stores cookies, localStorage, and IndexedDB, then loads a copy of that state into a fresh browser session. We deliberately skip sessionStorage, since replaying temporary tab state can break short-lived flows such as OAuth redirects and CSRF handling.

Changes made inside one of those browsers don’t rewrite the original saved profile. Think of it as a fresh copy of the authentication state, or a reference by value and not address. If the underlying authentication state expires, the profile needs to be refreshed.

That is deliberately separate from session persistence, which comes in two flavors. A standard session flags the running browser as reconnectable mid-script, so you can drop back into the same process, same page, and same in-memory state within a short window.

Persisting state goes further: the Session API keeps cookies, localStorage, and cache in an isolated userDataDir that survives full browser restarts for days, and processKeepAlive decides whether the process stays up in between. The two combine: pass a profile name when you create the session, and the browser starts from that saved auth state and then persists normally.

One caveat for Playwright users: both of the keep-the-process-alive paths rely on browser.disconnect(), which Playwright doesn’t expose, so with Playwright you get the on-disk state but not the live process.

This distinction forces you to decide what you actually need. Do you need the same authentication state, or do you need the same browser?

Once your scraper runs for weeks rather than minutes, those stop being interchangeable.

Plan for the logout

If I could give one piece of advice to anyone building an authenticated scraper, it’s to stop trying to keep the session alive forever. We spent a lot of time on this problem while building Browserless, and the biggest shift for us was treating a logout as a state to plan for and not a bug to get rid of.

Every session goes stale eventually. What separates a scraper that runs for weeks from one that falls over on day three is whether it notices. So I save the full picture: cookies, localStorage, and IndexedDB, but not sessionStorage. I ask the app whether I’m still logged in instead of trusting a cookie. And I only log in again once I’ve confirmed the session is gone, then replace the snapshot so the next worker isn’t starting from last Monday.

As for snapshots versus live browsers, I’d stop thinking of it as a choice. Most setups I’d put into production use both.

You won’t build a session that never expires, and I wouldn’t try. Build one that knows when it’s expired and fixes itself before anyone gets paged. And if you’ve found a pattern that holds up longer than this, I’d like to hear about it!

You could scroll through feeds, follow hashtags, and check what each platform labels as trending. But that only gives you part of the picture. This happens because conversations move across platforms, engagement changes over time, and what looks like growing interest might just be the same content reposted over and over.

That’s where social media scraping comes in and makes the game easier. If you’re wondering why, the answer is simple: it lets you collect posts and engagement metrics systematically, then track how conversations evolve. However, collecting the data isn’t enough. You need a pipeline that separates meaningful signals from noise, handles differences between platforms, and accounts for the legal requirements of collecting and using social data.

In this article, we’ll discuss how to choose your sources, build the collection infrastructure, extract and clean trend signals with Python, and turn them into useful analysis.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

Understanding the Social Media Data Landscape

Before writing code, you need to map out the platform landscape you’re targeting. Social media scraping isn’t a one-size-fits-all discipline because each platform exposes data differently, enforces its own limits, and deploys different anti-bot measures.

So, you can’t just pick a social media platform at random and think to apply a scraping strategy to other platforms. You must align your platform choice with your trend-analysis goals and with the technical features each platform has.

Checking for Platform-Specific Data Structures

Once you’ve picked your social media targets, you’ll quickly notice that no two platforms structure their data the same way:

• Some platforms expose posts and engagement metrics through JSON-based GraphQL responses.

• Others return JSON too, but through obfuscated and session-dependent internal APIs. And these are harder to reverse-engineer.

• Others rely on server-rendered HTML. And this requires a more fragile extraction, as it is selector-based. But remember that, in such cases, a possible solution is using GPT vision to extract the data.

To make an example, X (formerly Twitter) fetches data through internal GraphQL endpoints visible in the browser through the DevTools:

Instead, LinkedIn fetches feed and post data through internal JSON API calls using obfuscated endpoint paths, rather than readable REST routes:

This means your social media scraping pipeline can’t use a single generic extraction strategy. You need tailored logic for each source, focusing on the specific social media.

API vs. Direct Scraping Trade-offs

You’ll also have to decide early on whether to lean on official APIs or go the direct scraping route. APIs give you stability and predictable schemas, but they often come with strict rate limits, high costs, and severe limitations regarding the data you can retrieve.

For instance, X is very specific about how developers should use their APIs and what happens if they don’t:

Even the LinkedIn API is very restrictive on use cases. For example, the public API doesn’t expose feed/post/engagement data for general use. This means that access is essentially locked to partners (limited marketing/talent solutions):

Direct scraping, on the other hand, gives you far more flexibility and access to richer data, but it raises the stakes. Basically, you’re taking on:

• Greater legal exposure.

• A heavier maintenance burden since platforms change their front-end structure without warning.

• Infrastructure management. You need to account for: geo-targeted proxies, a wide pool of residential proxies, CAPTCHA-solving capabilities, throttle and rate limiting management as these are high-traffic websites, and anything in between.

Only pay for data you actually get — one request, one credit, every time. No surcharge for JS rendering, no charge when a call gets blocked. Claim your free 10,000 requests with coupon code WEBSCR

Grab the deal!

Legal and Ethical Considerations When Scraping Social Media for Trends

Before moving into code and infrastructure, you need to think through the legal terrain you’re operating in. Otherwise, the risk is getting into legal trouble once you get the data you need.

If you’ve been web scraping for a while, you know that accounting for ethical web scraping is a best practice for any scraping case. But note that social media scraping is a delicate case. This is because it sits at the intersection of platform terms of service, privacy regulations like GDPR and CCPA, and copyright law that governs user-generated content.

This means that if you want to stay safe, you need a compliance framework in place from the beginning. And this matters even if you’re only collecting public posts for trend analysis. For this specific case, this is what matters most:

• Compliance with platform terms of service: Most platforms explicitly ban automated access in their terms of service, and they don’t always distinguish between a well-intentioned research project and a malicious bot. If you violate those terms, you’re risking account bans, IP blocks, or even legal action from the platform itself. That’s why you want to do a proper risk assessment before you start scraping.

• Handling personally identifiable information (PII): Even when you’re only after trend data, you’ll almost certainly capture PII (usernames, bios, locations, and sometimes more) along the way. That’s where things get tricky under regulations like GDPR, which don’t care whether the data was “public” when you scraped it. If you think you need this kind of data for some reason, you better build in an anonymization layer and apply it to the data.

• Respecting robots.txt: That’s probably the evergreen rule. And at The Web Scraping Club, we deeply discussed the implications of the robots.txt file. Remember that it’s not legally binding on its own, but ignoring it undermines your compliance framework. And if you do that, it can be used as evidence of bad faith if a platform pursues legal action over your social media scraping activity. So, treat it as a baseline signal of the platform’s intent, then layer your ToS and privacy review on top of it rather than relying on robots.txt alone.

And for the sake of being tied to reality, let’s check LinkedIn’s robots.txt file:

The above image reports only the initial part of it. But, as you can see, it’s very clear: you can not use automated software to access LinkedIn unless you ask for specific permission via e-mail. And you’re not sure you’ll get permission from LinkedIn, of course.

Building a Scalable Scraping Architecture

A trend-analysis pipeline isn’t a simple scraper running on a cron job. Once you’re dealing with high-velocity, high-volume streams of short-lived social content, you need distributed crawling, proxy rotation, and storage systems built to keep up with the momentum.

So, let’s consider the case where you decided to go with your custom scraper because the platform APIs don’t cover your case. In that case, after assessing the legal risk and defining the legal framework you’ll work on, you need to build the infrastructure.

Proxy Rotation and Anti-Detection Techniques

Social platforms don’t take kindly to automated traffic, and their anti-bot systems have become aggressive because they use machine learning to intercept detection triggers. To stay under the radar, you need to master at least the following:

• Residential proxies, since they’re far less likely to get flagged than datacenter IPs.

• Header randomization, so your requests don’t all look like they’re coming from the same bot.

• Browser fingerprint spoofing, which helps you avoid the CAPTCHAs and soft bans that kick in the moment a platform notices a suspicious pattern.

None of these alone is a silver bullet. But combined, they buy you the runway you need to keep collecting data without getting IP-banned every few hours.

Headless Browsers vs. Lightweight HTTP Clients

Here’s where you need to be strategic about resource allocation. Dynamic, JS-rendered feeds (think infinite-scroll timelines) often leave you no choice but to use headless browsers like Playwright or Puppeteer, since the content simply isn’t there until JavaScript executes. But that comes at a cost: headless browsers are slow and resource-hungry.

So, for static or semi-static endpoints, you’re better off with lightweight HTTP-based scrapers. They’re faster, cheaper to run at scale, and don’t need a full browser context spun up just to grab a JSON payload. The trick is knowing which endpoints actually need the heavy tooling and which don’t.

Scheduling and Real-Time Data Pipelines

Trend detection is a race against freshness. And a trend you catch six hours late isn’t a trend anymore. That’s why your scheduling strategy matters as much as your extraction logic. The rule of thumb is the following:

• Cron-based batch jobs work fine if you’re tracking slower-moving trends and can tolerate some lag.

• Streaming architectures (Kafka, message queues) are what you need if you’re after near-real-time detection. This is because, in that case, you need to process data as it arrives instead of waiting for the next batch window.

Which one you pick really comes down to how fast you need to know something’s trending, and how much infrastructure complexity you’re willing to take on to get there.

What If You Don’t Want to Manage The Infrastructure?

If you’re a scraping expert, you certainly know the burden of managing the scraping infrastructure: residential proxies are very expensive, CAPTCHAs block you, selectors change every one in a while, parallel requests take time to execute, and so on. Furthermore, the legalities of social media scraping get you up at night.

So…are there any other alternatives other than using official platform APIs and scraping on your own? The answer is yes!

As you know, the scraping market is strong, and lots of companies provide several services in it. Among the companies that provide scraping services, some offer social media scraping services. I don’t want to end up in a list here because everything is easily searchable on your favourite provider’s website. Just note that the majority of them provide pre-collected datasets and/or scraping APIs to retrieve data from social media.

In simple terms, you just need to make an API call and collect posts from Instagram, Facebook, TikTok, or whatever social media you’re targeting. The advantages of this approach are the following:

• The scraping service manages the infrastructure for you.

• You don’t need to manage the legalities behind the data extraction part. But, yes: you still need to account for the legalities of anyting related, like data retention, storage, and usage.

• If you’re ambitious and want to scale, you can do it along with the scraping service you’re using.

• You don’t need to think about how the platform displays the data. Either they expose APIs or render HTML, you make an API call, and you’re good to go.

• The price is generally fair. I mean: if you don’t own proxies or any other scraping infrastructure, API calls for retrieving data from social media platforms need just a few dollars. Furthermore, depending on the scale of your project, you may simply benefit from the service’s free trial.

So, if you’re scraping social media to intercept trends, the suggestion is to take a look at these solutions before writing your own custom code. You may be surprised by how you can find the right solution for your case.

Check the TWSC YouTube Channel

Data Management Strategies for Trend Signals

Grabbing raw posts from social media is only half the job. And that’s it, regardless of the data extraction method you used. To actually detect trends, your social media scraping pipeline needs to extract structured signals from content noise.

In social media posts, the fields that actually tell you if something is trending are hashtags, topics, engagement velocity, and mention frequency. So, let’s see how to manage these.

Extracting Hashtags, Mentions, and Keywords

Post text is messy. So, before you can count anything meaningful, you need to isolate the actual trend indicators from the surrounding noise. A “traditional” approach to do that is using regex.

That method gets you most of the way for hashtags and mentions. For example, consider the code below:

import re  
from collections import Counter  
  
def extract_signals(text):  
    hashtags = re.findall(r"#(\w+)", text)  
    mentions = re.findall(r"@(\w+)", text)  
    return hashtags, mentions  
  
posts = [  
    "AI is eating the world #AI #MachineLearning cc @openai",  
    "New drop today #sneakers #AI incoming",  
]  
  
hashtag_counter = Counter()  
mention_counter = Counter()  
  
for post in posts:  
    tags, mentions = extract_signals(post)  
    hashtag_counter.update(tags)  
    mention_counter.update(mentions)  
  
print(hashtag_counter.most_common())  

NOTE: For the sake of simplicity, the posts[] list reports short content. An example from a complete scraping scenario works by opening a CSV or JSON file, extracting the content, and applying regex.

The result from such a basic example is the following:

[('AI', 2), ('MachineLearning', 1), ('sneakers', 1)]

The above result is a list of tuples. Each tuple shows the hashtag and how many times it appears in the analyzed content. This is an elegant result because regex gets you hashtags and mentions cleanly, but it won’t catch implicit topics people are talking about without tagging them. For that case, you need more than regex. A solution is to use NLP-based tokenization for entity extraction on top of your script, like the following:

# pip install spacy
# python -m spacy download en_core_web_sm

import spacy  
  
nlp = spacy.load("en_core_web_sm")  
  
def extract_keywords(text):  
    doc = nlp(text.lower())  
    return [  
        token.lemma_ for token in doc  
        if not token.is_stop and not token.is_punct and token.pos_ in ("NOUN", "PROPN")  
    ]  
  
keywords = extract_keywords("New drop today, the sneakers are insane")  
print(keywords)   

And the result is:

['drop', 'today', 'sneaker']

In that case, this content can tell you:

• drop → Something has it or is hitting the market.

• today → That something occurred today.

• sneaker → The actual product.

Combine both approaches, and you’ve got a keyword pool you can feed into frequency counting or topic clustering downstream, without hand-tagging every post yourself.

As a final consideration, note that the NLP-based approach also has limitations. The main argument we could discuss is that this solution is based on the vocabulary you use. So, another technique is to use AI to extract entities and patterns from text. If you want to deepen that, we already covered this in the article “Using AI to Detect Patterns in Scraped Data”.

Capturing Engagement Metrics Over Time

Hashtags, mentions, and keywords tell you what people are talking about. Engagement metrics tell you how fast the news is spreading. And that’s a moving target. This means that you can’t scrape a post once and call it a day. You need to hit the same post at certain time intervals and build a time series out of it. The following code helps you do that:

import time  
import csv  
from datetime import datetime  
  
def scrape_engagement(post_id, fetch_fn):  
    data = fetch_fn(post_id)  
    return {  
        "post_id": post_id,  
        "timestamp": datetime.utcnow().isoformat(),  
        "likes": data["likes"],  
        "shares": data["shares"],  
        "comments": data["comments"],  
        "views": data.get("views", 0),  
    }  
  
def track_post(post_id, fetch_fn, interval_sec=900, iterations=8):  
    with open(f"{post_id}_engagement.csv", "a", newline="") as f:  
        writer = csv.DictWriter(f, fieldnames=["post_id", "timestamp", "likes", "shares", "comments", "views"])  
        if f.tell() == 0:  
            writer.writeheader()  
  
        for _ in range(iterations):  
            row = scrape_engagement(post_id, fetch_fn)  
            writer.writerow(row)  
            f.flush()  
            time.sleep(interval_sec)  

In this example code:

• fetch_fn is whatever extraction logic you’re using for that specific platform.

• The interval_sec and iterations are up to you. Tighter intervals catch virality earlier, but they also burn through your proxy pool faster.

Once you’ve got this data logged, you can compute engagement velocity (delta between snapshots) and feed it straight into the time-series analysis you’ll run downstream. Then, build a dashboard - for example, using Streamlit - and use Pandas, Matplotlib, and related libraries to analyze the scraped data and get trends out of your time series.

Need help with your scraping project?

Data Cleaning and Normalization

Social media trends don’t live in a single platform. A topic can break on Reddit, spike on X six hours later, and only show up on TikTok in a couple of days. So, if you’re only scraping one source, you risk tracking a fragment of the whole trend. As a consequence, if raw scraped data is messy by default when you get it from a single source, you can imagine the chaos with multiple sources.

This means your pipeline needs to standardize data across platforms. Otherwise, you’ll be comparing apples to oranges without realizing it. So let’s see some techniques to do so.

Deduplication and Bot/Spam Filtering

Bots and spammers are a real trouble for the web (they were so even in the “forums era”, if you’re old enough to remember it 😁). In the context of trend detection, if you consider them and reposts, you get inflated engagement. In other words, if you don’t filter them out, you’ll end up “detecting” trends that are really just coordinated inauthentic behavior.

Deduplication is the easy part. For that, content hashing gets you most of the way there as follows:

import hashlib

posts = [
    {"text": "AI is eating the world #AI #MachineLearning"},
    {"text": "AI is eating the world #AI #MachineLearning"},  # exact duplicate
    {"text": "  ai IS eating the world #AI #MachineLearning  "},  # duplicate after normalization
    {"text": "New drop today #sneakers #AI incoming"},
    {"text": "Another unrelated post about crypto"},
]

def content_hash(text):
    normalized = text.strip().lower()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

seen_hashes = set()
unique_posts = []

for post in posts:
    h = content_hash(post["text"])
    if h not in seen_hashes:
        seen_hashes.add(h)
        unique_posts.append(post)

print(f"Original posts: {len(posts)}")
print(f"Unique posts: {len(unique_posts)}")
for p in unique_posts:
    print("-", p["text"])

This code:

• Defines a content_hash() function that strips whitespace and lowercases the post text, then computes a SHA-256 hash of it. Note that normalizing before hashing means that formatting differences (extra spaces, capitalization) don’t prevent identical content from being detected as duplicates.

• Iterates over the list of posts, computing a hash for each one.

• Keeps a seen_hashes set to track which content hashes have already been encountered.

So, if a post’s hash is new, it’s added to both seen_hashes and the unique_posts list; if the hash already exists, the post is skipped as a duplicate. The result, unique_posts, contains only the first occurrence of each distinct piece of content. When runing it, you’ll see the following:

Original posts: 5
Unique posts: 3
- AI is eating the world #AI #MachineLearning
- New drop today #sneakers #AI incoming
- Another unrelated post about crypto

Now, the code above accounts only for exact matches. However, that won’t catch near-duplicates (slightly reworded reposts). For that, you’ll want something like MinHash or embedding similarity:

# pip install sentence-transformers

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")

posts = [
    {"text": "AI is eating the world"},
    {"text": "Artificial intelligence is taking over the world"},
    {"text": "New sneaker drop today"},
    {"text": "Sneakers just dropped today"},
    {"text": "Completely unrelated topic about cooking pasta"},
]

def find_near_duplicates(posts, threshold=0.6):
    texts = [p["text"] for p in posts]
    embeddings = model.encode(texts, convert_to_tensor=True)
    duplicates = []

    for i in range(len(texts)):
        for j in range(i + 1, len(texts)):
            sim = util.cos_sim(embeddings[i], embeddings[j]).item()
            if sim > threshold:
                duplicates.append((i, j, sim))
    return duplicates

results = find_near_duplicates(posts, threshold=0.6)
for i, j, sim in results:
    print(f"({i}, {j}) sim={sim:.3f} -> '{posts\[i\]['text']}' | '{posts\[j\]['text']}'")

This snippet does the following:

• Loads a pre-trained sentence embedding model (all-MiniLM-L6-v2) via SentenceTransformer.

• Encodes each post’s text into a dense vector embedding that captures its semantic meaning, not just exact wording.

• Compares every pair of posts using cosine similarity (util.cos_sim) to measure how semantically close their content is.

• Flags pairs whose similarity score exceeds the given threshold as near-duplicates. This catches reworded or paraphrased reposts that exact-hash matching (like the previous content_hash example) would miss.

• Returns a list of tuples (i, j, similarity_score) identifying which post pairs are likely duplicates.

As this solution uses machine learning, the actual similarity scores depend on the model run. So, the score you get can be slightly different from the following:

(0, 1) sim=0.72 -> 'AI is eating the world' | 'Artificial intelligence is taking over the world' (2, 3) sim=0.81 -> 'New sneaker drop today' | 'Sneakers just dropped today'

And, yes: this is a pretty good result!

Normalizing Timestamps and Multilingual Text

Cross-platform trend detection breaks down fast if you don’t normalize timestamps. This is because every platform reports time differently: some give you UTC ISO strings, others hand you Unix epochs, and some (looking at you, scraped HTML!) give you relative strings like “3h ago” that you have to resolve at scrape time before they become meaningless.

Timestamp conversion is the easy of the two:

from datetime import datetime, timezone
import dateutil.parser

def normalize_timestamp(raw_ts):
    if isinstance(raw_ts, (int, float)):
        return datetime.fromtimestamp(raw_ts, tz=timezone.utc)
    return dateutil.parser.parse(raw_ts).astimezone(timezone.utc)

print(normalize_timestamp(1735689600))
print(normalize_timestamp("2025-01-01T00:00:00+05:30"))

But relative timestamps need to be resolved against your scrape time, not stored as-is:

import re
from datetime import timedelta

def resolve_relative_time(raw, scraped_at):
    match = re.match(r"(\d+)([hmd])", raw)
    if not match:
        return scraped_at
    value, unit = int(match.group(1)), match.group(2)
    delta = {"m": timedelta(minutes=value), "h": timedelta(hours=value), "d": timedelta(days=value)}[unit]
    return scraped_at - delta

If you think that managing time is hard, consider that language is the harder problem. A trend can absolutely break in a non-English-speaking market first, and if your pipeline only handles English, you’ll miss it entirely. But the problem is that multilingual analysis can’t just be “run everything through a translator”. That’s where a lot of “translation” pipelines get sloppy.

To account for that, you first need language detection:

# pip install langdetect

from langdetect import detect
def detect_language(text):
    try:
        return detect(text)
    except Exception:
        return "unknown"
texts = [
    "AI is eating the world, and everyone is talking about it.",
    "L'intelligence artificielle est en train de conquérir le monde.",
]
for t in texts:
    print(f"'{t}' -> {detect_language(t)}")

And the result you get is as expected:

'AI is eating the world, and everyone is talking about it.' -> en
'L'intelligence artificielle est en train de conquérir le monde.' -> fr

Then, you can go through translation. Beyond translation quality, tokenization itself gets messy across languages. Whitespace-based tokenizers (fine for English) fall apart on languages without clear word boundaries, like Japanese or Chinese, where you need dedicated segmenters.

Overall, covering every nuance of multilingual translation and tokenization in depth is beyond the scope of this article. But knowing where these pitfalls lie helps you design a pipeline that’s ready to handle them as your trend-detection needs grow.

Analyzing Trends from Scraped Data

Clean data is a reliable starting point, but it doesn’t tell you what’s trending. Now, your social media scraping pipeline needs to distinguish normal conversation from unusual growth, identify what people are actually discussing, and measure how that discussion changes over time.

And there’s an important distinction to make: a popular topic isn’t necessarily an emerging trend. A topic with consistently high mention volume might just be part of the platform’s normal activity. Meanwhile, a smaller conversation that’s spreading across independent communities could deserve your attention much earlier.

Basically, you need two complementary layers of analysis:

• Time-series analysis for virality detection: Measure changes in mention volume, participation, and engagement against an appropriate baseline.

• Sentiment analysis and topic modeling: Identify the themes behind those changes and understand how people feel about the specific subjects involved.

Both kinds of analysis can require a lot of work and, surely, require specialized engineers. So, allow me to just give you the high-level ideas that matter.

Time-Series Analysis for Virality Detection

Start by deciding what you’re counting and when you’re counting it. This sounds obvious, but it’s where you can accidentally turn a collection artifact into a trend.

For mention volume, you should group posts into fixed time windows using their publication timestamps. For engagement growth, use the timestamps of your repeated metric observations. These aren’t interchangeable, because a post published yesterday can receive new engagement today.

So, your time-series dataset should distinguish:

• Mention volume: The number of distinct collected posts mentioning a topic within each window.

• Unique authors: The number of distinct participating accounts, measured within each platform unless cross-platform identities are reliably linked.

• Engagement increments: The change in a post’s likes, comments, shares, or views between observations.

• Collection coverage: Whether collection succeeded, which sources were available, and whether sampling or query settings changed.

That last point matters more than it might seem because if your scraper stops collecting for an hour, you have missing observations.

Sentiment Analysis and Topic Modeling

Time-series analysis tells you that something changed. NLP helps you understand what changed.
But don’t make the mistake of collapsing sentiment and topic detection into the same task because:

• A topic model groups related discussions.

• A sentiment model estimates expressed polarity toward something.

This means that neither, on its own, explains why a conversation is growing.

Also, note that the results - especially related to sentiment analysis - highly depend on the specific neural network you’re using. This is a wide topic, and we already covered it with a practical tutorial. If you want to deepen it, check out our article “Sentiment Analysis on Scraped Product Reviews: Step-by-step Tutorial”.

Putting It All Together: From Social Media Scraping to Trend Insights

At this point, you’ve seen the main pieces of the pipeline. But the actual value comes from connecting them. Social media scraping gets you the posts; extraction, cleaning, and analysis turn those posts into signals you can interpret.

So, before you start building, here’s how the whole process fits together:

In practical terms, these are the stages your pipeline needs to cover:

• Define what you’re tracking and where you can collect it: Choose platforms around your trend-analysis goals, then review the relevant access rules and legal requirements. Using a scraping provider changes who operates the infrastructure, but it doesn’t automatically remove your responsibilities around data collection and use.

• Collect posts and revisit engagement metrics: Decide whether to use official APIs, your own collectors, or managed services depending on your requirements. Store source identifiers, publication timestamps, and observation timestamps alongside the content. Remember that a dataset collected once can support historical analysis, but tracking engagement growth requires repeated observations.

• Extract signals, then clean and normalize them: Isolate hashtags, mentions, keywords, and engagement counters. Standardize timestamps and handle language differences before comparing sources. Also, distinguish duplicate collection from actual reposting: collecting the same post twice is a data-quality issue, while independent reposts can be evidence of a conversation spreading. Don’t automatically discard that signal.

• Analyze changes rather than raw popularity alone: Compare mention volume and engagement velocity against historical activity, then use topic modeling and sentiment analysis to understand the discussions behind those changes. And keep checking collection coverage.

• Visualize the results and monitor the pipeline: We haven’t explored data visualization in depth here, but it’s the final layer that makes your results easier to interpret. A Streamlit dashboard, for example, can display mention-volume curves, engagement growth, and topic or sentiment breakdowns. Just make sure it also shows data freshness and collection gaps, so readers know what they’re looking at.

Overall, the thing to remember is this one: you don’t get reliable trends by only collecting more posts. You get them by keeping the whole pipeline consistent, from source selection to the dashboard. And when a spike appears, you should be able to trace it back to the underlying posts and collection conditions.

Conclusion

In this article, you learned that social media scraping for trend analysis is about more than collecting posts and counting hashtags. You need a pipeline that brings together platform-specific collection, clean data, repeated engagement measurements, and analysis that distinguishes unusual growth from normal activity.

The value is in how you connect those pieces. A spike in mentions doesn’t tell you much if you can’t tell whether it came from independent conversations, repeated content, or a change in your collection coverage. But a system that helps you understand what’s gaining attention, where it’s spreading, and how the discussion is changing? That’s where the real value is!

So, let us know: Are you using social media scraping to track trends? Which platforms are you collecting from, and what’s been the hardest part of turning those posts into useful signals? Let’s discuss in the comments!

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

To fill this gap, today I’m creating a case study for smythstoys.com, the Irish site of Smyths Toys. I picked it because a reader described the exact problem on the r/webscraping subreddit: Smyths runs Imperva, the product pages are server-rendered HTML, and the one internal endpoint that returns stock is protected too. The idea in that thread was to keep one hot cookie per proxy, forged using a browser, and replay it over cheap datacenter IPs. I wanted to check three things: does that actually work, how long does the cookie last, and when does it eventually break?

This time I worked two ways at once. I built a bypass that warms a session with a browser, solves the challenge, and then runs on a plain HTTP client. And I ran Manus against the same site to read the protection from the inside (so I could explain why the bypass keeps working, instead of only showing that it does). The bypass is mine, while the deep read of the sensor, the token, and the renewal protocol is Manus’s, checked against my own captures where the two overlap.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

Manus, and how I used it here

Manus is an agent that pairs a language model with a sandboxed Linux VM and a Chromium browser. It can drive a page, run Node and Python, and instrument the browser while it executes, which is what makes it useful here. You may have noticed that in the latest The Lab issues I used that a lot: in THE LAB #114 against Kasada and THE LAB #115 against DataDome.
This is mainly for two reasons: it uses the sandboxed VMs, so it can spawn a browser and run tests in its virtual environment, and, at the same time, it’s not so heavily guardrailed, so these tasks can be performed with ease.

For this article, Manus hooked JSON.stringify and the btoa boundary before the page scripts ran, so it logged the exact sensor request and the clear-text fingerprint the moment they were built. Then it ran cold, warm and mutation matrices against the token endpoint. Its run was on 18 September 2026, in headless Chromium 151.0.7922.71, anonymous, thirteen page navigations, no credentials and no CAPTCHA solving. One detail from that setup matters later: its browser reported navigator.webdriver=true, so it announced itself as automated.

I trust the numbers Manus reported across those runs, and I checked the ones that agree with my own captures. Where I quote a Manus figure I say so, because I did not reproduce its in-browser instrumentation myself. Manus also did not vary the IP, the User-Agent or the TLS fingerprint, and it did not solve a CAPTCHA, so those parts of the story are mine.

The protection at a high level

Before jumping into the details, let’s see what happens if we cURL the target website:

$ curl -s -D - -A "Mozilla/5.0 ... Chrome/140.0.0.0 Safari/537.36" https://www.smythstoys.com/ie/en-ie/
HTTP/2 200
x-cdn: Imperva
x-iinfo: 10-32110005-0 NNNN RT(...) ...
set-cookie: visid_incap_2483049=...; Domain=.smythstoys.com; ...
set-cookie: incap_ses_416_2483049=...; Domain=.smythstoys.com; ...

The x-cdn: Imperva header and the visid_incap and incap_ses cookies are the Incapsula signature. Of course, what we receive it’s not the home page of the website but an error page. It’s a 1,158 byte HTML shell with an iframe that says “Request unsuccessful. Incapsula incident ID: ...”. So the edge already decided my plain client is not welcome, before any product page is served.

There are five pieces here, and it helps to name them before going deep on any one. The edge WAF answers the navigation with the interstitial (the “Pardon Our Interruption” page that sits between you and the shop), or with the real shop once you are cleared. A throwaway fingerprint cookie called ___utmvc is built in the browser and lives for about twenty seconds. The reese84 sensor is the real collector, two large obfuscated scripts of around 800 KB each loaded from paths that look like random english words. The token exchange is the POST that sends the sensor result and gets the reese84 token back. And the GeeTest CAPTCHA is what Imperva falls back to when the sensor result alone is not enough.

Tired of getting blocked while scraping the web?
ScrapingBee handles proxies, browsers, anti-bot systems, and retries so you can focus on your data. Get clean Markdown, JSON, or HTML from the web with up to a 99.9% success rate. ScrapingBee is SOC 2 Type II and GDPR compliant, and trusted by 4,000+ developers.

Discover ScrapingBee

Manus drew the whole thing as one diagram, which is the clearest way to see how these pieces connect.

The server-side scoring is out of scope, for the simple reason that nobody outside Imperva can see it. Everything I claim below is something one of us saw live, in the cookie jar, or in the sensor code as it ran. We have covered the general idea before: cookies and sessions as a clearance store in THE LAB #94, and a browser-to-HTTP handoff in THE LAB #100. This is the same shape, applied to Imperva, with the extra question of what the token is tied to.

The bypass: warm a session, then run without a browser

The bypass has three tools, each for one job.

Scrapling does the warming. Its StealthySession launches a Chromium browser and keeps cookies across requests. I use it because it gives me a page_action callback with the real Playwright page inside, so I can watch the network and drive the challenge frame by hand.

CapSolver solves the CAPTCHA. Imperva shows a GeeTest challenge when the sensor alone is not enough, and CapSolver has a GeeTest task type that returns the three values GeeTest wants back.

curl_cffi does the replay. It sends HTTP requests with the TLS fingerprint of a real Chrome, without a browser, so I can test whether the warmed cookie works on its own.

All of it runs in the shared venv on Python 3.11. My code is in the 116-Imperva-Reese84 directory of The Lab repository, with the scripts and the captured results, so you can rerun every number that carries my name below.

When I warm a session with Scrapling and log every request on smythstoys.com, the order is always the same. The browser loads the home page and gets the interstitial. The sensor scripts load. The sensor POSTs its blob to a random path, and a second POST returns the token:

POST /mbit-And-Dirers-him-Face-and-sure-such-Parry-qui?d=www.smythstoys.com
{"token":"3:nHkG6Ry2bxW+gbWD+elLTg==:RZgGQ1gd...","renewInSec":699,"cookieDomain":"www.smythstoys.com"}

That token value becomes the reese84 cookie. The renewInSec is 699, so the browser is told to refresh the token in about eleven and a half minutes. The cookieDomain is the host, so the cookie is scoped to www.smythstoys.com. On Smyths this token alone was never enough for my headless Chromium. Every warming run ended on a GeeTest challenge, so I had to solve it.

Check the TWSC YouTube Channel

The challenge flow is readable in the interstitial frame. A GET to /_Incapsula_Resource?SWCNGEEC=... returns the GeeTest parameters:

{"challenge": "230a350828feeac904b737841e83ecd8", "gt": "ce33de396f8d04030f6eca8fbd225070", "new_captcha": true, "success": 1}

The gt was the same across every run, ce33de396f8d04030f6eca8fbd225070, which is the site’s GeeTest id. The challenge changes each time. When the widget is solved, the page POSTs the result to /_Incapsula_Resource?SWCGHOEL=gee&... with geetest_challenge, geetest_validate and geetest_seccode, then reloads the parent frame.

My warm_session.py does exactly that, from inside the frame. It reads the two URLs out of the frame’s HTML, calls the GET to get gt and challenge, sends those to CapSolver as a GeeTestTaskProxyLess task, and gets back the three values. Then it POSTs them to the SWCGHOEL=gee URL using the frame’s own fetch, so the request carries the session cookies, and reloads the page.

CapSolver returned a solution in about ten to twelve seconds each time. It’s not perfectly reliable: I saw it fail with ERROR_CAPTCHA_SOLVE_FAILED code 1001 on some challenges, so the script retries up to three times. GeeTest solves cost one dollar per thousand on CapSolver when I ran this, so a failed attempt followed by a good one is still a fraction of a cent. After the solved reload, the page title finally read “Smyths Toys Superstores | Buy Toys for Kids”, which is the real shop. Here is the tail of a real run, warmed through an Irish residential proxy:

attempt 1: challenge 6abbf70a... gt ce33de396f8d04030f6eca8fbd225070
capsolver solved in 10.3s
imperva captcha submit -> 200 '<html><head><script>window.parent.location.reload(true);</script></head>...'
SAVED status=REAL_AFTER_CAPTCHA cookies=[incap_ses, reese84, incap_sh, visid_incap, nlbi, nlbi]

Now the part that makes the whole thing worth doing: does the clearance survive without the browser? I take the cookie jar from that warmed session (the full set of cookies the browser collected) and replay it with curl_cffi, no browser, impersonating Chrome. I fetch the home page and the Magic The Gathering category page the reader mentioned:

jar: warm_pub01.json  status=REAL_AFTER_CAPTCHA  minted 86s ago
warm proxy (same exit)   | home      | 200  603618 REAL | 'Smyths Toys Superstores | Buy Toys for Kids'
warm proxy (same exit)   | category  | 200  471472 REAL | 'Magic The Gathering | Smyths Toys Ireland'
replay proxy             | home      | 200  603752 REAL | 'Smyths Toys Superstores | Buy Toys for Kids'
replay proxy             | category  | 200  471605 REAL | 'Magic The Gathering | Smyths Toys Ireland'
direct (no proxy)        | home      | 200  603751 REAL | 'Smyths Toys Superstores | Buy Toys for Kids'
direct (no proxy)        | category  | 200  471605 REAL | 'Magic The Gathering | Smyths Toys Ireland'

The warmed jar returns the real page from the same Irish residential IP that created it, from a datacenter IP in Ohio, and from my own connection in Italy. I ran the same jar from a second Irish residential IP too, with the same result. So, at least in my tests, the clearance is not bound to the minting IP. This is what makes the reader’s architecture work: you warm on a good residential IP, then you replay over whatever is cheapest.

I went further and varied the HTTP client itself, same warmed jar, direct connection:

chrome136 + Chrome UA (match)      | 200  467891 REAL
firefox135 + Firefox UA            | 200  467891 REAL
chrome136 + Firefox UA             | 200  467891 REAL
no impersonation + Chrome UA       | 200  467891 REAL

On this site, once the jar is warmed, the TLS fingerprint and the User-Agent stop mattering. Even curl_cffi with no impersonation, which sends a TLS handshake no browser would send, gets the real page. So the warmed reese84 works as a bearer token (a token that works for whoever presents it, with nothing tying it to one client): any client that presents the jar gets the real page back.

That is the working bypass. The rest of the article explains why it holds, using the Manus report.

Why the replay works

Read more

Given that there are so many aspects to consider, such as your knowledge of web scraping techniques and experience in the field, I tried to put some numbers to that.

What I carved out was a general good practice: if you need something just once, buy it. If you need a feed that runs for years, build it yourself, since you can spread out the setup cost and keep the running costs low. This assumption was based on the fact that, at that time, the most important voice cost was a developer's hourly rate.

Three years later, AI and LLMs changed these assumptions. Once you’ve properly set up your coding environment with skills and best practices, the cost of creating and even maintaining a scraper is near 0.

This makes the idea of building everything in-house more attractive again, and in this article we’ll dig into some numbers to understand if this is really true or not.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

Buy versus build is three separate decisions

The first problem is the question itself. “Do we build or buy our scraping?” contains three layers, each with a different answer.

There are the code production costs, given by who writes the scraper and who fixes it when the site changes. There are infrastructure costs, which include proxies, stealth browsers, anti-bot bypass, and captcha solving. And there are the execution costs: who runs the scrapers and who keeps them running.

Until 2023, the code layer was the first cost to consider, followed by the infrastructure ones. Depending on the size of the scraping project, the costs of building could lean towards the first or the second layer. The execution layer, instead, depends on the tool: do you need a browser for scraping? If so, you need way more CPU and memory capacity compared to a browserless scraper.

The code layer in 2026

It would be easy to say that now code is “free”, so this layer went down to zero.

The reality is that the cost is near zero (you still buy tokens and have a human in a seat, for much less time), just for experienced teams that spent weeks on building the harness for their scraping agents.

I can share my example: I’ve spent some time on this task, creating a skill for every open source tool I’m using and also for the ones I’m not, but they could be useful for building scrapers on the hardest targets. Today there are 32 of them, about 33000 words, covering Camoufox, Patchright, nodriver, rnet, curl_cffi, Scrapling, RayoBrowse, Kameleo, Ghidra, Frida, and many others. Each skill says what the tool does, which parameters matter, and when to avoid it (don’t shoot a fly with a bazooka; it’s the first rule for a successful scraping project).
My actual problem is that these skills are going to be stale soon, so the next step is to set up a monitor for each of these items and check the newest versions on GitHub, so skills can be updated automatically.

That’s, in my opinion, the best way to leverage LLMs to create and maintain in-house scraping projects.

If you’re starting today and you’re not interested in learning about scraping, since you have just a bunch of random requests, then scraping APIs are the tool for you. You just set them up and get the data, usually with no hassle even from medium-protected websites.
You save money and time, unless the projects requires a huge amount of results to be scraped. The breaking point between buy and sell here depends strictly on your expertise and the size of the project.

Oxylabs Web API — built for agentic search

AI agents hallucinate, fresh data doesn’t. Our new Web API delivers fresh, real-time web data so your agents stay accurate, relevant, and ready to scale.Become an early user, try our new Web API, and share feedback so we can build a solution that better fits your needs.

Request Early Access

The infrastructure layer

Let’s say you decided to build your scraper in-house and wrote a working scraper using LLMs that returns a bunch of records. Now you have to launch a full run and maybe do it recurrently. That’s where things usually break, and the costs rise.

The buy vs. build on this layer in this case means: should we glue together in-house a stack of tools (proxies, TLS fingerprinting spoofing, browser), with the related running costs?

Buying here means getting a proxy provider, a web unblocker, or a browser-as-a-service. You pay per gigabyte or per request, according to what you’re buying. A vendor sells you a success rate, and it is worth more the less you want to think about the target.

While for the code layer, a structured team-building approach seems the obvious choice, here the choice is not so easy. There are more factors to keep in mind, apart from costs and development time: how critical is this scraping project? How much should I pay for the peace of mind of delegating this project to a vendor? How do I keep up with which tools are now bypassing anti-bot X? (spoiler: an auto-updated knowledge base like The Web Scraping Club Wiki).

The interesting part compared to 2023 is that now that almost every big name in the proxy industry is moving towards browser scraping, which can be used both for tackling hard websites and for agentic automation, unblocker APIs are getting a bit overshadowed. I’ve found that their prices have lowered to 1-2 USD per 1k requests, which makes the costs comparable to a high-priced residential proxy in some cases (which are usually sold per GB).

This is another challenge in answering the question of buy vs. build. How do I know the total price of my scraping project? I should consider the bandwidth for the proxies, the total number of requests in case of an unblocker API or Scraping Browser.

We compared the pricing models in The Web Unblocker Cost Benchmark (data is a bit stale, updating it this week).

The execution layer

The execution layer, where your scraper actually runs, depends mainly on the tools you chose earlier. You can manage your own cloud setup, go serverless, or deploy the spider to a platform and forget the machine, sized properly. I went through the trade-offs in Optimizing costs for large-scale scraping operations: serverless is cheap for short, sporadic jobs but expensive when something runs all day. This is the layer that costs the least and, even with LLMs, can be managed more easily than before, so I won’t spend too much time on it.

Running the numbers again

Let me redo the 2023 exercise at the same $20 an hour, so the two can be compared. The hours in both columns are my estimates. For the model cost, I use Claude Opus 5 at its list price today: $5 per million input tokens and $25 per million output. A build session on an easy site burns something like 1M input and 100K output tokens once you count the retries, three times that on a medium site, fifteen times on a hard one. Batch requests cost half the list price, and cache reads cost a tenth of the input price, so the table shows the expensive version of the model's cost.

I added a traffic column to the calculation. When you buy an external tool, usually proxies are included, while when you build, the proxies are a separate invoice, and usually not a marginal one.

So I measured it. I took four ecommerce listing pages, on ikea.it, mediaworld.it, etsy.com and asos.com, and counted the bytes twice. A median to use in this calculation is 100 KB for an HTML fetch and 1.7 MB for a full render, with residential proxies at $4 per GB, a scraping browser at $8 per GB, and an unblocker at $1.50 per 1,000 requests with the proxies included.

I’ve made several assumptions for creating the following table, on top of the token calculated before:

• A structured web scraping team is working on it, so we’ve all our skills and guidelines in place

• I decided to compare build costs with unblockers for websites with no or light anti-bot protection, since we assume the HTML parsing phase is done in-house. This favors the buy solution, since scraping APIs are generally more expensive, but it’s also true that HTML parsing incurs a marginal cost when building a scraper with LLMs.

• I categorized websites between light anti-bot (don’t require a browser for being scraped) and hard anti-bot (requiring a browser in-house or a scraping browser to buy).

• Infrastructure costs are not included, since they’re relevant only in the hard anti-bot scenario.

Considering the three oversimplified scenarios in the table, a website without bot protection is not worth buying, since the costs of writing and running a scraper are near 0.

The middle scenario is the most ambiguous: depending on your time constraints, your real-world tool pricing, and LLM plans, it can be more convenient to buy or build.

On the hardest tier, both sides have to run a browser and carry the same 1.7 MB, so proxy and execution costs are way higher than in previous scenarios, and building should be convenient only after more than 100k requests are made.

Check the TWSC YouTube Channel

Generally speaking, the break-even fell everywhere: by about two-thirds on the easy sites, by more than half in the middle, and by barely a tenth at the top. The cheaper the code got, the less it helped on the targets where the code was never the expensive part.

Maintenance moves the same way. In 2023 I budgeted two hours a month per scraper. When the fix is a selector or a changed JSON path, an agent proposes the patch and I confirm it, so half an hour is closer to reality. On the hard tier, nothing improves, because there the work is usually understanding how to bypass the website again, not just a selector change.

Need help with your scraping project?

The five percent nobody sells

The commercial tools cover most of the web, and they cover it well. Then there’s a top slice of sites, few of them and usually the ones the business cares about most. There, the problem is a virtual machine inside an obfuscated script, or a sensor reading properties straight from the browser engine. No vendor publishes a price list for a specific interpreter on a specific domain, with an SLA.

Chatting around at the latest Oxycon, despite the fall in code creation costs, the R&D time spent on certain websites (especially Chinese ecommerces with custom-built solutions) makes them non-lucrative for companies selling scraping data.

Other considerations

Building now is cheap to start now and expensive to leave. It may take an afternoon to build fifty scrapers, but managing and checking their results require a small, dedicated team. The per-scraper cost fell, and the number of scrapers went up, so the total can easily be higher than before.

And there is the failure mode described in Stuck? More of the Same Won’t Do, published here in 2025: teams that build keep building well past the point where it pays. Agents make that easier, because progress feels continuous even when you’re stuck. Decide in advance how many weeks you give an internal attempt before you go and buy.

Final considerations

Build the code because that cost has collapsed, and what you get is yours. Buy the infrastructure, if you don’t want scraping to be your main business. The execution layer follows whatever ops capacity you already have.

The time you save goes into checking that the data is right. No model does that part for you yet.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

In this hands-on post, I’ll put webclaw to the test and share my final take!

webclaw Explained: What It Is and What It Offers

webclaw is a web data extraction toolkit built in Rust that acts as a drop-in Firecrawl alternative. Its core mission is straightforward: turn any URL into clean, LLM-ready content. What sets it apart is that it’s designed specifically for AI agents and LLM pipelines.

The GitHub repository of the project boasts over 2.3k stars. You can either self-host it or use it via the API through the cloud version.

Main Features

The core capabilities exposed by WebClaw include:

• 10+ REST endpoints: Scrape, crawl, extract, search, map, batch, summarize, research, brand, diff, and lead enrichment endpoints + Firecrawl-compatible endpoints available through the same API interface.

• Multiple output formats: Get web pages as Markdown, JSON, plain text, or an LLM-optimized format with links and other metadata.

• JavaScript rendering: The CLI version can access pages that require JavaScript rendering.

• Document parsing: Extract text from PDFs, DOCX, XLSX, and CSV files.

• Content change tracking (diff): Compare current snapshots with previous extractions to detect what has changed.

• Proxy rotation support: Automatically loads proxies.txt from your working directory for automated proxy rotation.

• Browser actions: Click, type, scroll, wait, take screenshots, press keys, and execute custom JavaScript before extraction.

• Direct integration with AI agents and harnesses: Exposes 10+ tools over stdio for Claude, Cursor, Windsurf, Claude Code, Codex, and Antigravity.

Learn more in the official docs.

Architecture and Products

webclaw’s architecture is straightforward, with the same underlying extraction engine for:

• webclaw: A CLI for extracting, crawling, summarizing, and tracking changes from the terminal.

• webclaw-server: A REST API server that provides access to webclaw through JSON endpoints.

• SDKs: Libraries for Python, TypeScript, and Go that simplify integration with the webclaw REST API.

• webclaw-mcp: An MCP server that exposes 10+ tools for use with any MCP-compatible AI client.

As a result, whether you’re pulling web data through the CLI, REST API, an SDK, or the MCP server, the same core code runs under the hood. This unified architecture means you get consistent, predictable behavior whether you scrape from the terminal, call the REST API directly, integrate webclaw into your scripts, or use it via an AI agent.

Let me introduce you to the three webclaw interfaces!

CLI

The webclaw command-line tool is available through the webclaw command. It provides a range of options for extracting, crawling, and processing web content directly from the terminal.

Common commands include:

• webclaw <url>: Extract a single URL to Markdown

• webclaw https://docs.example.com --crawl --depth 2 --max-pages 100: Crawl a website with depth and page limits

• webclaw https://example.com --map: Discover URLs from a website’s sitemap.xml and robots.txt.

• webclaw https://example.com --extract-json @schema.json: Extract structured JSON based on a schema using the configured LLM provider (Ollama, OpenAI, or Anthropic).

• webclaw https://example.com --diff-with snapshot.json: Detect changes since a previous snapshot.

• webclaw https://example.com --include "article" --exclude "nav,footer": Control which page elements are extracted using CSS selectors.

• webclaw vertical amazon https://amazon.com/... : Use a site-specific vertical extractor, returning typed JSON instead of generic Markdown.

• webclaw https://example.com --cloud: Route extraction through the webclaw cloud

For the complete list of commands and options, see the webclaw CLI documentation.

Common use cases include terminal-based web data collection, automation workflows, and one-off scraping tasks. One of the CLI’s main advantages is that you can pipe commands together. For example, you can pipe a JSON response through jq to extract only the fields you need.

REST API

webclaw comes with an axum-based HTTP server that handles authentication, CORS, gzip compression, and asynchronous job management. The main endpoints include:

The workflow is simple: send a POST request with a URL and your desired options, and webclaw returns the extracted data in the requested format.

For users who prefer a managed solution, the cloud version of the APIs is available at api.webclaw.io. This is managed directly by webclaw and offers automatic scaling, proxy rotation, and other premium features out of the box.

SDKs

webclaw provides official SDKs for Python, TypeScript, and Go. They simplify integration with the cloud REST API, while offering typed clients, built-in error handling, and crawl polling. All SDKs support the full API surface and authenticate using a webclaw API key.

In detail, the Python SDK (available as the webclaw PyPI package) works like this:

# pip install webclaw
from webclaw import Webclaw

with Webclaw("YOUR_WEBCLAW_API_KEY") as client:
    # Scrape the "https://example.com" page via webclaw's cloud APIs
    result = client.scrape(
        "https://example.com",
        formats=["markdown"]
    )
    
    # Print the returned Markdown version of the page
    print(result.markdown)

The TypeScript and Go SDKs provide equivalent functionality for their respective ecosystems.

MCP Server

The MCP server exposes 14 tools over STDIO:

The easiest way to set up webclaw with Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, or another MCP-compatible client is to run:

npx create-webclaw

Alternatively, you can configure individual clients manually. For Claude Code, launch:

claude mcp add webclaw -- npx -y @webclaw/mcp

For Claude Desktop, add the following to your claude_desktop_config.json configuration:

{
  "mcpServers": {
    "webclaw": {
      "command": "npx",
      "args": ["-y", "@webclaw/mcp"],
      "env": {
        "WEBCLAW_API_KEY": "<YOUR_WEBCLAW_API_KEY>"
      }
    }
  }
}

The MCP server uses a local-first architecture. When webclaw detects a bot-protected or JavaScript-heavy page, it automatically falls back to the webclaw cloud API if a WEBCLAW_API_KEY is configured.

Deployment Options

webclaw comes with two deployment models:

• Self-hosting: A single open-source Rust binary that exposes the extraction engine through a local HTTP server.

• Cloud API: A fully managed service, where you sign up, create an API key, and start extracting data by calling webclaw’s remote APIs.

Explore the pros and cons of each option!

Self-Hosting

webclaw’s full Rust stack is available under the AGPL-3.0 license and can be self-hosted using Cargo installs or via Docker images. The self-hosted version also supports Ollama, allowing you to run the LLM-powered extract and summarize operations entirely on your own hardware.

🎯 Best for:

• Users who want unlimited local usage without a credit system.

• Sites that don’t require JavaScript rendering and don’t involve bot protection.

• Users who need full control over your data and infrastructure.

⬇️ Drawbacks:

• You’re responsible for infrastructure, monitoring, maintenance, and scaling.

• Synchronous crawls are limited to 500 pages.

• You don’t get built-in bot-protection handling, JavaScript rendering, or proxy rotation.

Cloud API

The webclaw Cloud API is a fully managed service, so there are no servers to run or infrastructure to maintain. Compared to the self-hosting option, it’s suitable for more demanding scraping workloads.

🎯 Best for:

• Sites that require JavaScript rendering.

• Users who need scalable web data collection and don’t want to manage the infrastructure themselves.

• Teams that want usage tracking, billing, and predictable monthly quotas.

⬇️ Drawbacks:

• Usage is subject to monthly page quotas based on your plan.

• Your data is processed through the webclaw cloud infrastructure.

• Monthly rate limits based on your plan.

webclaw in Action: Getting Started

In this section, I’ll walk through practical examples of using webclaw through its CLI and REST API. Keep in mind that you can achieve the same results through the SDKs or directly from AI agents using MCP integration.

Disclaimer: webclaw provided me with a Scale plan with 1,000,000 credits to thoroughly test the product.

Installation and Setup

On a system with a working Rust toolchain, run this command to install the webclaw CLI:

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli

Once the installation is complete, verify it by running:

webclaw --version

Alternatively, for a simpler setup, you can run webclaw directly with Docker:

docker run --rm ghcr.io/0xmassi/webclaw "<TARGET_URL>"

To check that the Docker setup is working, run:

docker run --rm ghcr.io/0xmassi/webclaw --version

In both cases, the command should return:

For more guidance, refer to the “Getting Started” guide.

To use webclaw’s Cloud API features, create an account. On the “API Keys” page of the dashboard, you’ll find a default API key that was automatically created when your account was set up:

Alternatively, click “Create key” and follow the setup wizard to generate a new webclaw API key.

Example #1: Scrape a Web Page in Multiple Formats

Suppose you want to scrape the “Quotes to Scrape” website. Achieve that with the webclaw CLI as below:

webclaw "https://quotes.toscrape.com/"

Note: If you’re a Docker user, remember to replace webclaw with docker run --rm ghcr.io/0xmassi/webclaw.

By default, webclaw will return Markdown content:

Now, assume you want to access the same page in JSON format. Specify the --format "json" option:

webclaw "https://quotes.toscrape.com/" --format "json"

This time, you will get a structured JSON response containing the web page in Markdown:

Notice how it also contains a links array with all the URLs on the page.

Finally, test the special LLM-ready output:

webclaw "https://quotes.toscrape.com/" --format "llm"

The result will look like this:

The produced data format combines the structured metadata of the JSON output with the readability of Markdown. It’s optimized for LLM consumption, removing unnecessary line breaks and whitespace to reduce token usage while keeping the content structured and easy for AI models to process.

Note that you can achieve the same result by calling the webclaw Cloud API like this:

curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer <YOUR_WEBCLAW_API_KEY>" \
  -H "Content-Type: application/json" \
 -d '{"url":"https://quotes.toscrape.com/","format":["markdown"]}'

When you pass the output through a JSON beautifier/parser, the result will be:

Example #2: Access a Dynamic Web Page Requiring JavaScript Rendering

Now, target a webpage that requires JavaScript rendering, such as the JavaScript version of “Quote to Scrape”:

webclaw "https://quotes.toscrape.com/js/"

The result will be:

Notice that the returned Markdown contains only the page’s static content. It doesn’t include the content dynamically added by JavaScript.

To scrape JavaScript-dependent pages, you must rely on the webclaw Cloud API:

curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer <YOUR_WEBCLAW_API_KEY>" \
  -H "Content-Type: application/json" \
 -d '{"url":"https://quotes.toscrape.com/js/"}'

This time, the markdown field contains the complete page, including the content rendered dynamically by JavaScript:

Note: In my tests, the Cloud API wasn’t able to scrape any pages that the CLI couldn’t already scrape. At the time of testing, this suggests that the anti-bot bypass functionality may not be working as expected.

Check the TWSC YouTube Channel

Example #3: Scrape with CSS Selector Filtering

Suppose you want to extract only a particular section of a webpage:

You can use the --include option to select specific elements (or --exclude to filter them out). Here, you only want to extract the #prodInfos element. So, write:

webclaw "https://world.openfoodfacts.org/product/6111242100992/perly-jaouda" --include "#prodInfos"

The result will contain only that specific section of the page:

You can achieve the same outcome with the Cloud API by using the include_selectors field in the request body:

curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer <YOUR_WEBCLAW_API_KEY>" \
  -H "Content-Type: application/json" \
 -d '{"url":"https://world.openfoodfacts.org/product/6111242100992/perly-jaouda","include_selectors":["#prodInfos"]}'

Now, suppose you want to scrape a webpage with a header and footer, such as a blog post. In this case, you are generally interested only in the main body content. webclaw supports that use case via the --only-main-content option:

webclaw "https://world.openfoodfacts.org/product/6111242100992/perly-jaouda" --only-main-content

Similarly, you can use the only_main_content field with the Cloud API:

curl -X POST https://api.webclaw.io/v1/scrape \
  -H "Authorization: Bearer <YOUR_WEBCLAW_API_KEY>" \
  -H "Content-Type: application/json" \
 -d '{"url":"https://world.openfoodfacts.org/product/6111242100992/perly-jaouda","only_main_content":true}'

Example #4: Crawl a Site

webclaw supports Breadth-First Search (BFS) crawling, allowing you to discover and scrape multiple pages starting from a single URL. For example, you can crawl up to 10 pages from Quote to Scrape with a maximum depth of 3:

webclaw "https://quotes.toscrape.com/" --crawl --depth 3 --max-pages 10

The command will return the discovered pages along with the Markdown content extracted from each one:

You can perform the same crawl through the Cloud API:

curl -X POST "https://api.webclaw.io/v1/crawl" \
  -H "Authorization: Bearer <YOUR_WEBCLAW_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://quotes.toscrape.com/","max_depth": 3,"max_pages": 10}'

Unlike the CLI command, the Cloud API processes the crawl asynchronously. The initial request returns an ID for the crawl task:

You can then use this ID to poll for the crawl status:

curl "https://api.webclaw.io/v1/crawl/88e08765-f44a-4e26-a4d6-3552bef983cb" \
  -H "Authorization: Bearer <YOUR_WEBCLAW_API_KEY>" \
  -H "Content-Type: application/json"

Once the crawl is complete, the response will look similar to this. The pages array contains the content extracted from all successfully scraped pages:

Need help with your scraping project?

Pricing Options and Plans

As of this writing, webclaw’s pricing plans can be summarized as follows:

All cloud plans can be billed monthly or annually, with annual billing offering two months for free.

Important notes:

• The credit pool is shared across all API endpoints.

• A free tier with 3 runs per day is available indefinitely, with no credit card required.

• The Starter plan includes a card-required 7-day free trial, giving you temporary access to the plan before you commit.

• Self-hosting provides unlimited local usage with no credit system.

The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Final Thoughts on webclaw

webclaw is definitely an interesting solution, and I personally found the CLI easy to use. The API interface is also quite intuitive, and its Firecrawl compatibility is definitely a plus.

The project is still young (the first commit was made in March 2026), and this shows in a few areas (e.g., unexpected behavior of input parameters or defaults). I also found some inconsistencies between the GitHub page and the official website and documentation. Still, that’s fairly common with fast-evolving projects, I have to say. (That’s why technical writers like me still have a job… for now, at least! 😉)

I especially appreciated the support from Valerio Massimiani, the founder of webclaw. He provided me with a Scale plan, remained available to answer my questions, took care of my FAQ requests, and was responsive throughout the process.

On the other hand, I found the self-hosting setup quite difficult, especially on Windows and when installing via Cargo. Docker, by comparison, was super easy.

I also found the overall anti-bot experience somewhat inconsistent. webclaw was able to bypass challenging sites such as L’Occitane, Temu, and Nike even from the CLI. However, when the CLI failed because of anti-bot protection (e.g., on Cloudflare-protected sites, Hermès, Canada Goose, and others), the cloud version failed as well. At least, in my tests…

That being said, the project has plenty of potential and is clearly moving forward quickly. I’ll definitely keep an eye on it and see how it develops, especially given how interesting the overall concept is and how easy it is to integrate webclaw with local AI agents.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

FAQ

Disclaimer: The following FAQ questions were answered by Valerio Massimiani, the founder of the project.

How does webclaw differ from Firecrawl?

webclaw is written in Rust and skips the browser for most pages, so it’s faster and cheaper to run. Output is LLM-shaped by default, and an MCP server lets agents call it directly. Firecrawl-compatible /v2 endpoints mean existing Firecrawl code can point at it unchanged.

Can I use webclaw with my existing web scraping code?

Yes. There are REST endpoints plus official Python, TypeScript, and Go SDKs, so it drops into existing pipelines. If you already run Firecrawl, the /v2 compatibility layer covers scrape, crawl, map, and search, so most code moves by changing the base URL and key.

Does webclaw come with official agent skills?

Yes. npx skills add 0xMassi/webclaw-skill installs two: a webclaw skill covering the MCP tools, and a lead-enrichment skill that runs discovery and enrichment over /v1/extract. Both work in Claude Code and any agent that reads the skills format.

Does webclaw support proxy integration?

Yes. Self-hosted, pass --proxy for a single upstream or --proxy-file to rotate a pool per request; both also read from environment variables. HTTP and SOCKS5 are supported. On the managed API, proxying is handled for you and needs no configuration.

What’s the difference between the webclaw CLI and the REST API?

Same extraction engine, different entry point. The CLI runs locally, needs no key for most sites, and suits one-off scrapes and scripts. The REST API adds authentication, async jobs for crawl and research, and managed handling of protected pages. The output is identical.

How does the LLM optimization actually work?

The llm format strips navigation, footers, scripts, and tracking markup, then flattens what remains into compact prose with headings and links preserved. The result is a fraction of the original token count. Schema-based extraction goes further and returns typed JSON you can validate.

Is my data private with the webclaw Cloud API?

Content is fetched per request and returned to you. Results may be cached briefly so repeat requests for the same URL are faster. Public pages aren’t private data, but don’t route credentials through the API. Self-host if you need full control of the pipeline.

To help you understand that better, I decided to write a dedicated article on that matter. This article explains what tools are for LLMs, how tool calls work, and presents the limitations of web search tools. Also, it presents two practical sections where I test commonly used models and their web search tools. This will help you visually understand the whys behind the statements.

Let’s dive into it!

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

What are Tools in LLMs and How Does Tool Calling Work

Large Language Models are stateless systems. This means that their knowledge is frozen at a training cutoff, so they have no familiarity with the world beyond them when training is completed. This architectural constraint is a deliberate design boundary that keeps inference deterministic and safe. However, it creates a profound usability gap: a model that cannot act on the “current version of the world” is only marginally useful. This is the exact gap that tools were built to close.

Defining Tools: A Formal Perspective

In the context of LLMs, a tool is a formally declared, schema-bound capability that the model can invoke. But pay attention here: LLMs can invoke tools, but cannot execute them. This distinction is essential because the model does not run code: it produces a structured intent, a precise, machine-readable declaration that it wishes a specific function to be called with specific arguments.

This means that execution authority remains entirely with the host application or orchestration layer. For this reason, a tool can be best defined as a typed contract between the model and its runtime environment: it specifies a function name, a set of typed input parameters, and a natural-language description that the model uses to reason about when and why to invoke it.

A canonical tool definition, as described in the OpenAI function-calling specification, looks structurally like the following example:

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Retrieve the current weather for a given city. Use this when the user asks about live weather conditions.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and country, e.g. 'Rome, Italy'"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"]
                    }
                },
                "required": ["location"]
            }
        }
    }
]

This schema is injected into the model’s context, typically as a structured system-level payload, before inference begins. So, the model does not “learn” tools: it reads them at runtime, the same way a human engineer reads an API specification before writing an integration.

AI can help you make sense of web data, but first you need to collect it. For teams building controlled, repeatable scraping workflows, anyIP provides the access layer with high-quality residential and mobile proxies worldwide.

Power your data workflow with anyIP →

The Tool Calling Lifecycle

You can use LLMs via chat and via APIs, depending on your preference and needs. Regardless of the use, LLMs have the possibility to call tools. This paragraph presents the lifecycle of explicitly calling tools via APIs, from the first to the last step.

Step #1: Context Injection

The developer serializes all available tool definitions into the model’s context alongside the system prompt and conversation history. Modern APIs expose a dedicated tools parameter for this purpose, keeping tool schemas structurally separated from the natural-language turns—though under the hood, they are always rendered into the token stream the model attends to.

Step #2: Model Reasoning and Tool Selection

During inference, the model performs implicit reasoning over the user’s intent and the available tool schemas. But note that this works the same as the next-token prediction mechanism driving all LLM outputs. The only difference is that now it is conditioned on a richer context that includes typed function signatures.

At this point, the model must determine: “Is a tool call warranted here? If so, which tool? What arguments should be passed?“.

This decision is governed entirely by the attention mechanism attending to both the user query and the tool descriptions. This is precisely why the description field in a tool schema is a signal that directly influences the model’s routing behavior.

Step #3: Structured Output Generation

When the model decides to invoke a tool, it does not generate a natural-language response. Instead, it emits a tool call object. This is a structured payload containing the resolved function name and a JSON-serialized argument map. In the OpenAI API format, this surfaces as a tool_calls array on the assistant message object, with finish_reason set to “tool_calls” rather than “stop”, signaling the host application that the turn is not complete:

{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {
      "id": "call_abc123",
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "arguments": "{\"location\": \"Rome, Italy\", \"unit\": \"celsius\"}"
      }
    }
  ]
}

Note that arguments is a JSON-encoded string, not a nested object. This is intentional because it decouples the model’s output format from the host’s parsing logic and makes partial streaming of arguments tractable.

Step #4: Host-side Execution

The orchestration layer intercepts the tool_calls payload, deserializes the arguments, validates them against the declared schema, and dispatches the call to the actual function implementation. This may be a REST API call, a database query, a local Python function, a subprocess, or any arbitrary computation, depending on the case. Note that the model has no visibility into this execution layer: it simply waits until the process is completed.

Step #5: Result Injection and Continuation

Once execution completes, the host appends the function result back into the conversation as a tool message, which is a distinct message role (“role”: “tool”) that carries the tool_call_id for correlation and the result payload as a string:

{
  "role": "tool",
  "tool_call_id": "call_abc123",
  "content": "{\"temperature\": 28, \"condition\": \"Sunny\", \"humidity\": 45}"
}

The full conversation history is then fed back into the model for a continuation inference pass. Only at this point does the model generate the final response in natural language. This way, the response is grounded in that the model could not have accessed it from its weights alone.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Web Search Tools Do Not Substitute Traditional Web Scraping: Here’s Why

One of the most common tools surfaced to LLMs is the web search tool, and its use is understandable: it is an obvious remedy to the knowledge cutoff problem. However, practitioners often make a critical architectural assumption: they treat the web search tool as a general-purpose solution for web-based information retrieval.

This assumption is a huge mistake, because web search and web scraping are different mechanisms operating at different layers of the information stack. This section discusses the reasons behind the fact that conflating the two is wrong.

What a Web Search Tool Actually Does

As previously introduced, when a model invokes a tool, the orchestration layer dispatches the call to the actual function implementation. When it comes to web search tools, what typically happens is that the underlying LLM’s implementation delegates to a search engine API.

Now, depending on the usage (aka, you are using the LLM via chat or via API), and also depending on the actual type of web search tool integrated into the LLM, after accessing the web, the tool can return one or more of the following:

• Page titles

• URLs

• Short snippets/descriptions (often 160–3000+ chars depending on the provider)

• Metadata (e.g., publication dates)

• The entire page content or rich excerpts, pre-formatted for LLM consumption (but this only happens for specific web search tools)

Now, the point that is interesting for scraping professionals is that the output of the web search tool feeds the LLM, and it responds to the user query using that output. This means that you can write a query to an LLM, and it can confidently hallucinate a wide response that is based only on the data retrieved in the URL and page title.

The point is that you actually have no clue about what has happened under the hood. For example, you may be using an external web search tool called via APIs, thinking you have more control over the process. But the tool may fail at fetching the whole page content because of anti-bots. So, it retrieves only metadata, page title, and URL, and the model hallucinates the response based on that.

Yes, this is very bad news…

The Semantic Query Constraint of Web Search

Another thing to consider is that web search tools operate in query space. This means that the model generates a natural-language or query string, the search API processes it, and the response is a ranked list of results (regardless of the actual access to the complete web content, which comes in a second step). This is basically a semantic approximation of the information that users need, mediated by the search engine’s interpretation of query intent. In other words: the result you obtain is basically a black box, because no one knows how search engines rank the results.

This has a direct consequence on information precision. When a model needs to answer a question that requires information from a specific document, a table on a specific page, a structured dataset at a known URL, or a real-time feed from a specific API, a search query is a lossy and unreliable retrieval mechanism. What happens in this case is that the model must express its information need as a query string, the search engine must interpret that string, rank the relevant URLs, and return snippets, and only then can the model reason over that output. At each step in this chain, there is information loss and potential for a mismatch between the actual information needed and what is returned.

Web scraping, in contrast, operates in URL space. The retrieval is deterministic given a target URL: the scraper issues a request to a specific resource and returns its content. End of the story.

This is the correct retrieval primitive for any workflow where the information source is known in advance or where the crawl path can be programmatically determined.

Check the TWSC YouTube Channel

Why Web Search Tools Cannot Serve as a RAG Retrieval Layer

This is perhaps the most consequential architectural confusion in modern LLM system design, and it deserves specific treatment. It is also highly tied to the fact that web search tools operate in the query space.

Retrieval-Augmented Generation, in its canonical form, is a two-phase architecture:

• In the offline phase, a corpus of documents is chunked, embedded into a dense vector space using an embedding model, and stored in a vector database.

• In the online phase, a query is embedded into the same vector space, a nearest-neighbor search retrieves the most semantically relevant chunks, and those chunks are injected into the model’s context window as grounding material.

The key property of this architecture is that the retrieval operates over a controlled, pre-indexed, owned corpus, and the retrieved content is the full, raw text of the original document chunks, not a summary of them.

Web search tools, as discussed above, operate on an entirely different substrate. So, there is no owned corpus, no embedding model under your control, no vector store you can inspect, audit, or fine-tune.

The HTML-to-Markdown Pipeline: The Correct Content Format for LLM Ingestion

Even setting aside all the preceding architectural arguments, there is a final, purely pragmatic reason why web scraping provides a qualitatively superior content retrieval primitive for LLM-based systems. Web scraping ****gives you the raw document, which you can transform into any downstream representation you need.

A web scraper that issues a request to a target URL receives the full HTTP response body. This raw HTML is the ground truth of the page’s content. From it, you can extract structured data via DOM traversal using libraries like BeautifulSoup or lxml, execute JavaScript to capture dynamically rendered content via headless browsers, parse semantic HTML elements with full structural fidelity, and apply custom filtering to strip navigation chrome, cookie banners, and boilerplate that would otherwise pollute the model’s context.

Another important point to consider is that raw HTML can be converted to Markdown. Markdown is the optimal serialization format for LLM context injection for a set of concrete reasons:

• It preserves document hierarchy.

• It encodes list and table structure without the verbose tag overhead of HTML.

• It represents code blocks with fenced syntax that models trained on large code corpora recognize and parse with high reliability.

• It strips rendering-irrelevant attributes and style information that consume context tokens without contributing to informational density.

• It is the format in which the vast majority of LLM training corpora are natively expressed.

For such reasons, a model reasoning over a Markdown-serialized document is, in a certain way, reasoning in its native register.

Need help with your scraping project?

Testing Web Search Tools

In this section, you will understand the whys behind what you read above. You will see the tests I performed and the results I obtained. Specifically, the tool calls will be performed via chat. This is because the (partial) results of what happens under the hood are easier to show, and these are useful for your understanding.

This section has two different testing sub-sections:

• In the first one, you’ll see what happens when prompting different LLMs, asking them to search for specific sources.

• In the second one, you’ll see what happens when you specifically ask an LLM to open a specific URL.

DISCLAIMER: Note that native tool calling (calling tools via the chat) is more probabilistic and less controllable than invoking them via APIs. This is because the exact internal mechanisms by which specific providers implement native tool invocation policies are not publicly documented. However, the examples still fit the needs of this article.

Testing Examples Type #1: Asking LLMs to Search for Consolidating Trends

Let’s see how different LLMs react when asked to search for specific sources.

Claude

Let’s start the tests by prompting Claude with the following:

Tell me about recent analyst reports and news articles about consolidation trends in the AI industry.

Here’s the chat image:

Below is what happens:

As you can see, Claude searches the web. And it does so with two different mechanisms, based on the prompt:

• The first one actually searches for AI industry consolidating trends. That is basically exactly what it has been prompted to do.

• The second one searches for AI startup acquisitions in June…sure, startup acquisitions can describe trends. But how can June be taken as a representation of the whole 2026, until now? This is basically an assumption that the model made itself under the hood.

Now, let’s try to slightly change the prompt like so:

Find recent analyst reports and news articles about consolidation trends in the AI industry.

Here’s what happens:

Now, the results about “AI industry consolidation trends” are the same, but the search and the results in the second box are completely different. And the prompt only changed from “tell me” to “find”…

So, as you can see, the model makes its own assumptions. And this influences the response, of course.

ChatGPT

ChatGPT will use different web sources, given the same prompts as the two above. Below is the result for the first prompt:

And then for the second prompt:

This means that, if you want to make a benchmark between ChatGPT’s and Claude’s responses, you will have to account for the fact that they use different sources. This is exactly the point stated previously: using the research engine is a black box that users can not manage.

Also, as the image shows, the results are not based on two different web searches. Actually, the model does not show if it uses one or more searches.

Perplexity

Perplexity’s approach is a mix between the ones from Claude and ChatGPT:

• It uses different sources to ground the responses.

• It uses different searches on the web. You can count 5 (the ones with the magnifying glass icons).

Below is what happens under the hood with the first prompt:

The following image is about the second prompt:

Testing Examples Type #2: Forcing Search Tools to Read URLs

Imagine a scenario where you insert a prompt, the search tool returns the list of URLs, and you want it to read them, but the model can not access it. This may happen for several reasons:

• The content is behind a paywall.

• The web search tool was not able to go beyond the anti-bot wall.

• There was an error during the request.

These cases can happen more than you think. For the sake of this tutorial, let’s get a target page beyond a paywall so that the result is surely inaccessible:

Claude

Let’s see what happens with Claude:

As expected, the web search tool can not access content behind a paywall. However, as you can see, it did get a response because it says: ”Based on other coverage of the same story, here’s what it’s about”.

Now the response it gave is not interesting, so I’m not showing it for brevity. What is interesting is trying to understand what it has done under the hood. Basically, it was able to understand from the URL that the article is about IBM and a profit warning. So it autonomously made a web search. Let’s unpack it:

The image shows exactly the kind of non-deterministic approach described previously. As you can see, the system:

• Tried to open the proposed URL, but failed at it.

• Autonomously tried a different approach, searching on the web, based on the metadata extracted from the URL.

• It gave the response based on the content it was able to “read” from the sources it got from the web search.

So here is an interesting thing that happened: the system told you that it was not able to access the specific URL and that it used “other coverage of the same story”, but it did not tell you how. So, how do you think you can get deterministic results?

ChatGPT and Perplexity

What is worth considering is that ChatGPT and Perplexity use similar but different approaches between them, and with respect to Claude. Below is the result for ChatGPT:

As you can see, ChatGPT does not tell you that it cannot access the target URL, unlike Claude. Also, its search mode is different from Claude’s, because it searches (also) for other content from the WSJ that can refer to the target one.

Below is Perplexity’s result:

As you can see, Perplexity produces a result that is similar to ChatGPT’s:

• It doesn’t tell you it can access the target URL.

• It searches for content (also) on other articles on the WSJ.

Overall, this demonstrates that the result you get is non-deterministic and depends on the model you use.

Architectural Guidance: When to Use Each Primitive

At this point you may be asking: ”So, when should I use web search tools?”. Or:” Should I use web search tools at all?”.

Good questions! The first answer is: “Yes, you should use them”. But let’s subdivide the case for the explanations on “when”.

Using Web Search Tools via Chat

I would argue that the use of web search tools via chat is “the canonical” one. This is basically because when you prompt a model, and it kind of understands that it should use the web search tool, it will. This is included in the “non-deterministic” things already presented.

However, you’ve also seen that there are cases when the web search fails. Regarding the reasons why this happens, a solution to improve web search via chat is using MCPs from scraping providers. Here at The Web Scraping Club, we already made the case with “How to Give Claude Real-Time Web Access With the Decodo MCP”.

Using Web Search Tools via APIs

Considering using web search tools via APIs, the correct engineering posture is to treat web search tools and web scrapers not as alternatives but as complementary primitives operating at different layers of a retrieval architecture:

• Web search tools are appropriate at the discovery layer: When the model needs to identify which URLs are relevant to a user’s query, a search tool is an efficient mechanism for surfacing candidates from the indexed web.

• Web scrapers are appropriate at the retrieval layer: Once a target URL is known, either because a search surfaced it, because it is statically configured, or because a crawl traversal discovered it, a scraper is the correct instrument for extracting full, structured, high-fidelity content from that resource.

Below is a schema that represents this complementarity:

For example, a production-grade retrieval pipeline for a domain-specific RAG system that needs to retrieve fresh and up-to-date data every once in a while will typically combine both:

• A search tool for open-domain candidate discovery.

• A scraper for full-document extraction from the identified URLs, a Markdown conversion step for context-optimal serialization, a chunking and embedding step for vector indexing, and a nearest-neighbor retrieval step for query-time grounding.

In such cases, the web search tool and the web scraper are each necessary components of this pipeline. Neither is sufficient alone, and treating either as a drop-in substitute for the other is a mistake.

As a final consideration, note that it makes sense to use web search tools on the discovery layer because of how they work under the hood. As covered above, the web search tool is basically a trigger for APIs that retrieve content ranked from search engines. This is basically what every major scraping provider has to offer in terms of SERP/search APIs. In other words, if you are already a customer of one of the most common scraping providers, instead of using the web search tool that your favourite LLM has, you can directly use your scraping provider’s SERP/search API.

Conclusion

In this article, you learned how tools work in LLMs. You’ve also seen the limitations of web search tools and their best use cases, using them via chat or via API.

Overall, for a deterministic and scientific approach, web scraping remains the best data retrieval layer.

So, let us know in the comments: are you using web search tools or web scraping for ingesting LLMs with recent data?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Everyone who scrapes a site protected by DataDome knows the basic theory. The tag reads the browser fingerprint, then it watches how you use the page, and sends the result to a server that gives you a score. You can read about that online almost everywhere, even on this Substack. This time I wanted to go a bit deeper and read what the tag really does. I used Manus for the first analysis of the code, and then I checked every claim using Claude.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

In THE LAB #114 I opened Kasada on nike.com and spent most of the article on its interpreter. Two custom virtual machines, 279 opcodes in total, and a decoder based on the clock, so the file stops working after fifteen hours.

DataDome is the other name you meet all the time, so I expected something similar. It’s different. The sensor is 124 KB of minified JavaScript, the method names are still readable, and the string table is plain base64. This difference is the most interesting thing I found, so at the end I put the two products side by side.

The target is leboncoin.fr, the French classifieds site. It runs DataDome JavaScript Tag 5.9.4.

How DataDome is installed on leboncoin

There are five different pieces, and when you say “DataDome blocked me” you should know which one you mean.

A remote component in front of the application answers a navigation with the real page or with a 403 that contains the challenge bootstrap. dd.leboncoin.fr/tags.js is the sensor. It’s hosted on S3 behind CloudFront, on a hostname that looks like part of leboncoin (the response headers say server: AmazonS3 and via: cloudfront.net). dd.leboncoin.fr/js/ receives the data the sensor collects. ct.captcha-delivery.com serves the two bootstraps, i.js for the device check and c.js for the CAPTCHA. geo.captcha-delivery.com serves the page the user sees.

The sensor is the piece I want to study, because it’s the only one where I can read the code. The server-side scoring is out of scope, for the simple reason that nobody outside DataDome can see it.

We wrote about DataDome many times, in THE LAB #2, in the cookie work of THE LAB #94 and in the agentic test of THE LAB #103. In THE LAB #106 we also saw which properties DataDome reads, from inside the browser engine, with camoufox-reverse. All those articles asked if we could pass, or what the browser layer shows, but this time the focus is on opening the tag file and seeing what’s inside.

When CAPTCHAs interrupt a workflow, reliable infrastructure matters. anyIP provides residential and mobile proxies with consistent sessions for smoother web-data collection.

Start Scraping with AnyIP

Method

I used two tools, Camoufox and Manus, and the article is the result of the two approaches. On one side, the runs and the report of Manus. On the other side, the check of every claim of that report, done by Claude on the bytes and on the wire.

I launched the first pass with Manus on Le Bon Coin, similarly to what I used for Kasada in #114. The challenge path and the fingerprint inventory in this article are its results, on several runs. I take these runs as they are. Manus worked with a stateful Chromium session, some direct curl requests and static analysis of the files served by the site. No external sources, no stealth tools.

Its outcome matrix has eleven samples. The home page loaded with a 200 and the tag loaded with it. A HEAD and a fetch GET to the same page, sent from that same session, both got a 403. Four curl requests to the home page and to the ad page, with and without a Chrome User-Agent, all got a 403 with a challenge bootstrap of 771 to 774 bytes. The User-Agent only changed the bootstrap type, rt=c or rt=i. The navigation to the ad page got a 403 and a device check. After the human check, the frame became a CAPTCHA, then “Access is temporarily restricted”, and the block followed the session back to the home page. The ad page never arrived.

Then Manus reviewed its own report, with three blind audits on separate parts of the evidence and one adversarial pass that tried to prove each claim false. When the evidence was not enough for a claim, it lowered the claim. The first one was if the CAPTCHA was ever accepted by the server. This is the material I started from. Camoufox was admitted where the Chromium of Manus was not, so the numbers from the wire below, the POST body, and the live x-dd-b header, come from my captures.

My collection was read-only. I didn’t replace native APIs, I didn’t modify requests, I didn’t replay tokens and I didn’t decrypt payloads. Three captures with the same profile and the same target, with cache busting between them, from a residential connection in Italy. The tag didn’t change between the Manus run and mine, so both of us read the same 123,989 bytes, SHA-256 16a5a027...3151.

Every claim has an evidence level, and there are only three:

• TABLE means the string is in the decoded string table and nothing more

• SOURCE means the identifier is in a place where the code can execute it

• WIRE means I saw it in a request or a response I captured

A string in the table has not been read by anything. A property that is read has not necessarily been sent. Keeping these three levels separate is most of the work in this kind of analysis, and I lost track of it more than once while reading the string table.

The claim table that puts all of this together is results/verification.md, available in the same directory of The Lab repository, available for paying users.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

What the tag sends

The sensor sends one POST to dd.leboncoin.fr/js/, from the main frame, with no query string. I have two of these bodies, 5,069 and 4,996 bytes of form-urlencoded data, and both have the same nine fields in the same order. This is the second one, as recorded by capture_hardblock_scope.py. The opaque values are replaced by their length and a hash prefix:

POST https://dd.leboncoin.fr/js/   frame: https://www.leboncoin.fr/   body: 4996 bytes
jspl           len 4698   sha256[:12] df708add1476
eventCounters  []
jsType         ch
cid            len 128    sha256[:12] db1e75da8bbc
ddk            len 30     sha256[:12] d1cd9afe92aa
Referer        https%3A%2F%2Fwww.leboncoin.fr%2F
request        %2F
responsePage   origin
ddv            5.9.4

Four of these fields are worth a look. ddv is 5.9.4, so the version is in the payload and not only in the file banner. jsType was ch, the challenge collection, while the initial one has a different value. responsePage was origin. eventCounters was [], empty, because a headless run with no real interaction has no behaviour data to send. That empty array is what my client looked like from the other side.

jspl is the real signal payload, 4,771 characters in one capture and 4,698 in the other, and it stays opaque. cid was 128 characters, the same length of the datadome cookie. The cookie is readable from JavaScript, with Secure and SameSite=Lax, on the domain .leboncoin.fr. ddk is 30 hex characters and it was identical in all captures, so it’s the client key of the site and it has nothing to do with the session.

There is also a diagnostic sample. When Math.random() <= .05 (line 2937 of the beautified tag), the tag serializes window.ddoptions in a field called opts and window.ddCaptchaOptions in xhr_opts. One request in twenty sends the DataDome configuration of the site back to DataDome.

Check the TWSC YouTube Channel

What the fingerprint is made of

jspl is opaque on the wire, but the code that builds it is only partly obfuscated, and this is where the Manus report is most useful. The report divides the collection in two classes, the environment fingerprint and the behaviour telemetry. For the fingerprint, it groups the probes in families. I keep the same grouping here, and I checked every name below in the decoded string table and in the beautified source with Claude. If I don’t say otherwise, these are TABLE level: the tag has the probe, but from the bytes I can’t show which branch runs it in a given session.

Read more

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

Oxycon 2026

The Oxycon 2026 starts in 8 days, and there’s a lot of beef on the plate. As in the past editions, it’s free and it’s online, and the recordings will be available only to registered attendees. If you don’t want to miss some great talks, you’d better register soon here.

This year I’m on a panel instead of giving a talk. It’s titled “The Adaptive Web: Turning Industry Challenges into Competitive Advantage”, and I’ll share the stage with Andreea Stroe from Adobe, Hocine Amrane from NielsenIQ, and Andrey Gourine from Limy AI. Forty minutes with people who look at web data from very different seats than mine, so I expect to learn more than I speak.

The full agenda has 10 sessions and 18 speakers. A few I’m curious about:

• Etienne Ellie from ScrapingBee on “Web Hydration: Loading Dynamic Content Without a Browser”. Getting data without rendering the page is a topic I keep coming back to in The Lab.

• “We Made Our Web Search Worse on Purpose” by Emilis Strimaitis from Hostinger, if only for the title.

• The panel “Is the Web Closing? What It Could Mean for Businesses, Journalists, and Researchers” with Carl Miller, Paul Bradshaw and Krishna Sood. The pay-per-crawl discussion I return to below is the same question seen from the other side of the fence.

• Tadas Gedgaudas from topYappers closing with “Best Open-Source Scraping Repositories of 2026”, which is only for live viewers.

As I did with the 2025 edition, I’ll publish a wrap-up right after the event.

FuelTurin

The second date is an event I’m organizing together with DataFuel, an Italian company that launched its brand and scraping API this year. On Wednesday, October 7, during the Wave by Vento week in Turin, we’re hosting a networking aperitivo for people who work with web data and AI.

Wave by Vento is the new name of Italian Tech Week. It runs from October 7 to 9, with great speakers (Jony Ive, Luca Ferrari, CEO of Bending Spoons, Nik Storonsky, CEO of Revolut, and others) on the main stage, and it brings many people from our industry to Turin on those days. A rooftop aperitivo felt like the simplest way to get around 40 of them in the same room: founders, web data experts, AI and agent builders.

The format is simple. You bring what you work on; we cover drinks and food. No pitches, no slides, no demos, no badge scanning. Registration is by approval, so tell us what you do when you sign up.

Register on Luma. Otium Rooftop, Turin, 19:00.

What the SERP API vendors did about Google’s goto links

On August 30 I published what I found about Google’s goto links, the change that replaced every destination URL in the search results with a google.com/goto?url= token. Two findings are worth repeating here, because they explain the answers that came from the vendors.

The comments under that article added a detail I couldn’t reproduce on my side. BigBry gets the go-to links in a human-controlled incognito window, with no CDP involved, and in Firefox when logged out. Michael Vederman drove Chrome and Edge with three different stacks and wrote that “it all boiled down to using a signed-in profile or not”, although after a few more hits Firefox gave him the wrappers logged in or not. In my runs, a private window never showed a wrapper, so the observations don’t line up perfectly. My reading is that the signed-in state weighs more than my two windows suggested, and that the classifier looks at more than one signal. If you run your own test, check the account state before you check the flags.

Resolving the tokens is cheap. A plain GET with no cookies, no proxy and no TLS impersonation returns a 302 with the destination in the Location header. I ran 1,200 resolutions from one home IP at concurrency 20 and got 110 requests per second sustained, with zero 429s. Tokens captured one day still resolve the next morning. Two traps remain: HEAD answers 200 without a Location header, so it looks like a success and gives you nothing, and the same result gets a new token on every impression, so there is no lookup table to build.

DataForSEO published on August 27 that it resolves the organic goto URLs for 99.99% of returned SERPs, with the remaining cases concentrated in AI Overviews and local packs. SerpApi announced on September 5 that direct URLs are back in its responses, and told PPC Land that the change “required updates to how Google result links were decoded and normalized”, without saying whether that means following the redirects or how much it costs them. In the same article, Johannes Beus from SISTRIX described the goto links as “a change requiring an added-cost workaround”.

So everyone who sells Google’s index reacted quickly to this update, and the impact on the SERP scraping ecosystem was minimal.

For this month, that’s it; see you in October with the next edition of the Crawl of Fame.

While its primary goal is to help web applications adapt their behavior based on the user’s hardware, the API can also serve as a new hardware signal for hardware fingerprinting.

So, is Chrome’s new CPU Performance API a real challenge for bot detection, or is it just another trend in the browser fingerprinting puzzle? In this post, I’ll cover everything you need to know about the API, how it works, and what it means for anti-scraping systems.

An Overview of Chrome’s CPU Performance API

Before diving into the implications of Chrome’s new CPU Performance API for automated bot detection, let me first present it to you.

What Is Chrome’s CPU Performance API?

Introduced in Chrome 152, the CPU Performance API is a new Web API that lets websites estimate how powerful a user’s CPU is without disclosing detailed hardware information.

The API assigns the device running Chrome to a performance tier. Websites can access that tier (represented as a small integer) through the navigator.cpuPerformance JavaScript property. For more information, refer to the CPU Performance API Explainer.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

How It Works

Instead of revealing the processor model, clock speed, number of cores, or other fingerprinting-sensitive details, navigator.cpuPerformance returns a small integer representing the device’s CPU performance tier. As of now, the specification defines four possible tiers:

• 1: Low-performance devices.

• 2: Entry-level to mid-range devices.

• 3: High-performance devices.

• 4: Top-tier devices.

The higher the number, the more computationally capable the device is.

Note: There’s also a special 0 tier, which indicates that Chrome couldn’t determine the device’s performance tier.

To test the CPU Performance API in your browser, run the following snippet in the Chrome DevTools Console:

if ("cpuPerformance" in navigator) {
  console.log(navigator.cpuPerformance); // 0, 1, 2, 3, or 4
}

The API is also designed to be future-proof. Rather than reclassifying existing devices, newer and more powerful hardware would be assigned new tiers (such as 5 or 6), ensuring that today’s devices continue to receive the same classification over time.

One important aspect is that the reported tier is static. In other words, it represents the device’s hardware class rather than its current CPU utilization. A powerful machine remains a tier-4 device even if it’s temporarily under heavy load or running on low battery.

Why Was It Introduced in the First Place?

The main goal behind Chrome’s CPU Performance API is to give web devs a quick, static, and reliable way to understand a device’s capabilities.

Before this API, websites that needed to adapt their experience based on hardware performance often had to run custom JavaScript or WebAssembly benchmarks. These approaches consume CPU cycles and can produce inconsistent results depending on the device’s current workload.

With the CPU Performance API, Chrome handles the classification internally and exposes a consolidated performance tier through navigator.cpuPerformance.

This becomes particularly useful as modern web apps increasingly run tasks that require a lot of computing power. Knowing a device’s performance tier allows web applications to make smarter decisions before launching expensive workloads.

For example, imagine a browser-based photo editor that uses AI-powered image enhancement. On a high-performance device, it could run the AI model locally for faster results and better privacy. On a lower-tier device, it could switch to a lighter model or delegate the processing to the server instead.

If an application also needs to react to changing system conditions, Chrome recommends combining the CPU Performance API with the Compute Pressure API, which enables you to observe the pressure of system resources (e.g., monitoring CPU usage).

The CPU Performance API answers the question, “How powerful is this device?”, while the Compute Pressure API answers, “How busy is the CPU right now?” Together, they allow applications to make both long-term and real-time performance decisions.

Only pay for data you actually get — one request, one credit, every time. No surcharge for JS rendering, no charge when a call gets blocked. Claim your free 10,000 requests with coupon code WEBSCR

Grab the deal!

Is the CPU Performance API Something New for Hardware Fingerprinting?

No, the CPU Performance API isn’t a revolution in browser and device fingerprinting. It’s merely a convenient API that offers an aggregated performance tier value.

In particular, there are already other APIs available for retrieving hardware-related information. Some are web standards, while others are Chromium-specific features that web developers (and, yes, also anti-bot solutions) can use to understand a device’s characteristics.

Explore these APIs!

WorkerNavigator

The WorkerNavigator interface exposes multiple properties that can reveal information about the user’s hardware.

One example is navigator.hardwareConcurrency, a read-only JavaScript property that returns the number of logical processors available to run threads on the user’s device.

Web devs commonly employ this property to optimize resource usage. For example, a web app could create a pool of Workers based on the number of available logical processors:

const cores = navigator.hardwareConcurrency;
console.log(`Detected ${cores} logical processors`);

A possible output could be:

Clearly, the same information can also be useful for hardware fingerprinting. While the number of logical processors alone isn’t enough to uniquely identify a device, it provides a signal about the hardware capabilities of the user’s machine

Note: Browsers may intentionally report a lower value than the actual number of logical processors. This helps reduce fingerprinting risks and provides a more realistic estimate of how many threads a web application can run efficiently in parallel.

Device Memory API

The Device Memory API is available through the aforementioned WorkerNavigator interface. Its navigator.deviceMemory property returns an approximate amount of RAM available on the user’s device:

const memory = navigator.deviceMemory;
console.log(`Approximate device memory: ${memory}GB`);

The Device Memory API intentionally reduces precision to limit fingerprinting risks. Rather than returning the exact amount of RAM, it reports a coarsened value based on powers of two and clamps it within browser-defined limits.

However, these limits have changed over time. Since Chrome 147, desktop Chrome can report 2, 4, 8, 16, or 32 GiB, while Chrome on Android reports 1, 2, 4, or 8 GiB. Consequently, a desktop with 12 GiB of RAM might report 8 or 16 GiB, depending on the browser’s approximation, whereas Android devices remain capped at 8 GiB in Chrome 147.

The reported value should therefore be treated as an approximate capability signal, not an exact measurement of physical RAM. For example, a desktop with a high logical-core count might report 16 or 32 GiB, but a lower value is not, by itself, evidence of spoofing.

WebGPU API

The WebGPU API gives web applications direct access to the user’s GPU for high-performance graphics and general-purpose computations.

It’s been developed as the successor to WebGL. The WebGPU API enables more advanced use cases, such as complex 3D rendering, GPU-accelerated simulations, and running machine learning workloads directly in the browser.

From a hardware fingerprinting perspective, WebGPU can expose information about the user’s graphics hardware through properties such as the GPU adapter details:

const adapter = await navigator.gpu.requestAdapter();

if (adapter) {
  console.log(adapter.info);
}

This type of information can become a valuable fingerprinting signal because GPU models and capabilities vary considerably across devices.

Summarizer API

The Summarizer API is an experimental Chromium-based browser API that allows websites to generate text summaries using a built-in, on-device AI model.

Before using the API, websites can call Summarizer.availability() to determine whether the device can run the local AI model:

const availability = await Summarizer.availability({
  expectedInputLanguages: ["en-US"],
  outputLanguage: "en-US",
});
console.log(availability);

The possible outputs are:

• downloadable: The browser supports the requested configuration, but it needs to download an AI model or additional fine-tuning data before it can run it.

• downloading: The browser supports the requested configuration, but an ongoing download must complete before the API becomes available.

• available: The browser already has everything required to run the requested configuration without any additional downloads.

• unavailable: The browser or device doesn’t support the requested configuration.

Behind the scenes, this classification depends on hardware requirements such as available RAM, VRAM, and CPU cores. For instance, devices with less than 16GB of RAM mightn’t support the API. As a result, the Summarizer API is also relevant from a hardware fingerprinting perspective.

Check the TWSC YouTube Channel

Is Chrome’s New CPU Performance Tier Fingerprinting an Issue for Web Scraping?

No, not necessarily.

Since navigator.cpuPerformance is exposed to JavaScript, it can be spoofed by definition. A browser automation tool (used for web scraping or automation) can modify the property’s value to report any performance tier it wants.

In detail, there are two main approaches to spoofing navigator.cpuPerformance.

The first is a JavaScript-level override. The idea is to let the automation framework inject a JS script before the page loads and redefine the property using Object.defineProperty():

Object.defineProperty(navigator, "cpuPerformance", {
  get: () => 4
});

Now, when the page’s fingerprinting scripts run, navigator.cpuPerformance will always return 4. That is the case even if the scraper is running on a server with much weaker hardware.

The main limitation of this method is that sophisticated bot detection systems (e.g., advanced WAFs) can usually spot JavaScript-level spoofing. That’s possible as native browser properties have well-defined behaviors (e.g., they’re read-only), whereas properties overridden with Object.defineProperty() may expose inconsistencies.

The second approach is to patch Chromium itself, typically by modifying the underlying C++ implementation that exposes navigator.cpuPerformance. In this case, the spoofed value is generated by the browser engine itself, making it virtually impossible to distinguish from a genuine browser.

The problem is that, for most scraping users, maintaining a custom Chromium build isn’t practical. The solution is to rely on existing stealth browser solutions. These include:

• Open-source projects such as Camoufox (and its forks), Patchright, and nodriver.

• Docker-based solutions like Kameleo Docker and rayobrowse.

• Cloud, premium solutions like Bright Data’s Browser API, Browserbase, and Oxylab’s Headless Browser.

These tools already implement low-level fingerprinting protections, allowing you to launch stealth browser sessions.

So, why all the fuss? If spoofing the API is relatively straightforward, why dedicate an entire article to it? Because, as is generally the case in our industry, it’s more complex than that…

Need help with your scraping project?

When the CPU Tier Becomes a Real Issue for Bot Detection

By itself, navigator.cpuPerformance isn’t a particularly interesting fingerprinting signal. After all, you can just spoof its value.

What anti-bot systems actually care about is consistency. Modern fingerprinting (at any level, but especially at the browser and hardware levels) rarely relies on a single signal. Quite the opposite, detection systems correlate dozens of hardware, browser, and behavioral signals to determine whether they describe the same device.

The CPU Performance API is another piece of that puzzle. For example, a browser claiming to be running on a high-end Tier 4 machine should probably also expose a reasonable number of logical CPU cores through navigator.hardwareConcurrency, enough RAM via navigator.deviceMemory, and a GPU consistent with a modern desktop. (That’s also why I decided to cover other APIs that contribute to hardware fingerprinting when writing this article.)

If those values don’t line up (e.g., a Tier 4 CPU paired with only two logical cores and 2GB of RAM), that inconsistency immediately becomes suspicious. Considering that most production-ready scrapers run on servers while spoofing multiple signals to appear as regular user sessions, missing even a single fingerprinting property, such as navigator.cpuPerformance, can be enough to trigger bot detection.

Ultimately, the CPU Performance API isn’t the real challenge. As is the case with browser fingerprinting, the hard part is making sure every exposed hardware signal tells the same believable story.

Conclusion

In this blog post, I walked you through what Chrome’s CPU Performance API is, why Chrome introduced it, how it works, and why navigator.cpuPerformance matters in the context of hardware fingerprinting for automated bot detection.

I hope you enjoyed the article and learned something new along the way. If you have any questions or want to share your thoughts, feel free to leave a comment below. Until next time!

FAQ

Which versions of Chrome support the CPU Performance API?

The CPU Performance API was introduced in Chrome 152. It’s expected to become available with the Chrome 152 stable release. Other Chromium-based browsers may adopt it in future versions.

Does the new Chrome CPU Performance API introduce privacy issues?

As covered in the official specification, Chrome’s navigator.cpuPerformance property intentionally returns only broad performance tiers instead of detailed CPU information in order to reduce potential privacy implications.

Can I avoid navigator.cpuPerformance detection by automating Firefox instead of Chrome?

As of this writing, yes. The CPU Performance API is a Chromium-specific feature and isn’t available in Firefox. However, Gecko-based browsers could implement something similar in the future.

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Their core offer includes pre-compiled datasets, SERP APIs, Web Search APIs, and a Web Unlocker API (which is coming soon).

Let me introduce you to the company and show you how easy it is to use the services they offer with a couple of hands-on tutorials.

Let’s go for it!

What is AntsData?

AntsData is a platform that collects, cleans, and delivers data from across the internet. Thanks to its infrastructure, it delivers data ready for AI workflows and analytics:

Its solutions include:

• Google SERP extraction at scale: AntsData SERP API allows you to retrieve structured search results from Google without managing proxies, solving CAPTCHAs, or building your custom scraping infrastructure. Use Python, curl, JavaScript, or even their playground and get the data without infrastructure overhead. By using this API, you can extract all major SERP features, including organic results, paid ads, featured snippets, knowledge panels, and more.

• Web Scraper APIs: Get clean data in JSON from YouTube, X, Amazon, and more websites. These are the best solutions if you are doing sentiment analysis from social media posts, price tracking, market research scraping, or if you are building RAG systems.

• AI-ready datasets: For data enrichment cases, or if you don’t have the right coding skills, AntsData provides ready-to-use datasets.

• Custom datasets: If the analytics-ready datasets are not enough for you, you can ask AntsData to create custom datasets for you. By defining the scraping scenario that fits your needs, the company assists you in delivering the right data for your specific case.

On the side of pricing, the company offers a pretty wide range of options, where the interesting one is probably the pay-as-you-go plan with no minimum commitment:

How to Use AntsData to Retrieve Web Data From Google SERP

Let’s start by retrieving web data using AntsData Google SERP API.

Introduction to the SERP API

AntsData SERP endpoints are the kind of resources you use for “ask and get an answer” search. In other words, you submit a search query, the server blocks until done, and returns the results. This is a synchronous mechanism, which is different from the async model that works like this: “submit → poll → fetch”. What’s particularly important to remember is that:

• The synchronous search is the ideal solution for small jobs.

• Large jobs that exceed the sync window, instead, are more suitable for pollable, asynchronous runs.

Here’s a schema that summarizes these two different approaches:

This sync API supports the following endpoints, allowing you to retrieve different kinds of results from Google searches:

In this tutorial, I’ll use the search endpoint. Below are its parameters:

Let’s get into the actual tutorial!

Prerequisites

To reproduce this tutorial, you need the following:

1. A valid AntsData account.

2. Python 3.10+ installed on your machine.

Below is the dashboard you’ll see after creating your account:

Note that the company provides you with a free trial option, offering 5$ in credits after registration.

Use the SERP API Via the Playground

To use the Google SERP API, navigate to the Web Access section and click on Playground:

Set up your request parameters directly in the playground:

For this tutorial, the parameters I set up are the following:

• Results type: web results. This makes the platform return only web data from Google results.

• Research query: best running shoes. This is the query for the search engine.

• Search location: United States.

• Language: ****en

• Google domain: google.com

• Page: 2

• Result limits: 10

NOTE: To learn more about all the parameters and how to set up their values, read AntsData developers’ technical documentation.

When ready, click on Run request, and you’ll see the results directly in the dashboard:

Below is a partial JSON from the retrieved data:

[
  {
    "organicResults": {
      "url": "https://www.google.com/search?q=best+running+shoes\u0026uule=w+CAIQICINVW5pdGVkIFN0YXRlcw\u0026gl=us\u0026start=10\u0026pws=0\u0026hl=en",
      "organic": [
        {
          "pos": 2,
          "pos_overall": 2,
          "title": "Top Rated Running Shoes | REI Co-op",
          "favicon_text": "REI",
          "url_shown": "https://www.rei.com › top-rated-running-shoes",
          "url": "https://www.rei.com/s/top-rated-running-shoes",
          "desc": "Shop for Top Rated Running Shoes at REI - Browse our extensive selection of trusted outdoor brands and high-quality recreation gear."
        },
        {
          "pos": 3,
          "pos_overall": 3,
          "title": "Best Running Shoes For Men | On United States",
          "favicon_text": "on.com",
          "url_shown": "https://www.on.com › Shop",
          "url": "https://www.on.com/en-us/shop/o/best-running-shoes-for-men?srsltid=AfmBOoqlJHi39Rz0hqhbsQbcn-zIx1pvDbxx8lL7pqB3Qenwkr84hnFc",
          "desc": "Best sellers · Cloudswift 4 Road Running Shoe in Wolf/Sand · Cloudrunner 2 Waterproof Road Running Shoe in Sand/Dew · Cloudsurfer Max Wide Road Running Shoe in ...Read more"
        },

...< Omitted for brevity>...

  }
]

In the Overview tab, you can download the JSON file with the complete response data:

At this point, you may want to know how much this call cost. To discover it, go to Billing:

To talk numbers, I spent 0.002$ for 8 results.

Use the SERP API in Python

If you want to incorporate the data into an AI workflow, you can call the endpoints in Python, curl, and Node.js. The playground provides you with code you can directly use in your local scripts, after setting up the parameters.

If you want to use a Python script, change the autogenerated code from the playground so that the script can save the results in a JSON file.

Before changing the code, first create a folder on your local machine:

mkdir shoes-scraping

Navigate into it:

cd shoes-scraping

Create a virtual environment:

python -m venv venv

Activate the virtual environment:

source venv/bin/activate # On Windows: ./venv/scripts/activate

In the activated virtual environment, install the requests library:

pip install requests

Create a Python file called shoes-scraping.py and write the following code in it:

import requests
import json

url = "https://api.antsdata.com/v1/serp/google/search"
headers = {
    "Authorization": "Bearer <YOUR-API-TOKEN>",
    "Content-Type": "application/json",
}
payload = {
  "recordLimit": 10,
  "dataFormat": "json",
  "searchType": "web",
  "query": "best running shoes",
  "geo_location": "United States",
  "host_language": "en",
  "domain": "com",
  "page": 2
}

resp = requests.request("POST", url, json=payload, headers=headers)

# Parse the response text
data = resp.json()

# Save results as a JSON file
with open("running-shoes.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=4)

print("Results saved to running-shoes.json")

And, of course, you will obtain the same result as before.

Check the TWSC YouTube Channel

How to Use AntsData to Retrieve Data from Amazon

In this section, you’ll discover how to use the Amazon Web Search API. But before going directly into a hands-on tutorial, let’s introduce this API.

Introduction to the Web Search APIs

All AntsData Web Search APIs use an asynchronous mechanism when retrieving data from the web. Specifically speaking, the Amazon Web Search API supports two endpoints:

In the upcoming tutorial section, you’ll discover how to use the sellers endpoint.

Use the Amazon Web Search API Via the Playground

First of all, navigate to the Amazon Web Scraper API in the AntsData dashboard:

In the platform, you can insert multiple target URLs. For this tutorial, I defined a couple. One from Amazon best sellers like electronic accessories for cameras and photos:

The other from Amazon bestsellers electronics cables for cameras and photo products:

Set up the environment directly in the playground. Select Discover by category URLs in Amazon Best Sellers and attach the target URLs in the Inputs section:

Then click on Run manually. You will automatically be redirected to the Run section, where you’ll see that your job is queued:

When the process is complete, you can download the JSON file and see how much it costed:

Below is a partial result from the JSON:

[
  {
    "url": "https://www.amazon.com/Best-Sellers-Electronics-Camera-Photo-Accessories/zgbs/electronics/172435/ref=zg_bs_nav_electronics_2_281407",
    "depth": 0,
    "name": "Best Sellers in Camera \u0026 Photo Accessories"
  },
  {
    "url": "https://www.amazon.com/gp/bestsellers/electronics/502394",
    "depth": 1,
    "name": "Best Sellers in Camera \u0026 Photo Products"
  },
  {
    "url": "https://www.amazon.com/gp/bestsellers/electronics/14015081",
    "depth": 1,
    "name": "Best Sellers in Camera \u0026 Photo Cables \u0026 Cords"
  },
  {
    "url": "https://www.amazon.com/gp/bestsellers/electronics/505106",
    "depth": 1,
    "name": "Best Sellers in Camcorder Accessories"
  },

<...OMITTED FOR BREVITY>

  {
    "url": "https://www.amazon.com/gp/bestsellers/electronics/2243862011",
    "depth": 4,
    "name": "Best Sellers in Laptop Cooling Pads \u0026 External Fans"
  }
]

Note that AntsData charges 0.55$ per 1000 results with its Web Scraper APIs. In this case, I spent 0.55$ for a total of 82 entries retrieved.

Need help with your scraping project?

AntsData Pros and Cons

Overall, here’s a summary of the pros and cons from my tests.

👍 Pros:

• The APIs have good and comprehensive documentation. The parameters are well defined, and responses are clearly explained, even in the case when the server doesn’t return a 200.

• The platform is intuitive to use and to navigate, even for junior engineers.

• The playground works well and reports code that can actually be incorporated into your workflows as is or with just minor adjustments.

👎 Cons:

• The company claims that developers can use their MCP with the Web Scraper APIs. I tested it and, unfortunately, it doesn’t work. Specifically, the target URL returns a 404. Of course, that could be a temporary error; however, that was the result at the time I tested it.

• The offer is limited in terms of datasets and available Web Scraper APIs, compared to similarly sized competitors.

Conclusion

AntsData positions itself as a middle-ground provider in the crowded web scraping market, focusing on a friction point for engineers: speed to delivery. For teams building RAG pipelines or training LLMs, the “AI-ready” nature of their JSON output significantly reduces the time spent in the data-cleaning purgatory.

However, it isn’t without its growing opportunities. The limited library of specialized APIs and the current instability of their MCP integration suggest that AntsData is still in its scaling phase.

So, let’s discuss in the comments: did you already know AntsData? What’s your experience with them?

In the same article, Derek Perkins at Nozzle measured what this means in practice. If you want to resolve all the links for a five-page ranking, you need between 500 and 1,000 requests. You can’t decode these links yourself, and HEAD requests won’t give you the answer. The only way to know where each result points is to ask Google for every single link.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

Most people talked about this as if it were a typical rollout, something Google would turn on for everyone over time. That’s why Barry Schwartz checked for it on his own machine and didn’t see anything, while Nozzle saw it almost everywhere.

Both were right in their own way since this isn’t a standard rollout. It’s Google responding to something.

If you see it this way, the cost story isn’t as dramatic as it sounds. I’ll show you the numbers, but to sum up: those 500 to 1,000 requests are real, they’re cheap, and honestly, they’re not the real problem.

Two windows, one variable

Here’s an experiment you can try yourself. It only takes about four minutes.

Start by opening Chrome with a fresh profile and search for something commercial (I used “best running shoes”). In the console, count how many result links go directly to the destination and how many use the goto wrapper, using this script.

(()=>{const f=()=>{const a=[...document.querySelectorAll('#rso a[href], #search a[href]')];const h=x=>x.getAttribute('href')||'';return `goto=${a.filter(x=>h(x).includes('/goto?')).length} direct=${a.filter(x=>/^https?:\/\//.test(h(x))&&!/google\.|gstatic/.test(h(x))).length} total=${a.length}`};console.log('t0',f());setTimeout(()=>console.log('t+5s',f()),5000);setTimeout(()=>console.log('t+20s',f()),20000);})()

Then, open a second Chrome window, again with a fresh profile and the same connection, but this time add the --remote-debugging-port flag. Don’t attach anything to that port and don’t automate anything. Just type the same query by hand.

window                                  goto  direct  total
fresh profile, no debug port               0     213    215
fresh profile, --remote-debugging-port    96       0    106
fresh profile, --remote-debugging-port    83       0     93

HTTP/3 and QUIC run on UDP, so the proxy layer has to carry it too. anyIP gives compatible clients production-ready SOCKS5 UDP ASSOCIATE across its residential and mobile network for your SERP scraping.

Explore anyIP UDP Proxies →

Everything else stays the same: same machine, same home connection, same query, just a few minutes apart. Without the flag, you see 213 destination URLs in the HTML, just like before. With the flag, you get zero direct URLs and 96 wrapped links instead.

Both results are stable. I checked each page at load, after five seconds, and after twenty seconds, and the counts never changed. So it’s not JavaScript changing the page later. The difference comes from what Google sends you right from the start.

I ran nine more tests using a stealth browser with full automation, and every time I got zero direct URLs, always wrapped. But when I browse normally, logged in or in a private window, I never see any wrappers at all.

So, from my little experiment, it seems that Google detects when I’m using an automated solution and serves me different results. The key factor for showing URLs or not is whether Google thinks it’s talking to a human or a script.

What this actually means

If you look at the reports with this in mind, it all makes sense. Schwartz didn’t see the behavior because he was just using his browser. Nozzle saw it everywhere because they’re a rank tracker, and Google had already classified their traffic. What people called a ‘bucketed rollout’ was really just two different groups seeing two different things.

This also changes what those 500 to 1,000 requests really mean. That’s not the new baseline cost for scraping Google. It’s what you pay after Google has recognized you. If you haven’t been flagged, the addresses are still in the markup, ready to grab for free.

For me, this is more interesting than just a link format change. Blocking is obvious: you’re either in or out, and you know it.

Here, Google still gives you results, the page loads, your parser can still find titles, positions, and snippets. Everything works. The only thing missing is that one field, and now you have to pay a network round trip for each row to get it.

There’s no warning when this happens. A pipeline that worked fine in June still returns HTTP 200 and well-formed HTML today, but the data quality quietly drops in the meantime.

What the wrapped page contains

Ten captures on the treatment side, across two queries, nine from a stealth browser and one read over CDP from a page a person opened by hand:

in-result anchors : 408
goto wrappers     : 339
direct URLs       :   0
old /url? wrappers:   0

In every capture, I found zero direct destinations. The old /url?q= wrapper, which at least left the address in plain text, is gone too. Now, every organic link, every video timestamp, and every ‘translate this page’ link is wrapped. They all have a ping attribute.

The difference between the number of anchors and wrappers comes from Google’s own navigation. For example, on the CRM software pricing page, 18 out of 38 in-result anchors are related searches, and three are ad links. All 17 external destinations are wrapped.

The token is protobuf, and that doesn’t help you

One question I had was: can the token be reverted to the URL? Of course, not.

Every token starts with CAES. Base64 decoded, that’s 08 01 12: protobuf field 1 as a varint set to 1, then field 2 as a length-delimited blob. A field walk over the decoded bytes confirms it.

field 1, wire 0, value 1
field 2, wire 2, len 99..119

The blob inside field 2 is where the destination should be, and it isn’t there. Every one of the 183 distinct tokens we collected opens with the same five bytes, 01 EB 3B 30 15, and everything after is high entropy noise. We measured 6.27 to 6.44 bits per byte over roughly a hundred bytes, which is what you get on random or encrypted data. Base64, base64url, percent decoding first, gzip and zlib on the result, none of them produce a single printable URL fragment.

What you see here is a constant header followed by an opaque payload, just like you’d expect from a key identifier and ciphertext. You can’t decode this yourself because you don’t have the key. Reversing an encoding is one thing, but getting a key from Google is a whole different story. So, the only way to get the destination is to let Google look it up for you, one request at a time.

Check the TWSC YouTube Channel

HEAD is not refused, which is worse

Another idea could be to use HEAD requests, so that in cases of redirects, the Location header contains the final URL. This could be a faster way to check which URL the token corresponds to.

What we measured from our test is annoying instead. Across 50 wrapped links resolved from the home line with curl_cffi and redirects disabled:

HEAD  ->  200 on all 50, Location header present on 0
GET   ->  302 on all 50, Location header present on 50
median GET latency  : 59 ms
mean bytes per GET  : 1068 B (headers plus body, redirect not followed)

HEAD requests aren’t blocked. You get a 200 response, but no redirect. If you just log the requests, you’ll see fifty successes but get nothing useful, and you’ll have to figure out why. The usual HTTP cache path doesn’t fail with an error. It just looks like it worked, which can waste a lot of time before you realize what’s going on.

GET requests work as expected: one per link, you get a 302 with a Location header, and you have the destination. I tried fifty of these from a home IP and didn’t hit any rate limits, so the throttle is set higher than that.

The mapping is not stable, so there is no lookup table

The last mitigation idea I had was to create a lookup table between tokens and final URLs. We compared the token attached to each result title across all captures. That covers three reloads inside one accepted session and several entirely separate browser sessions, hours apart, which is a harder test than reloading. Forty-six results appeared in more than one capture.

results seen in more than one capture : 46
token identical                       : 0
token changed                         : 46

None of the tokens are stable. The same link to the same page on the same query gets a new token every time. The CAES and 01 EB 3B 30 15 prefix stays the same, but the rest changes completely. In one case, I saw a result with 24 different tokens across my tests.

This answers a question that came up in the original discussion. You can’t build a lookup table and reuse it, because there’s nothing stable to key it on. The payload is tied to the impression, not the destination, so you pay the full resolution cost every time. I tried to find a caching trick here, but there isn’t one.

Those reloads showed me something else. They were automated navigations using a script over CDP, but the session was started by a human. Google allowed all of them. Once Google accepts a session, it keeps working even if you automate it. The real check happens when you submit the query, not after.

Where the destination leaks anyway

In three out of ten captures, I actually got some destinations in the clear, and I doubt that’s intentional.

When Google offers to translate a result, the “translate this page” link is usually a wrapper, but sometimes it’s a plain translate.google.com/translate?u= with the target sitting in the clear:

https://www.salesforce.com/eu/crm/pricing/
https://www.zoho.com/en-us/crm/zohocrm-pricing.html
https://www.creatio.com/glossary/crm-pricing

I checked if those were the same destinations you’d otherwise have to resolve. Looking at the first eight wrapped links and resolving them, six were already available for free in a translate link. The other two were Italian results, which Google didn’t offer to translate.

This happened in three out of ten captures, using the same queries and the same browser. I couldn’t figure out what triggers it. I think of it as a discount that sometimes shows up, so always check for it before you spend a request.

Need help with your scraping project?

What’s the cost of this modification?

On a wrapped page, I saw between 59 and 113 wrapper anchors, depending on how many results there were. After deduplication, that’s 59 unique tokens on my reference page. For a five-page ranking, that’s 565 requests if you resolve every anchor, or 295 after deduplication. The 500-1,000 range makes sense if you just resolve everything, which is probably what most people do.

The good news is that solving these tokens actually takes almost nothing. I tried resolving the same links four different ways: using curl_cffi with full Chrome TLS impersonation, and with plain requests, both with and without a Referer header. No cookies, no session, no proxy involved.

requests, no referer      12/12 resolved, 302 + Location, median 124 ms, 1269 B
requests, with referer    12/12 resolved, 302 + Location, median 137 ms, 1268 B
curl_cffi, no referer     12/12 resolved, 302 + Location, median 115 ms, 1479 B
curl_cffi, with referer   12/12 resolved, 302 + Location, median 123 ms, 1479 B

Plain requests work just fine. The stealth setup you needed to get the page isn’t needed to resolve the links. You don’t need a residential proxy, cookies, or even a referer, at least at this small scale.

Tokens also stay valid. I captured links one day and resolved all 12 of them the next morning, so you don’t have to resolve them right away. You can collect now and resolve later, even in bulk.

Now, about throughput. This is where people expected to hit a wall. I ran 1,200 resolutions from a regular IP using a plain client, increasing concurrency as I went:

concurrency  1:  60/60 resolved,  7.8 req/s
concurrency  5:  60/60 resolved, 35.1 req/s
concurrency 10:  60/60 resolved, 56.9 req/s
concurrency 20: 200/200 per chunk, six chunks, 110 req/s sustained

A tenth of a second, maybe two. In a multi-second answer, that’s real but manageable.

So, if the costs for solving these tokens are minimal, why did Google bother to put this feature in place?

The answer is that Google is now a synchronous dependency inside a competitor’s live product. Every user question that reaches Google’s index now needs a set of real-time callbacks to google.com before the answering system can read a single page.

That gives Google three things it didn’t have before.

First, a throttle it controls, where the visible symptom is someone else’s product getting slow.

Second, a live view of how many results each consumer dereferences and when, which is a decent proxy for their query volume.

Third, the machinery to apply both selectively, since the classifier that decides who gets wrapped markup is the same one that could decide who gets rate-limited.

This means more Google control over a certain type of third-party services, but not its direct competitors in the LLM space.

Claude’s web search appears to run on Brave Search. Anthropic has never said so publicly, but Brave Search has been on its published subprocessor list since March 2025 and is still there in July 2026, the search function carries a BraveSearchParams parameter, and the citations match what Brave returns for the same query. ChatGPT is more mixed. It launched on Bing, then built its own crawler and index, and now runs a blended stack with Bing still contributing alongside third-party providers.

Neither of them reads a Google results page, so they’re not touched by this deploy.

That narrows the target a lot. The companies with real exposure are the ones whose whole product is Google’s index resold with low latency, the SERP API vendors that sit between Google and everyone building on top of search. They can’t route around this like an assistant with its own index, because Google’s index is what they sell. Their customers want results in a second or two, which puts the resolution stage inside a latency budget set by someone else.

The most prominent vendor in the reachable group is the one Google currently has in court.

As a Google spokesperson said to Seroundtable.com: “We have a long history of deploying technical measures against evolving forms of abuse, and we regularly take steps to protect our services and users.” No target, no timetable, no documentation, and no acknowledgment that two clients asking the same question get different pages.

None of this is a rule anyone has to enforce, and that’s what makes it durable. A block can be litigated, and Google already lost once, on the grounds that reading a page counts as circumvention. Serving valid HTML with one field removed to clients you have classified isn’t a refusal of service, and there’s nothing to appeal against.

The cost for this particular field is actually small, and that’s important to keep in mind. What matters isn’t the 295 requests. It’s that Google can now separate the two groups well enough to send them different markup, quietly, with no announcement and no error in your logs. The field it removed this time is cheap to buy back with a GET. Nothing about the mechanism says the next one will be.

That program has gotten harder to look at. Kasada doesn’t ship obfuscated JavaScript anymore, at least not in the sense that a beautifier helps you. It ships a custom virtual machine and a blob of bytecode for it. You can pretty-print the file all day, and you’ll still be reading an interpreter loop, not logic. So I stopped asking how to get past it. I wanted to know what it costs to open one of these things and what you’re allowed to claim once you have one.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

To find out, I pointed an autonomous agent at Kasada on nike.com. What I wanted out of it, mostly, was the list. Which values does this thing actually reach for when it builds your fingerprint? Not a bypass, not a token generator, just the inventory of what it reads off your browser and how solid the evidence is for each entry.

That list came back at 412 entries, and it lives in the code rather than in this article. You’ll find it in fingerprint_inventory.csv, with the category, the reference count, and the byte positions for every one.

Pasting a 412-row CSV into a newsletter helps nobody, so what I’ll do here is walk the shape of it and spend most of the time on how you get to a list like that without fooling yourself. As always, all the code and the results will be available on our GitHub repository, available to paid readers under the folder 114.KASADA-NIKE.

Everything below is the client side of j-1.2.726 as observed on August 26, 2026, plus the parts I re-ran myself, because I wasn’t going to publish somebody else’s numbers on trust.

The tools

Manus AI did the collection and the first pass of analysis. It’s an agent that pairs a language model with a sandboxed Linux VM and a Chromium browser, so it can fetch assets, run Node and Python, drive a page, and iterate on its own output inside the sandbox. We used it the same way in THE LAB #108 against Akamai on Net-a-Porter. The reason it gets this class of work rather than a chat assistant is the guardrail gap. Ask a heavily guarded model to deobfuscate a production anti-bot sensor and enumerate the signals it scores, and you hit a wall.

Node 22 performs deobfuscation, and the important point is that it never executes anything Kasada served. We reimplemented the decoders from the beautified source and ran them over the captured bytes, so nothing from nike.com was ever evaluated.

The Chrome DevTools Protocol side is deliberately boring. It reads browser APIs, resource timing, and deterministic canvas, WebGL and audio probes, and that’s all it does. No hooks, no replaced natives, no Proxies over DOM APIs. Python 3.11 from our shared venv runs the comparison afterward.

How Kasada is laid out on nike.com

Everything comes off one UUID path:

/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/

The page loads p.js. That script fires an mfc request and creates two iframe navigations to an fp endpoint.

The new anti-bot solution - Toughest walls, lowest price. Claim your free 10 000 requests with coupon code WSCLUB

Claim your credits

That script fires an mfc request and creates two iframe navigations to an fp endpoint. The fp response is a tiny HTML document that bootstraps the real work, and here it is as captured, with session values stripped:

<!DOCTYPE html><html><head></head><body>
<script>
window.KPSDK={};
KPSDK.now=typeof performance!=='undefined'&&performance.now
  ? performance.now.bind(performance)
  : Date.now.bind(Date);
KPSDK.start=KPSDK.now();
window.parent.postMessage('KPSDK:MC:<redacted>', '*');
</script>
<script src="/149e9513-.../2d206a39-.../ips.js?KP_UIDz=<redacted>&x-kpsdk-v=j-1.2.726&x-kpsdk-im=<redacted>"></script>
</body></html>

Four things are worth pulling out of those nine lines. window.KPSDK gets created here, not in p.js. The clock is pinned to performance.now bound to performance, with Date.now as fallback, and KPSDK.start is stamped immediately, so everything downstream measures against that origin. The iframe communicates with the parent via postMessage using a KPSDK:MC: prefix. And ips.js gets loaded with three query parameters, one of which is the SDK version in cleartext.

So there are four layers here, and I should say which one I’m after. The loader and its integrity checks live in p.js. The fingerprinting program lives in ips.js. Then there’s whatever transport carries the result back, and behind that the server-side decision.

ips.js is the one I care about, because that’s where the browser values get read. p.js comes along for the ride, since you can’t reach ips.js without going through it. The transport we’ll get to, and the answer is unsatisfying. The server-side scoring is out of scope for the boring reason that nobody outside Kasada can see it, and I’d rather say that than dress a guess up as a finding.

The rules we set before collecting anything

This part is boring and it’s the reason the rest is usable.

The collection was read-only by construction. No native API replacement, no getters or Proxies added, no scripts or requests modified, no payload decryption, no token reuse. Five loads of https://www.nike.com/men, same profile, same session, cache-busting on each, with the same snapshot script every time. Arrays and keys that may return in arbitrary order are sorted before hashing. API errors are recorded as errors rather than being replaced with an invented value. Raw tokens never make it into the artifacts, only length, alphabet class and SHA-256.

More important, every claim carries an evidence level:

The census below lives at B and D. Level E was never reached, and level F was never attempted against the live service. If you’ve read anti-bot writeups that announce “83 signals collected and transmitted”, the number almost always comes from level A. A string sitting in a table hasn’t been read by anything, and a property that does get read hasn’t necessarily been sent anywhere.

Check the TWSC YouTube Channel

Two decoders, and one of them watches the clock

Both assets hide their actual content in a string blob that is converted into an integer array. The blobs are large. p.js was 163,120 bytes with a main blob of 136,736 characters; ips.js was 655,603 bytes with a main blob of 603,415.

The p.js decoder is a variable-radix varint and nothing else. Alphabet of 72 characters, radix 48, no seed, no checksum. It produces 74,029 integers.

ips.js keeps the same varint base and wraps it in a time lock. Alphabet of 71, radix 46, and a correction seed derived from the wall clock:

Math.round(Date.now() / 18000081) * 11

Each decoded integer has a decimal digit of that seed subtracted from it, cycling through the digits and starting at an index equal to the sum of those digits. After 14 values, the decoder checks the running result against the constant 1611118298. If the checksum misses, that seed was wrong. Three candidates get tried: the current window and the two neighbors, and during the capture, the seed that worked was 1,092,509. With the right seed, you get 305,595 integers.

I re-ran both decodes on my own machine, out of the archived assets, and they came back identical:

p.js    74,029 integers   string table 6,145 chars   sha256 6ad4ec90...   86 opcodes
ips.js 305,595 integers   string table 38,946 chars  sha256 c47829...    193 opcodes

Now, 18000081 milliseconds is five hours. Three candidate windows mean that a captured ips.js only decodes within a 15-hour band; after that, it’s a permanent blob. That’s the kind of claim that’s easy to assert and annoying to prove, because the obvious way to test it is to wait. So instead of waiting, we walked a synthetic clock past the decoder and watched where it broke:

clock -10h  window seed 1092498  decoded 305595 integers with seed 1092509
clock  -5h  window seed 1092509  decoded 305595 integers with seed 1092509
clock +  0h  window seed 1092520  decoded 305595 integers with seed 1092509
clock +  5h  window seed 1092531  DECODE FAILED (checksum mismatch on all 3 candidates)
clock + 10h  window seed 1092542  DECODE FAILED
clock + 24h  window seed 1092575  DECODE FAILED
clock + 72h  window seed 1092685  DECODE FAILED

asset seed        1092509 (window 99319)
band opens        2026-08-26T05:44:04.717Z
band closes       2026-08-26T20:44:04.960Z
band width        15.00 hours
fetched at        2026-08-26T15:30:09.000Z
already spent     9.77 hours of the band
usable after that 5.23 hours

Need help with your scraping project?

I ran that at 20:16 UTC on the day of the capture, with about half an hour to spare. Here’s the part that took me a second to appreciate. The validity band is exactly fifteen hours wide, three windows of five hours, and it’s anchored to the window the asset was minted in. It is not anchored to when you fetched it.

Nike handed us that ips.js at 15:30 UTC, which was 9 hours and 46 minutes into its own band. So we didn’t get fifteen hours to work with; we got five hours and fourteen minutes. Fetch the same file an hour and a quarter later in its cycle, and you’d be down to four. Fetch it near the end of the band and you’d get minutes.

You have no way to know in advance where in that band your download lands, because the asset doesn’t tell you when it was minted. You find out by decoding it and reading which of the three candidate seeds came back valid. That’s what the seed field in the decoder output is for, and it’s the first thing I’d log in any tooling.

After 20:44 that file is bytes. No amount of cleverness gets the bytecode back out of it, because the key was the clock and the clock moved on. So if you’re building anything that involves reading ips.js, the reading must occur in the same session as the fetch. Archiving the assets to analyze at your leisure doesn’t work, and I’d rather you learn that here than from a folder of files that no longer open. The script is at seed_window.js.

The string table, and why substring search lies

Both files pull a string table out of the integer array. The index comes from the last decoded integer XORed with the array length plus four, the slice gets removed, and the value decoder reads the whole thing back as one long string. p.js yields 6,145 characters, ips.js yields 38,946, and 99.87% of the latter is printable ASCII.

Here’s the first 600 characters of the ips.js table, verbatim:

guaapδonmouseentervoiceURIsteptracker2khq_globalProxyscreenToptwqflro_mdοnamejxrkgvdthtsubtlescaleeoremoveWin64local-network-accessdylibVersionw1(async()=>{try{innerText7SsUqMprototypevtgetComputedStylexouterHeightroundiwawaMediaRecorderMediameallgpd🔫ιbbvsfMUjziJmhaxcanvaappendMimeTypeArrayeventListenerscchannels[object Object]ypkjpkWIN64if_pn_pcounterarcsr-rsqtdmatchkhazh-hkuuxskkenaudio/webm;codecs=opusavkdppxgetPrototypeOfGestureEventsololveecanvaeditormakmsorientationaltyvjxktluKidoPizzeriastorage-accessdevice-aspect-ratiorndangela2stytivperiodic-background-syncsab(async()=>{try{cons

There are no delimiters. onmouseenter, voiceURI, _globalProxy, screenTop, getComputedStyle, outerHeight, MediaRecorder and audio/webm;codecs=opus are all in there, and so are khq, jxrkgvdtht, KidoPizzeria, angela2 and a revolver emoji. Substring-matching a property name against that soup will find things that aren’t there. canva appears twice, and it isn’t the design tool.

The way to get a defensible answer is to stop searching for substrings and reconstruct structural references instead. The value decoder encodes a string as a triple of a tag, a length and an offset into the table. Scan the integer stream for well-formed triples and you get values the bytecode can genuinely address, not characters that happen to sit next to each other. That scan found 14,734 references resolving to 3,789 distinct values.

Level A says the letters w-e-b-d-r-i-v-e-r appear somewhere in that soup. Level B says the interpreter holds a reference it can resolve to webdriver. Only the second one is worth writing down.

Two VMs, 86 opcodes and 193

The two files run different machines, and this surprised me.

p.js is the straightforward one. It carries a literal array of 86 opcode functions wrapped in a Proxy, and the handler adds a layer of indirection so the opcode number in the bytecode doesn’t map directly to a slot in the array. The instruction reader is three tokens long:

Read more

We came up in the sneakerbotting world. When you are racing thousands of other bots for a pair of shoes that sells out in four seconds, you do not open a browser. A browser is too slow, too heavy, and gives away too much. You send raw HTTP requests, and you win or lose on a very short list of signals: does your TLS handshake look like Chrome, are your headers in the exact order Chrome sends them, and do you carry a valid antibot token.

That is the whole game. And it turns out the same three signals are all you need to pull one of the hardest DataDome targets on the web, hermes.com, straight off its own JSON API.

This post is a showcase of a request-based Hermès scraper we built and are open-sourcing. No browser, no headless Chrome, no page rendering. We warm one session through DataDome, then hit the bck.hermes.com API directly and get back clean structured JSON. We will walk the flow, share the real numbers from scraping the entire US catalog, and show what it costs versus buying the dataset. Every number here is reproducible with a free Hyper Solutions key.

Everything below runs with zero browsers in the loop. The cost scales with kilobytes, not with a browser farm.

The three signals that actually matter

Strip away the folklore and a request-based bot against a modern antibot lives or dies on three things:

1. TLS fingerprint. Your ClientHello has to look like a real Chrome, JA3/JA4 and all. We use [bogdanfinn/tls-client](https://github.com/bogdanfinn/tls-client) with a Chrome profile and randomized extension order.

2. Header order. Not just which headers you send, but the order you send them in, including the HTTP/2 pseudo-header order. Get this wrong and you are flagged before the server reads a single value.

3. A valid antibot token. For DataDome that means a datadome cookie minted from real sensor data. This is the part you cannot fake by copying headers, and it is exactly what the Hyper Solutions API generates.

We have written before about why header order is not enough on its own and why TLS fingerprinting matters. Hermès is the case study that puts all three together, down to details like the API calls sending each header lowercase over HTTP/2, while DataDome’s own geo.captcha-delivery.com endpoints run HTTP/1.1 and want Chrome’s Title-Case names. We matched every request against a real browser capture to get those right.

Here is the header block for a bck.hermes.com API call, straight from the scraper. The order is declared explicitly, not left to the map:

header := http.Header{
    "sec-ch-ua-platform": {`"Windows"`},
    "x-xsrf-token":       {xsrf}, // minted by /sync-form, see below
    "user-agent":         {UserAgent},
    "accept":             {"application/json, text/plain, */*"},
    "sec-ch-ua":          {ClientHints},
    "origin":             {"https://www.hermes.com"},
    // ... sec-fetch-*, referer, accept-* ...
    http.HeaderOrderKey: {
        "sec-ch-ua-platform", "x-xsrf-token", "user-agent", "accept",
        "sec-ch-ua", /* ... exact Chrome order ... */, "cookie", "priority",
    },
    http.PHeaderOrderKey: {":method", ":authority", ":scheme", ":path"},
}

Why no browser, and why the API

The obvious question from the web scraping side is: why go to this trouble instead of driving a headless browser that handles DataDome for you?

Bandwidth. A browser loads the whole product page every time (HTML, JavaScript, CSS, fonts, images, and every XHR the page fires), which is megabytes per product, and you pay residential proxy rates on all of it. But here is the thing: that page is just a wrapper around a JSON API. Hermès’ own frontend fetches product data from bck.hermes.com, and once you hold a warmed session you can call that endpoint directly:

GET https://bck.hermes.com/product?productsku=H100044FPG2&locale=us_en

{ "sku": "H100044FPG2", "title": "Habascule single earring", "price": 445,
  "variants": { "colors": [...], "finishes": [...] },
  "categories_breadcrumbs": [...], "stock": {...}, ... }

That response is the structured product object: price, every color and finish variant, breadcrumbs, stock, dimensions, material. No HTML to parse, no rendering, no XHR replay. One small request in, clean JSON out. It weighs about 37 KB. The equivalent full page render is on the order of a hundred times heavier.

Warming the session through DataDome

Hermès puts DataDome in front of everything, so the one thing you cannot skip is the bootstrap. We warm a fresh session once, then reuse it for dozens of API calls:

1. Load the homepage. If DataDome challenges it (a 403 with an inline dd object, or an XHR challenge carrying a geo.captcha-delivery.com device link), we solve it, either interstitial or slider, through the Hyper Solutions API.

2. Post the tags. DataDome’s client tag fires ch then le on a real page load; we do the same to raise the session’s trust score.

3. Call /sync-form. This is what mints the x-xsrf-token cookie (and the rest of the ecom session cookies) that every subsequent API call echoes back as a header.

4. Hit the API. With the session warm, bck.hermes.com/products (category listings) and bck.hermes.com/product (full detail) return JSON directly. The session rotates after a set number of calls so no single datadome cookie is stretched too far.

Session bootstrap and API flow.

The DataDome solves and the tags posts are the only places we hit the Hyper Solutions API. Everything else is plain HTTP. That distinction is the whole cost model: sensor calls are the metered part, and we amortize them across every product a warm session pulls.

Finding every product

You do not need a sitemap. Hermès exposes its full category tree at bck.hermes.com/menu, and each category is a paginated listing endpoint:

GET https://bck.hermes.com/products?category=WOMEN&sort=relevance&pagesize=144&locale=us_en

Walk the menu, page through each top-level category, and collect the SKUs. A useful quirk we verified against the live data: a top-level category already aggregates its entire sub-tree. Pulling WOMEN returns all 2,785 women’s products, and every subcategory beneath it adds essentially nothing new. So the whole US catalog is a handful of listing crawls.

That catalog is 7,690 unique products for the US. Worth noting, because the products sitemap makes the catalog look like it has on the order of a million URLs, but most of those are not live, purchasable products. The API listing is the real catalog, and it is far smaller than the sitemap suggests.

The numbers

We scraped the entire US catalog end to end (discover every SKU, then fetch full detail for each) on a modest 8 concurrent sessions. Here is the product-detail run, measured end to end:

Discovering every SKU first (walking the menu and paginating each category) adds only about another minute, ~10 MB, and ~56 sensor calls on top. The scraper prints the derived ratios at the end of the run:

• ~26,000 products per GB of bandwidth (37 KB per product).

• ~4,500 products per 1,000 DataDome sensor calls (0.22 sensor calls per product).

Scrape summary and blog metrics for the US catalog run.

The sensor calls are dominated by session bootstrapping plus the occasional mid-run challenge; give each session more calls before it rotates, or cleaner proxies, and that ratio improves further.

Buy versus build

Two inputs set the cost: bandwidth on residential proxies (we quote Bright Data’s $3.50/GB, though cheaper proxies bring it down) and DataDome sensor data (Hyper Solutions pay-as-you-go at €3 per 1,000 calls).

The whole US catalog costs a few euros to scrape. The ~0.29 GB of bandwidth is about $1 of proxy, and the ~1,765 sensor calls are about €5. That is the entire purchasable US catalog, refreshed in about twenty minutes on a modest 8 concurrent sessions (~400 SKUs a minute, ~50 per thread per minute).

Scaled to all of Hermès’ markets, it grows but stays cheap. Per-market catalogs vary (across 18 markets we measured, Canada was the largest at ~10,000 SKUs, Brazil the smallest at ~3,000, averaging ~7,150), so across the 39 markets the country selector lists that is on the order of ~280,000 records. Held to the same measured ratios, that is roughly 11 GB of bandwidth (~$37 of proxy) and ~63,000 sensor calls (~€190).

Buying the data ready-made is a perfectly reasonable alternative, and for plenty of teams the right one: you are paying for convenience and for someone else maintaining the pipeline. As a reference point, Bright Data’s self-serve dataset estimator starts at $500 for up to 200,000 records ($2.50 per 1,000), with the per-record rate dropping only at much larger volumes.

The thing to notice is scale. A catalog like Hermès is small (7,690 US records, a few hundred thousand across every market), so it sits right at the entry of that pricing, well inside the smallest tier. If you already run request-based scrapers, producing the data yourself costs a fraction of the purchase price and gives you a dataset you can re-pull whenever prices or stock move, rather than a one-time snapshot. If you don’t, buying it may be worth the premium. For a target this size, though, the maths lean toward building.

One more advantage of going straight to the API is completeness. The /product response hands back the full structured object in a single call: price, every color and finish variant, dimensions, material, stock, breadcrumbs. A page-rendering pipeline has to reconstruct all of that from the DOM, and often misses the parts that never appear as plain text on the page. The scraper writes it verbatim, one raw product per line of JSONL, so nothing is normalized away. Here you get the whole record for free, at a fraction of the bandwidth.

(US figures are measured on a single run; multi-market totals are projected across the 39 markets from those ratios and the per-market catalog sizes we measured across 18 markets, and are illustrative.)

Try it yourself

The scraper is written in Go, uses bogdanfinn/tls-client for the fingerprint and fhttp for header ordering, and calls the Hyper Solutions API only for the DataDome sensor data. It is open source:

github.com/Hyper-Solutions/hermes-scraper

Two commands: one discovers every category and collects SKUs, the other fetches full product detail and prints the metrics above.

go run ./cmd/skus     -locale us_en -out skus.json
go run ./cmd/products -in skus.json -out products.jsonl

You need two things: a Hyper Solutions API key for the DataDome sensor data (free to start, no credit card), and a sticky residential proxy (request-based scraping still runs on residential IPs). Grab a key and point it at any locale.

Same idea we keep coming back to: figure out the antibot flow once, then run plain requests forever. Hermès and DataDome are a hard target, but the sneakerbot playbook (real fingerprint, real header order, real token) ports over cleanly. And it does it at a fraction of the bandwidth a browser would burn.

Want the numbers in your own terminal? Spin up a free Hyper Solutions trial, no credit card, clone the repo, and run the two commands above against any Hermès locale. The same API mints the sensor data for DataDome, Akamai, Incapsula, and Kasada, so it carries over to whatever request-based target you hit next.

This showcase uses publicly available product data and is provided for educational purposes only. Figures are from measured runs and multi-market projections, and are illustrative.

GoScrapy tries to replicate the same philosophy in Go, taking advantage of the language’s benefits, such as performance, concurrency, and simple deployment.

In this post, you’ll learn what GoScrapy is, what it offers, how it works, and how to use it through a complete end-to-end example. Let me present what this Go-based scraping framework brings to the table!

What Is GoScrapy?

GoScrapy is an open-source web scraping framework for Go inspired by Python’s Scrapy architecture.

The library adopts a batteries-included approach, providing a complete scraping engine with spiders, middlewares, pipelines, automatic retries, cookie management, CSS/XPath selectors, and concurrent request processing.

At the time of writing, the project has over 350 GitHub stars. It also recently underwent a major architectural revamp focused on improving performance and maintainability, as announced by the author on Reddit.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

Digging into the GoScrapy Scraping Library

Let me introduce you to the world of GoScrapy.

Why Use GoScrapy for Web Scraping in Go?

At first, you might wonder why you would use GoScrapy instead of Scrapy or Playwright, Puppeteer, Axios + Cheerio, or Requests + Beautiful Soup. After all, those are all excellent options, if not the industry standard.

However, there are many situations where Go is a better fit. Go offers excellent performance, lightweight concurrency through goroutines, fast startup times, and simple deployment as a single compiled binary. Similar advantages can also be found when building web scrapers in Rust.

Plus, if you’re already developing in Go, introducing a Python or JavaScript scraping stack means maintaining another language, runtime, and dependency ecosystem.

Main Features

Some of GoScrapy’s most notable features include:

• Scrapy-inspired architecture: Brings Python Scrapy’s proven design to Go, organizing spiders, callbacks, middlewares, and pipelines into a structured, maintainable scraping workflow.

• High-performance concurrency: Leverages Go’s goroutines and worker pools to process many requests in parallel, enabling fast, scalable web scraping with minimal overhead.

• CSS and XPath selectors: Supports chainable CSS and XPath selectors, making it easy to locate elements and extract both text and attribute values.

• Automatic project scaffolding: Comes with goscrapy startproject to generate a new project structure with a single CLI command.

• Built-in middlewares: Includes retry logic, cookie management, duplicate request filtering, and other middleware components to handle common scraping challenges.

• Flexible export pipelines: Export scraped data to CSV, JSON, MongoDB, Google Sheets, Firebase, or custom destinations by configuring reusable pipelines.

• Signal-driven architecture: Exposes strongly typed lifecycle events, allowing you to monitor engine activity and hook custom logic.

• Auto-discovery of spider methods: Autonomously detects lifecycle methods such as Open(), Close(), and Idle(), reducing boilerplate and simplifying spider implementation.

• Extensible core: Lets you replace or customize core components to fit advanced scraping requirements.

Start your scraping journey with Byteful: 10GB New Customer Trial | Use TWSC for 15% OFF | $1.75/GB Residential Data | ISP Proxies in 15+ Countries

Claim your 10GB here

Architecture

GoScrapy provides a complete scraping engine with spiders, middlewares, pipelines, retries, cookie management, and concurrent request processing, following the architecture that made Scrapy popular in Python.

Under the hood, GoScrapy uses a modular, event-driven design where components communicate through a central signal bus. This decouples the engine from individual components, enabling you to monitor activity or add custom logic at different stages of the scraping lifecycle.

Signals are triggered for important events, such as engine startup and shutdown, spider lifecycle changes, request scheduling, response handling, and item processing. Discover all available signals.

Declarative Mapping via Gosm

One of GoScrapy’s most interesting aspects is Gosm, a declarative mapping engine that eliminates much of the repetitive extraction code usually required in web scraping.

Gosm lets you describe both your data model and extraction logic directly in Go structs through tags, removing the need to manually handle selectors and value assignments. In particular, it supports CSS selectors, XPath, JSON paths, and attribute extraction.

Once the model is defined, GoScrapy exposes the gosm.Map() utility to automatically populate the struct of the custom type with the extracted data.

For example, you could define a model for a book like this:

type Book struct {
    Title  string `gos_css:".title"`
    Author string `gos_css:".author"`
    Cover  string `gos_css:"img@src"`
    URL    string `gos_css:".book-link@href"`
}

Note how, thanks to Gosm, each struct field is mapped to a specific element on the target page. For example, Title extracts the text inside the .title element, while Cover and URL retrieve values from the src and href attributes using the @attribute syntax.

Inside your spider parsing logic, you can then map the response into the struct with a couple of lines:

func (s *Spider) parse(ctx context.Context, resp core.IResponseReader) {
    // Gosm automatically fills the struct using the selectors defined in the tags
    var book Book
    gosm.Map(resp, &book)

    s.Yield(&book)
}

By separating the data model from the scraping workflow, Gosm keeps spiders cleaner.

GoScrapy in Action: A Complete Example

In this step-by-step section, I’ll guide you through scraping ScrapingCourse.com’s E-commerce page with GoScrapy:

The goal is to build a GoScrapy script that extracts all product data from the target site’s product listings, navigating through all pagination pages.

Prerequisites

For simplicity, I’ll assume you have Go 1.26 or later installed locally.

Step #1: Install GoScrapy and Set Up Your Project

Start by installing the GoScrapy CLI:

go install github.com/tech-engine/goscrapy/cmd/...@latest

Verify that the installation completed successfully:

goscrapy -v

You should see output similar to:

goscrapy version 0.15.4

Next, navigate to your project’s root directory and create a new GoScrapy project:

goscrapy startproject ecommerce_scraper

This command initializes a new Go module and generates all the required project files. It’ll also prompt you to install the project’s dependencies (go mod tidy). Type “Y” and press Enter to continue:

For manual dependency installation, run:

go mod tidy

Your project directory should now look like this:

your-goscrapy-project-folder/
├── go.mod
├── go.sum
├── main.go
└── ecommerce_scraper/
    ├── pipelines/
    ├── base.go
    ├── constants.go
    ├── errors.go
    ├── job.go
    ├── record.go
    ├── settings.go
    └── spider.go

Well done! You’ll notice that the GoScrapy project structure closely resembles that of Python’s Scrapy. In particular:

• spider.go is where you’ll implement your scraping logic.

• record.go defines the data structure for the scraped items and controls how extracted data is exported.

• settings.go contains the project’s configuration, allowing you to customize how the scraper runs.

For more details on the main GoScrapy files, refer to the docs.

Check the TWSC YouTube Channel

Step #2: Get Familiar with the Target Site

Open the target scraping site in your browser and start exploring its structure. You’ll notice that products are stored inside li[data-products=’item’] elements:

In detail, each product contains the following information:

• Name: Stored in the .product-name element.

• URL: Available in the href attribute of the .woocommerce-loop-product__link element.

• Image: Available in the src attribute of the img element.

• Price: Stored in the .price element.

• Availability: Stored in the class attribute of the product element itself. The availability status can be determined by checking for the presence of the “instock” or “outofstock” class.

Additionally, each page (except the last one) contains a link to the next page inside the nav#pagination a.next element. By following this link, you can sequentially scrape all product listing pages.

Now that you understand the target HTML structure, you’re ready to define the data models that GoScrapy will use to extract and store the scraped data!

Step #3: Define the Gosm Logic

Make sure that record.go contains the following:

type Record struct {
        J            *Job   `json:"-" csv:"-"`
        Name         string `json:"name" csv:"name"`
        URL          string `json:"url" csv:"url"`
        Image        string `json:"image" csv:"image"`
        Price        string `json:"price" csv:"price"`
        Availability string `json:"availability" csv:"availability"`
}

type Listing struct {
        ProductNames  []string `gos_css:"li[data-products='item'] .product-name"`
        ProductURLs   []string `gos_css:"li[data-products='item'] .woocommerce-loop-product__link@href"`
        ProductImages []string `gos_css:"li[data-products='item'] img@src"`
        ProductPrices []string `gos_css:"li[data-products='item'] .price"`
        ProductClass  []string `gos_css:"li[data-products='item']@class"`
        NextPage      string   `gos_css:"nav#pagination a.next@href"`
}

The Record struct defines the final output of the scraper. Each instance represents a single product and contains the fields that will be exported by the pipeline.

On the other hand, the Listing struct describes the data available on each product listing page. Notice that product-related fields are stored as arrays ([]string). The reason is that GoScrapy’s scraping approach focuses on extracting each field from all matching elements rather than mapping each product element individually. This pattern isn’t unique to GoScrapy, and other libraries like rvest in R follow a similar mechanism.

As a result, Gosm collects all product names, URLs, images, prices, and classes separately, allowing you to combine them later. Fantastic!

Step #4: Define the Scraping Logic

The data models with Gosm mappings are ready, so you can implement the spider logic to extract products, process the data, and follow pagination. To do so, make sure that your spider.go file contains:

package ecommerce_scraper

import (
    "context"
    "strings"

    "github.com/tech-engine/goscrapy/pkg/builtin/gosm"
    "github.com/tech-engine/goscrapy/pkg/core"
)


func(s * Spider) Open(ctx context.Context) {
    req: = s.Request(ctx)

        req.Url(
        "https://www.scrapingcourse.com/ecommerce",
    )

        s.Parse(
        req,
        s.parseListing,
    )
}

func(s * Spider) parseListing(
    ctx context.Context,
    resp core.IResponseReader,
) {

    var listing Listing

    _ = gosm.Map(resp, & listing)

    for i: = range listing.ProductNames {
        availability: = "unknown"

            if strings.Contains(listing.ProductClass[i], "instock") {
            availability = "instock"
        } else if strings.Contains(listing.ProductClass[i], "outofstock") {
            availability = "outofstock"
        }

        s.Yield( & Record {
            Name: listing.ProductNames[i],
            URL: listing.ProductURLs[i],
            Image: listing.ProductImages[i],
            Price: listing.ProductPrices[i],
            Availability: availability,
        })
    }

    if listing.NextPage != "" {
        s.Logger().Info("Following next page:", listing.NextPage)

        s.Parse(
            s.Request(ctx).Url(listing.NextPage),
            s.parseListing,
        )
    }
}

This GoScrapy spider implements this parsing flow:

1. Extract all product fields from the page in a single mapping operation: The gosm.Map() function applies the CSS selectors defined in Listing and populates the corresponding fields with all matching values.

2. Combine extracted fields into individual product records: Since Gosm extracts product fields as separate arrays, the for loop iterates over them by index. Values with the same index belong to the same product, helping you create a complete Record object.

3. Apply additional processing before yielding records: Some fields require custom logic before being exported. In this example, availability cannot be extracted directly because it’s stored as a class value on the product HTML element. The scraper checks whether the class contains “instock” or “outofstock” and converts it into a dedicated Availability field.

4. Take care of the pagination logic: The scraper checks whether a next page exists. If NextPage contains a URL, it creates a new request and recursively calls parseListing() to continue scraping until all product pages have been processed.

The above spider extracts products across multiple pages and sends each completed Record to the pipeline. Wonderful!

Need help with your scraping project?

Step #5: Define the Export Logic

By default, a GoScrapy project is configured to export scraped data to an itstimeitsnowornever.csv file. This behavior is controlled through the configuration defined in settings.go.

To export the scraped product data to a products.csv file, make sure that settings.go contains:

var export2CSV = csv.New[*Record](csv.Options{
    Filename: "products.csv",
})

Step #6: Run the Scraper

Launch the GoScrapy project with:

go run .

You should see output similar to the following:

Once the scraper finishes, a products.csv file will be created in the project folder. Open the file, and you’ll see the extracted product data:

The produced CSV file contains the information extracted from all 190 products spread across multiple listing pages. Mission complete!

Final Comment

Considering my experience with web scraping in Go, I can say that GoScrapy is definitely one of the most promising libraries in the ecosystem. The fact that it recreates Scrapy’s architecture significantly lowers the learning curve, especially if you’re already familiar with Scrapy’s project structure, workflows, and core concepts.

I found the library fast, and the Gosm mechanism was impressively interesting. At the same time, it’s important to note that the community around the project is relatively small, making it harder to find examples, tutorials, and external resources. Also, most LLMs have limited (or no) knowledge of the framework, and there’s no real official documentation.

That said, the examples available in the GoScrapy repository are enough to build a basic understanding of how the framework works. If you’re already comfortable with Go and have experience building spiders with Scrapy, you should be able to create a functional scraper within a few hours.

Overall, GoScrapy is a project worth keeping an eye on, particularly for Go enthusiasts!

I hope you found this guide useful. If you have any questions or comments, feel free to leave them below. Thanks for reading, and see you in the next one!

FAQ

How to set a proxy in GoScrapy?

GoScrapy supports proxy configuration through the PROXY_LIST setting in settings.go. Define your proxy URLs as a comma-separated list:

const PROXY_LIST = "http://user:pass@host:port,http://host:port"

Then, enable the setting in the init() function:

"PROXY_LIST": PROXY_LIST,

GoScrapy will use the configured proxies for outgoing requests. You can provide multiple proxies to distribute requests and improve reliability when scraping at scale.

Is GoScrapy’s API stable?

As of this writing (July 15, 2026), not yet. GoScrapy is currently in active v0.x development, and its Core API is still evolving toward a stable v1.0 release. While it’s already fully usable today, you should expect occasional breaking changes as the project continues to mature.

Why is GoScrapy licensed under the Business Source License (BSL)?

The BSL is designed to support the long-term sustainability of GoScrapy. You can use it freely in production, build commercial applications with it, and scrape data for your business. The main restriction is that you can’t repackage or offer GoScrapy itself as a competing managed scraping service or commercial fork.

Obscura is not. It is a browser engine written from scratch in Rust that embeds V8 for JavaScript, ships no Chromium and no Node.js, and claims to run in about 30 MB of RAM. I already put its fingerprint on the bench once, in THE LAB #111, where it scored well in a controlled probe. A lab score is one thing. What I wanted this time was to point it at a real anti-bot on a real site and see what a featherweight engine actually buys you.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

The site is Radisson. Their booking pages hide an availability API that returns every room rate for a hotel and date range, and that API sits behind Akamai Bot Manager. If you can talk to it, you can do something useful with it. You can track a single hotel’s nightly price over the next few months and find the cheapest and most expensive days to go. I picked Munich, because Oktoberfest 2026 runs from September 19 to October 4, and nothing moves a city’s hotel prices like a festival that pulls millions of visitors into one place.

So there are two questions stacked on top of each other. Can Obscura clear Akamai at all? And if it can, is a 30 MB engine a sensible way to mint the session that a fast HTTP client then reuses? That second pattern is not new. In THE LAB #100 I built exactly this hybrid, one browser login feeding thousands of curl_cffi requests, but with Camoufox, a real Firefox. So this is really a hands-on review of Obscura for one concrete job. Can the cheap engine take Camoufox’s place, and where it cannot, is that the tool’s fault or the target’s? That second question turned out to be the interesting one.

The tools on the table

Three pieces do the work.

Obscura is the from-scratch Rust engine. Because it implements its own DOM and Web APIs around V8 rather than shipping a real browser, it is small and fast, and that same design is the source of its quirks. It exposes a Chrome DevTools Protocol server, so we can drive it with Playwright over connect_over_cdp exactly like a headless Chrome. Its --stealth flag turns on a consistent spoofed fingerprint, which is the feature under test.

curl_cffi is the replay layer. It is a Python HTTP client that impersonates a real browser’s TLS fingerprint (JA3/JA4, HTTP/2 settings, header order) without running a browser at all. When TLS fingerprinting is the wall, it walks through it in milliseconds. When JavaScript execution is the wall, it cannot help. Knowing which wall is which is most of the job.

The third piece is the fingerprint bench from #111, which reads a browser’s own fingerprint surface from a local probe page and scores it out of 100. I reused it here to get a fresh, reproducible number for Obscura before sending it into the fight.

Two more browsers turn up later as controls, both from that same bench and both scoring a clean 100. CloakBrowser is a Chromium patched at the C++ level, and Camoufox is a Firefox fork with a coherent injected fingerprint. I ran the same proxy test through them to separate a tool problem from a proxy problem, and all three browsers gave the identical result.

How Radisson is actually protected

Before running anything, it pays to model the target. Radisson does not protect everything the same way, and the whole strategy depends on that split.

The HTML pages, like the room-display page you land on after picking a hotel and dates, come back fine to a good TLS impersonation. A plain curl gets nothing, HTTP 000, because its TLS handshake does not look like a browser. curl_cffi with a Chrome profile gets HTTP 200 and 380 KB of real HTML. So the first tier is TLS fingerprinting, and curl_cffi alone handles it.

The XHR APIs are a different story. Two matter here. content-api/hotels/{code}/rooms returns a hotel’s bookable rooms, and availability-api/stays-by-rooms returns the rates for a set of rooms on given dates. Both sit behind Akamai Bot Manager. A request with no cookies gets HTTP 403 with a 421-byte body. Akamai wants its cookie set first: _abck, bm_sz, ak_bmsc, bm_sv, bm_so, bm_s. Those cookies are minted by the Akamai sensor script, a chunk of obfuscated JavaScript that runs on page load, collects a pile of signals, and posts them back. Only after that handshake does _abck reach a validated state and the API start answering.

There is a third layer that shows up when you push too hard. Load the wrong path, or hit the site too often from one address, and Akamai serves a small interstitial instead of the page. It is the sec-cpt behavioral challenge, with a sec-if-cpt-container div and a friendly “Powered and protected by Akamai” line. That page runs its own script and expects the client to solve it before moving on.

So the layer we target is clear. curl_cffi covers the TLS tier and the raw HTTP replay. The one thing it cannot do is run the Akamai sensor to mint _abck. That is the job we hand to Obscura, and it is the exact point where a from-scratch engine either earns its place or does not.

Check the TWSC YouTube Channel

Testing the fingerprint first

There is no reason to send a browser at Akamai if its fingerprint falls over on the easy tests, so I scored it first.

We ran Obscura 0.1.10 headless with --stealth, driven through Playwright over CDP, against the #111 local probe and then against the public test pages people actually cite. The probe reads the browser’s own surface, and here Obscura looks the part. navigator.webdriver is false, on both the prototype and the instance. The user agent is Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... Chrome/145.0.0.0 Safari/537.36, a plausible Windows Chrome with no HeadlessChrome token. window.chrome is complete, with runtime, csi, loadTimes, and app all present. navigator.userAgentData reports Chrome 145 and Chromium 145. The WebGL unmasked renderer is a spoofed ANGLE (NVIDIA, NVIDIA GeForce RTX 30...). There are five plugins, a Europe/Berlin timezone, and no automation residue at all: no leaked CDP globals, no driver keys.

The score came out at 94 out of 100. That puts Obscura above every Chromium CDP tool in the bench, which sit at 70 because their user agent still carries a headless token, and well above vanilla Playwright and Selenium at 35. The one thing that costs it points is the native-code check. When you ask whether a Web API function is really native by inspecting its toString, a few of Obscura’s functions give themselves away as patched. That is the single tell, and the bench labels it “patched fns”.

The public suites are where it gets more interesting, and more honest. On Sannysoft, Obscura passes cleanly, 21 checks green and none red. But CreepJS never finishes. The page loads to about 8 KB and then stops, with no trust score ever computed. Pixelscan renders its landing chrome but never produces an automation verdict. iphey renders too, but the verdict banner it normally paints, trustworthy or suspicious, never populates. The two oracles from #111, BrowserScan and deviceandbrowserinfo, were inconclusive for the same reason. browserleaks is the exception that proves the point: its canvas and WebGL pages are light enough that Obscura runs them, and the WebGL page confirms the same spoofed NVIDIA GPU the probe reported.

The split is the interesting part. Obscura sails through the boolean checks, the ones that ask a single yes-or-no question about a property. It stalls on the heavy suites, the ones that run thousands of lines of fingerprinting JavaScript to completion. Those suites cannot flag it, but only because they never finish running inside the minimal engine. Hold that thought, because Akamai’s sensor is exactly that kind of heavy script.

Building the hybrid, Obscura mints and curl_cffi replays

The plan follows #100. Use the browser only to mint a valid Akamai session, then hand the cookies to curl_cffi and let it hammer the availability API for every date we care about. The code lives in our repository under obscura_radisson_deals.

The session layer is session.py. It launches Obscura as a CDP server with --stealth, connects Playwright, and loads the room-display page for the target hotel. Then it harvests the cookies and checks them by calling the content-api rooms endpoint. If that call returns 200, the session is good and we also get the room ids we need for scanning. If not, it reloads the page and tries again. This retry loop is not decoration. It is there because of what we found next.

def mint(self, attempts=6, settle_s=8.0):
    for attempt in range(attempts):
        if attempt == 0:
            self._page.goto(self.page_url, wait_until="load", timeout=45000)
        else:
            self._page.reload(wait_until="load", timeout=45000)
        time.sleep(settle_s)
        self.cookies = self._harvest()
        probe = resolve_rooms(self.cookies, self.hotel_code, self.currency, self.proxy)
        if probe["ok"]:
            self.room_ids = probe["room_ids"]
            return True
    return False

From a clean residential IP it works, and it works fast. On a good load, with bm_sv and bm_mi present in the jar, we took the harvested cookies straight to curl_cffi with a Chrome 131 TLS profile. The availability-api/stays-by-rooms endpoint answered HTTP 200 with hundreds of kilobytes of real rate JSON. Against the Munich airport hotel the first load minted a valid session and resolved its rooms on attempt zero. That is the whole thesis in one response. The 30 MB engine minted a session a fast HTTP client could reuse.

It is not always the first load. Sometimes the handshake does not complete, _abck stays in its unsolved state with a -1 in the second segment, and the API answers 403. Obscura harvests nine cookies on those loads too, but without bm_sv and bm_mi the session is not valid, so mint() reloads until one lands. On a trusted IP a reload or two is enough. The reason it can miss at all is the same trait we saw on CreepJS: the minimal engine runs the Akamai sensor part of the way and then not. Camoufox, a real Firefox, does not have that wobble.

There is a second signal worth recording. When we called the API from inside Obscura’s own page context instead of replaying externally, it did not return 403. It returned 500, an application error, which means the request cleared the Akamai wall and reached the backend. A 500 is not a bot block. So the in-browser path can get through too. It just needs the request shaped exactly the way the site’s own code shapes it, and on the room-display page the SPA kept navigating and tearing down our injected fetch before it settled.

You can find the full code used in this article in The Lab repository, available only to paying readers, in the folder “113.Obscura”.

The wall is the IP, not the fingerprint

Read more

Never imagined that someone would have read it. It was just an excuse for writing down my scraping notes in a more structured way.

Four years later, both TWSC and the industry look very different.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

Health of The Web Scraping Club

First of all, thank you.

Thank you to all the readers who think I’m saying something interesting enough to be subscribed to.

In this era, when we’re all fighting for attention, having 7,000+ people who choose to receive this newsletter and open it every week seems like a miracle.

This is despite no marketing efforts, 0 ads, and an issue with Google SEO that made the whole domain disappear from the results from one day to the next.

In the next few days, to solve this issue, there will be a new domain address that, according to my experiments, should fix this.

Let me also thank you, all the backers, paying readers, and companies that make this possible.

While still very lean, The Web Scraping Club in its current format could not run without sponsors and readers who choose to support it. And about the companies backing TWSC, let me spend a special word on the NetNut team, who are having a hard time now. I’ve met great people there, and they were among the first believers in this newsletter.

Behind the scenes, TWSC is far from perfect, but I’m doing my best to keep things running smoothly.
I’ve just launched a partner portal, where sponsors can monitor their campaign. For the readers, this year we’ve released the new version of the Proxy Price benchmark tool, which I promise will be updated more often in the future. Also, the Scraping Wiki, our repo with all the findings of The Web Scraping Club organized as a second brain, is a recent new entry. Please give it a look and share your feedback.

What’s next

Apart from fixing the Google SEO issue in the immediate term, I’ve got a couple of ideas that still need time to be implemented, but hopefully they’ll see the light before the end of the year.
Something I’ve been asked for a long time is a Job Board dedicated to open scraping positions and the great talent to fill them.
Is this something you’d like to have here?

Ideally, I’d like to revamp the YouTube channel. And last but not least, I’m working on a web-scraping event in Italy in October. More details soon, but be ready for it!

Check the TWSC YouTube Channel

How the industry changed

The industry over the last four years looks very different.

We got LLMs; we know it, so the coding part is way faster. But what they brought to the industry is the legitimacy of web scraping.

It seems the world has recognized the role of web data and browser automation in innovation. While web data was once seen as something shady for spying on your competitors, it is now the main moat for training LLMs, and browser automation is the key tool for giving your agents real-time data.

This changed the perception of the industry as a whole, and the recent funding rounds (Nimble, Oxylabs) are the proof.

Think about it: have you ever heard of a funding round for a proxy company before 2023?

Conclusion

It’s an exciting time for the industry, and I’m glad to work at The Web Scraping Club right now. I’ll be at conferences around Europe (Oxycon in September and the Extract Summit in Dublin), which will allow me to meet great professionals and learn about the latest trends.

While both TWSC and the industry have changed a lot over the years, the spirit of this newsletter hasn't: no marketing fluff, honest reviews, some hot takes, and my personal opinions. I know that in four years, not all articles can be a 10 out of 10, but I can assure you that every article written here was the result of sweat and hard work. And it will be the same for the future.

That’s where the MarkItDown Python library comes into play!

Here, I’ll show you everything you need to know about MarkItDown, from installation to practical examples, so you can convert HTML pages, PDFs, Word documents, Excel spreadsheets, and many other data sources into LLM-ready Markdown.

Why the Markdown Data Format Is More Relevant Than Ever

As I covered in a recent post, AI-ready web scrapers, RAG pipelines, and agent frameworks all have one thing in common: they rely heavily on Markdown!

That’s no coincidence... Mainstream LLMs (including OpenAI’s latest models) natively “speak” Markdown and often generate Markdown-formatted responses without being prompted.

Why? Because Markdown strikes the perfect balance between plain text and structured documents. It preserves headings, lists, tables, links, code blocks, and other semantic elements without the overhead and noise of HTML.

As a result, LLMs can understand and process Markdown efficiently while consuming far fewer tokens. This translates directly into lower inference costs and leaves more of the context window available for meaningful content.

In my experience (and as I demonstrated in a benchmark I conducted for Bright Data), Markdown has become the natural format for feeding documents, especially content extracted from the web, into LLMs.

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

Why Markdown Conversion Matters in Web Scraping Pipelines

Web data is rarely the final destination. Quite the opposite, scraped content generally represents the starting point of a larger data analysis pipeline, where the goal is to extract insights, enrich datasets, or power downstream applications.

In these workflows, the information you need tends to go beyond traditional web pages. PDFs, Word documents, videos, presentations, and other files can provide additional context required for accurate analysis and processing. These resources may come from the web and be collected during scraping, or they may be added later.

Regardless of the original format, data is increasingly processed by AI systems. Since LLMs naturally work well with Markdown, converting different sources into a clean, structured Markdown representation can greatly simplify your data pipelines.

That’s why you need a multi-format Markdown conversion library in your toolkit, such as MarkItDown!

The new anti-bot solution - Toughest walls, lowest price. Claim your free 10 000 requests with coupon code WSCLUB

Claim your credits

MarkItDown: A Complete Overview

Let me now introduce you to MarkItDown, one of the most popular libraries for converting multiple input formats into Markdown.

What Is MarkItDown?

MarkItDown is an open-source Python library for converting a wide range of documents and other data sources into Markdown. The project is developed and maintained by Microsoft.

It’s quickly become one of the most popular tools in its category (more on this later in the post), with over 165k GitHub stars, over 12 million monthly downloads, and a thriving community.

MarkItDown supports dozens of input formats, ranging from Office documents and PDFs to images, audio files, web pages, YouTube videos, and structured data formats such as JSON, XML, and CSV.

Key Features

The main capabilities supported by MarkItDown are:

• Broad format support: Converts dozens of input formats, including PDFs, Office documents, HTML, images, audio, EPUBs, ZIP archives, YouTube URLs, and structured data files.

• Python API: Offers a clean, developer-friendly API for integrating document conversion into Python applications.

• Native CLI: Includes a rich CLI for converting files directly from the terminal, making it ideal for scripting, batch processing, and DevOps workflows.

• Plugin architecture: Supports third-party plugins, allowing the community to extend the library with custom converters and additional processing features.

• AI-powered processing: Integrates with LLMs to generate image descriptions, perform OCR, and transcribe audio, enriching the resulting Markdown.

• Flexible dependency management: Lets you install only the converters you need, reducing dependencies and keeping deployments lightweight.

• MCP server support: Provides a lightweight Model Context Protocol server that allows AI agents to call MarkItDown as a tool.

• Azure AI integration: Works with Azure Document Intelligence and Azure Content Understanding for higher-quality OCR, layout analysis, and structured field extraction.

Supported Input Formats

Installation and Setup

Follow the steps below to add MarkItDown to your system.

Prerequisites

MarkItDown requires Python 3.10 or higher. It’s also recommended to install it inside a Python virtual environment.

Installation via pip

The easiest way to install MarkItDown is through the PyPI markitdown package with:

pip install 'markitdown[all]'

Note that [all] option adds all optional dependencies required to support every available input format.

If you only need support for specific file types, you can install the specific dependencies individually for more control. For example:

pip install 'markitdown[pdf, docx]'

This adds only the extra dependencies required for PDF and Word file conversion.

Available optional dependencies include:

• [all]: Installs all optional dependencies.

• [pptx]: Adds support for PowerPoint files.

• [docx]: Adds support for Word documents.

• [xlsx]: Adds support for Excel files.

• [xls]: Adds support for older Excel files.

• [pdf]: Adds support for PDF documents.

• [outlook]: Adds support for Outlook messages.

• [audio-transcription]: Adds support for WAV and MP3 audio transcription.

• [youtube-transcription]: Adds support for YouTube video transcription.

• [az-doc-intel]: Enables integration with Azure Document Intelligence.

• [az-content-understanding]: Enables integration with Azure Content Understanding.

Note: For installation from source or Docker-based usage, refer to the official MarkItDown docs.

MarkItDown in Action: Full Examples

Time to explore some practical examples of converting different input data formats into Markdown.

HTML Document to Markdown

Suppose you have an HTML page stored locally (e.g., as part of an offline web scraping procedure). You can convert it to Markdown using the MarkItDown CLI:

markitdown example.html -o example.md

Alternatively, you can achieve the same result via the Python API:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("example.html")

with open("example.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

Now, assume the input is the following HTML page (i.e., the homepage from example.com):

The resulting example.md file will contain:

Notice how MarkItDown focuses on the meaningful HTML content, ignoring tags like <style>.

The library extracts the relevant elements and converts them into a structured Markdown representation, keeping important information such as headings, links, and text formatting while removing unnecessary HTML overhead.

Web Page to Markdown

A particularly useful feature for web scraping workflows is that MarkItDown can also fetch and convert web pages directly from their URLs, eliminating the need to manually save the HTML first.

Just pass the URL of the target page directly to the MarkItDown CLI:

markitdown 'https://example.com' -o example.md

Or to its Python API:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("https://example.com")

with open("example.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

Under the hood, MarkItDown performs an HTTP GET request to the specified URL using Requests, retrieves the page HTML, and converts the resulting content into Markdown. As usual, keep in mind that the request may fail if the target website uses anti-bot protections.

The output will be the same example.md Markdown file shown in the previous example.

Check the TWSC YouTube Channel

PDF to Markdown

Many websites store valuable information in downloadable PDF files (e.g., shopping flyers, business reports, whitepapers, product documentation, etc.).

While you can send these files directly to an AI model, doing so often consumes unnecessary context and tokens because PDFs may contain complex layouts and additional formatting overhead.

MarkItDown converts PDFs into clean Markdown while preserving important elements such as headings, document structure, lists, and tables, making the content easier for LLMs to process.

Assume your input is the sample PDF below:

Convert it to Markdown with the MarkItDown CLI:

markitdown sample.pdf -o sample.md

Or in Python:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("sample.pdf")

with open("sample.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

The result will be this sample.md file:

As you can see, MarkItDown preserves all the meaningful content from the PDF.

Excel to Markdown

Now, suppose you have relevant internal data stored in an Excel spreadsheet and want to pass it to an LLM for analysis or processing:

Convert it to Markdown through the MarkItDown CLI:

markitdown spreadsheet.xls -o spreadsheet.md

Or use the Python API:

from markitdown import MarkItDown

md = MarkItDown()
result = md.convert("spreadsheet.xls")

with open("spreadsheet.md", "w", encoding="utf-8") as f:
    f.write(result.text_content)

The result will be a Markdown file containing a table with the same data as the original spreadsheet:

Or, for a better visual representation, through the Markdown viewer:

Need help with your scraping project?

MarkItDown Plugins

MarkItDown also supports third-party plugins that extend its built-in capabilities with additional converters and processing features. You can discover available plugins by searching GitHub for the #markitdown-plugin hashtag:

To list all installed plugins, run:

markitdown --list-plugins

Plugins are disabled by default. To enable them from the CLI, add the --use-plugins flag:

markitdown --use-plugins <SOURCE_FILE_PATH> -o <OUTPUT_FILE>

If you’re using the Python API, enable plugins by setting the enable_plugins parameter to True:

from markitdown import MarkItDown

md = MarkItDown(
  enable_plugins=True,
)

Once enabled, MarkItDown automatically loads all installed plugins.

MarkItDown Alternatives: Other Document-to-Markdown Libraries

MarkItDown isn’t the only library for converting documents to Markdown. Other popular libraries for converting different types of data into Markdown include:

Conclusion

Markdown is the language of LLMs and, as a result, the preferred format for AI-powered data processing and analysis pipelines. In this post, I covered how to use MarkItDown to convert web pages, HTML documents, PDFs, Word documents, Excel spreadsheets, PowerPoint presentations, YouTube videos, and many other file formats into clean, AI-friendly Markdown.

You now know how to get the most out of MarkItDown to prepare data for AI workflows, regardless of whether it comes from web scraping, downloaded files, or local documents.

I hope you found this guide useful. If you have any questions or comments, feel free to leave them below. Thanks for reading, and see you in the next one!

FAQ

How to set a proxy in MarkItDown?

MarkItDown doesn’t provide a dedicated proxy configuration option. Still, it relies on the Python requests library internally, which supports proxy configuration through special environment variables. Thus, you can set a proxy in MarkItDown with:

export HTTP_PROXY="http://user:password@your-proxy-server.com:port"
export HTTPS_PROXY="http://user:password@your-proxy-server.com:port"

Or, directly in the Python code with:

import os
from markitdown import MarkItDown

# Set your proxy addresses (replace with your proxy details)
os.environ["HTTP_PROXY"] = "http://user:password@your-proxy-server.com:port"
os.environ["HTTPS_PROXY"] = "http://user:password@your-proxy-server.com:port"

md = MarkItDown()
# ...

How to deal with 429 errors when converting YouTube videos to Markdown with MarkItDown?

Try to convert a YouTube video to Markdown, such as with:

markitdown "https://www.youtube.com/watch?v=1wtQAoBZLTg" -o video.md

The request is likely to fail with 429 Too Many Requests errors due to YouTube anti-bot protections. This can happen even with a fresh IP address. To improve reliability, try to route MarkItDown requests through a high-quality residential proxy.

How to use the MarkItDown MCP server?

The MarkItDown MCP server exposes a single tool that allows AI agents and MCP-compatible applications to convert different data sources into Markdown. The tool is convert_to_markdown, which accepts http:, https:, file:, and data: URIs.

Install the MarkItDown MCP server with:

pip install markitdown-mcp

Then, connect it to an MCP-compatible client, such as Claude Code, Claude Desktop, Codex, Gemini CLI, etc.

As AI usage is growing more and more every day, you’d like an AI-powered tool that can improve your scraping productivity, don’t you?! Especially when it comes to managing selectors, and DOM structure changes…

Well, there’s good news for you: this is the exact gap Bright Data Scraper Studio is designed to close!

This article discusses what Bright Data’s Scraper Studio is, why it exists, and how to use it with a step-by-step tutorial.

Let’s get into it!

What Scraper Studio Actually Is

Scraper Studio is a cloud-hosted, end-to-end scraping environment running entirely on top of Bright Data’s infrastructure. The tool is designed from the ground up with scraping primitives in mind, as it provides built-in proxy rotation, native JavaScript rendering, anti-bot handling, and direct integrations with data delivery targets like Amazon S3, Snowflake, Google Cloud, and Microsoft Azure.

The platform offers two build modes that sit on the same underlying scraper runtime:

• AI agent mode: You provide a target URL and a plain-English description of the data you want to extract. The AI agent visits the page, infers a structured output schema, and then generates the full scraper code for you. At the end of this automated process, you review and approve the schema before code generation begins. This gives you a checkpoint to catch misaligned fields before they propagate into your pipeline, allowing you to change them.

• Full IDE mode: A full browser-based JavaScript code editor with debugging tools for when you need direct, line-level control over the scraper logic. This is the mode you can use for when you’re dealing with multi-stage scrapers, custom pagination logic, complex state management between page navigations, or sites with particularly aggressive anti-bot stacks that need fine-tuned handling.

What’s more important is that both modes produce the same type of scraper. This means that you’re never locked into one approach: a scraper built in the AI Agent mode can be opened and edited in the IDE at any time, and any scraper can be updated using the Self-healing tool.

Scraper Studio Key Features

Let’s take a close look at Scraper Studio’s features:

• Built-in proxy infrastructure: Scraper Studio runs on Bright Data’s proxy network which, among other types, provides users with 400+ million residential IPs across 195 countries, with 99.99% uptime. For scraping professionals, this means no proxy pool management and, more importantly, no dealing with burned IPs.

• Anti-bot bypass, natively integrated: JavaScript rendering, fingerprinting evasion, and bot detection bypass are built into the runtime. This is significant because it means your scraper code doesn’t need to account for these concerns. You write the extraction logic, and the platform handles the unblocking automatically.

• AI-powered self-healing: This is arguably the most operationally relevant feature for running scrapers long-term. When a target site changes its structure, and your scraper breaks, the self-healing tool lets you describe the fix in plain English. The AI analyzes the current code, proposes changes with a diff view, and you approve or reject them. This means that you remain in control of the overall process, as changes aren’t auto-applied unless you explicitly enable auto-approval

• Scheduled runs and data delivery: Scraper Studio includes a scheduling layer, so you can configure periodic runs without external orchestration. Combine this with native delivery to S3, Snowflake, GCS, or Azure, and you have a complete data pipeline that starts from scraping and ends with structured output in your data warehouse, with no spaghetti code required.

• CLI and agent integration: If you prefer working outside the browser UI, scrapers can be built and triggered via the Bright Data CLI, and the AI Agent mode is accessible programmatically, currently compatible with tools like Claude Code, Cursor, and Codex. This means you can integrate Scraper Studio into existing developer workflows without being locked into the web interface.

• Wide pricing offer: The pricing offer may seem like a secondary theme, but it’s not. First and foremost, Bright Data offers you 5k page loads for free every month, backed by their expert support if you need it. The other pricing options are:

  • Pay-as-you-go: $1.5 per 1k page loads, and you pay only for successful requests.

  • Scale: $400/Mo that includes about 400k page loads per month and more.

  • Enterprise: A custom pricing tier for enterprises, based on their specific needs.

Scraper Studio Best Use Cases

If you’re already familiar with Bright Data products and offers, you may be asking why use Scraper Studio. Below is a list of practical examples when this is the right tool to use:

• No APIs or ready-to-use datasets: If you need to scrape a site that isn’t available on Bright Data’s pre-built Datasets Marketplace or on its web scraping APIs, and you want to own the scraper logic yourself but not the infrastructure, well, Scraper Studio is your best choice.

• Recurring pipelines: When you’re building a recurring data pipeline and don’t want to manage cron jobs, deployment infrastructure, or proxy pools separately.

• Quick prototypes: The AI Agent mode can go from a single URL to a working scraper in under 10 minutes. This makes it genuinely useful for validating whether a scraping approach is feasible before committing engineering time on a larger scale.

• You’re in a team with mixed technical levels: The AI Agent is accessible to non-engineers, and the IDE gives developers full control. They both produce the same type of scraper, deployable on the same infrastructure, and as it is interchangeable, your team can manage the same project with different technical levels.

How to Build a Web Scraper With Scraper Studio Using the AI Agent Mode

Time to take the theory into practice. In this section, you’ll learn how to create a web scraper with Scraper Studio using its AI agent mode. The target URL for this tutorial is from the Kappa’s website:

The scraper will retrieve the data from this target page, using the constraints via the prompt description.

Follow the next steps to build your scraper!

Step #1: Create a New AI Web Scraper Project

After creating a new Bright Data account, go to the Scrapers tab and click on Develop a web scraper under New:

You’ll be redirected to a page where you can insert the target URL and the prompt. For this tutorial, the prompt is the following:

Extract the product name, price, available variations / colors, sizes, image URLs, product description, product details.

Below is the visual result:

Note that this is also where you can immediately go for the IDE mode by clicking on Open IDE.

When you’re ready, click on Start scraping with AI, and Scraper Studio will make its magic!

Step #2: Analyze and Approve the Output Schema

The system will automatically produce the output below:

This is the output schema. It’s a structured list of data fields the scraper will extract, along with their data types. Compare this with the data in the target page, and click Approve if the output matches your expectations.

After your approval, Scraper Studio will start creating the web scraper. When the process is done, you’ll receive the following pop-up:

You’ll also get a notification email when the scraper is ready:

All right, your scraper is ready for data retrieval.

Check the TWSC YouTube Channel

Step #3: Manage Preferences and Retrieve The Data

When the scraper is ready, this doesn’t mean the data is scraped. Before scraping the data, you have to manage your preferences.

First of all, go into your scrapers and navigate to the Delivery preferences tab:

Here is where you can choose among:

• Data format for local download, like JSON, NDJSON, CSV, or XLS.

• Third-party connections like Amazon S3 bucket, AWS, Snowflake, and more.

• Get notified via email based on the status of the data delivery.

If you want a scraper that starts recurrently, navigate to the Subscriptions tab and set your schedule:

If you just want to test things around or if you need a one-time scraper, the Initiate manually tab is already sufficient. You just need to click on Start, and the scraper will begin retrieving the data:

Very well! You made it to retrieve data with your first scraper made with Scraper Studio.

Step #4: Check Runs and Go to IDE Mode

To check the status of each scraping run, go to the Runs section. This will display all the information you need on each data retrieval performed by the scraper:

The Actions section allows you to download the data in different formats. Below is the result in JSON:

[
  {
    "product_name": "Blue Men's Trackhouse Moto GP Team Motorcycle T-Shirt with Contrast Inserts",
    "product_subtitle": "AMIRILORE TRACKHOUSE Blue Diva-Grey Antrachite-NEON Yellow",
    "brand": "KAPPA",
    "price": {
      "value": 67,
      "currency": "USD",
      "symbol": "$"
    },
    "sku": "Style: 322J6HW - Color: A02",
    "available_sizes": [
      "XS",
      "S",
      "M",
      "L",
      "XL",
      "2XL",
      "3XL",
      "4XL",
      "5XL",
      "6XL"
    ],
    "image_urls": [
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02_5f5f425b-48d1-4181-af4c-ce863229cbf4.jpg>",
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02_aa19b0fa-ca41-4894-ad74-8d408cc31725.jpg>",
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02.jpg>",
      "<https://www.kappa.com/cdn/shop/files/UB322J6HWTRHA02_8bd76dc9-0550-48f9-a974-e3c38a4b707e.jpg>"
    ],
    "description": "Product Description GARA. T-shirt for man in polyester fabric. REGULAR FIT. Short sleeve, roundcollar, insert on shoulders in contrast color, sides and shoulders inserts. Omini logo, Team personalizations and sponsors sublimated. TRACKHOUSE.",
    "product_details": "Description Active Jerseys Gender Man Season Spring Composition Main Fabric: Recycled Polyester 100%",
    "input": {
      "url": "<https://www.kappa.com/products/active-jerseys-man-amirilore-trackhouse-blue-diva-grey-antrachite-neon-yellow-322j6hw-trh-a02>"
    }
  }
]

As you can see, the results are exactly how’d you expect.

To change the mode to the full IDE one, navigate to the Code tab:

This is where you can write your custom JavaScript to modify anything you need.

Terrific! You completed your first scraping project with Bright Data’s Scraper Studio.

Need help with your scraping project?

Comparison Table: Scraping via Custom Code Vs Using Scraper Studio

The table below summarizes the differences in scraping the same target using a scraper you coded yourself or using Scraper Studio:

Conclusion

In this article, you discovered what Bright Data’s Scraper Studio is, its features and best use cases, and how easy it is to use it.

As you’ve learned, Scraper Studio is an ideal solution for building a web scraper without any infrastructure overhead, but also without managing selectors manually. Thanks to it, you don’t have to deal with common challenges like infrastructure maintenance, proxy rotation, and anti-bot bypass, but also with DOM changes.

In a matter of a few minutes, you can go from a prompt written in natural language to a production-ready scraper, with automated data delivery to Amazon S3, Snowflake, and many other destinations, and with the possibility to schedule your scrapers and manage DOM changes without manual overhead.

So, let us know in the comments: have you ever tried Scraper Studio? What’s your experience with it?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share

Retrieval Augmented Generation (RAG) was designed to solve exactly such limitations. By decoupling what the model knows from what it can access at query time, RAG gives LLMs the ability to reason over fresh, domain-specific, and proprietary knowledge without any retraining. In theory, this closes the knowledge gap entirely. In practice, the quality of a RAG system is bound entirely by the quality of its retrieval layer, and this is where most implementations break down.

Indeed, many teams think that the solution is already offered by web search tools that, nowadays, are integrated in all commercial LLMs. This article makes the case for why that is the wrong call and why web scraping is the right retrieval layer that RAG actually requires.

Let’s get into it!

Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.

Activate starter plan

---

What is RAG?

Retrieval Augmented Generation (RAG) is an architectural pattern in natural language processing that augments the inference pipeline of a generative large language model (LLM). It does so by dynamically injecting externally retrieved, contextually relevant information into the prompt at query time. In simple terms, rather than relying exclusively on knowledge encoded in model weights during training, RAG decouples parametric memory (what the model learned) from non-parametric memory (an external, queryable knowledge store), combining both at runtime. This produces models’ outputs that are accurate and grounded.

The canonical RAG pipeline operates in two distinct phases:

1. Retrieval phase: Given a user query q, a retriever component encodes q into a vector embedding database and performs approximate nearest-neighbor (ANN) search against a pre-indexed corpus of document chunks.

2. Generation phase: The retrieved passages are concatenated with the original query and injected into the LLM’s context window as an augmented prompt. The model, then, conditions its output token generation on both its parametric weights and the retrieved evidence, producing a response that is grounded in the supplied documents.

Modern RAG implementations have evolved beyond this baseline. Advanced RAG patterns introduce query rewriting, hypothetical document embeddings (HyDE), re-ranking layers, and recursive or agentic retrieval loops. Modular RAG frameworks further decompose the pipeline into interchangeable components, enabling fine-grained control over each retrieval and generation step.

At the infrastructure level, modern RAG pipelines rely on three core components working in concert:

• A vector database for storing and querying document embeddings at scale.

• An embedding model responsible for encoding both documents and queries into a shared latent semantic space.

• A generative model (the LLM) that synthesizes retrieved evidence into a coherent, task-specific response.

Below is an image that shows the high-level architecture:

Why LLMs Need RAG

During the training phase, an LLM compresses knowledge from its training corpus into billions of parameters. This process is inherently lossy because the model doesn’t store facts verbatim: it learns statistical associations between tokens. As a result of this process:

• Factual recall is probabilistic, not deterministic: The model may confidently produce a plausible-sounding but incorrect answer on a user query, because it’s sampling from a learned distribution rather than retrieving from a verifiable source. The process of producing a plausible-sounding but incorrect response is called hallucination.

• Knowledge is frozen at the training cutoff: Any event, document, regulation, product update, or research finding that postdates the model’s training window is entirely invisible to the model. For enterprise applications where data changes frequently, this is a critical failure mode. This means that if you query a model on something that occurred after its training ended, it will surely hallucinate the response.

• Specific, private, or proprietary knowledge is absent by design: Internal documentation, CRM records, legal contracts, and domain-specific corpora are never part of a pre-training dataset.

RAG was designed precisely to address these constraints that are inherent to LLMs. Below is how RAG maps to concrete business scenarios that enterprises face every day.

Documentation-Aware Support Bots

One of the most immediate and impactful enterprise applications of RAG is the construction of intelligent support bots grounded in a product’s technical documentation. Consider a SaaS company with several pages of technical docs, API references, changelogs, and troubleshooting guides. An LLM cannot reliably answer user questions about that product. What happens is that it either lacks the knowledge entirely or, worse, generates plausible-but-wrong instructions that erode user trust.

With RAG, every user query triggers a retrieval step against the indexed documentation corpus. The bot responds based on what the docs actually say, not on what the model statistically associates with similar-sounding topics.

The same pattern extends naturally to internal knowledge bases like HR policy portals, IT helpdesks, onboarding assistants, and legal FAQ systems, where the source of truth is a controlled, company-owned corpus rather than the public web.

Legal and Regulatory Intelligence on Recent Developments

The legal industry represents one of the clearest demonstrations of RAG’s necessity. Why? Because court rulings, regulatory guidance, legislative amendments, and compliance frameworks are published continuously. Legal research on an LLM is inherently dangerous because its training cutoff means the model is blind to any case law or regulation published after that date. And, in law, a single overlooked precedent can have material consequences.

RAG enables a fundamentally different architecture: legal documents, newly published court opinions, and regulatory updates are scraped, processed, and indexed into the retrieval layer as soon as they are available. When a lawyer or compliance officer queries the system, the retriever fetches the latest relevant documents, and the LLM synthesizes a grounded answer.

Competitive Intelligence and Market Monitoring

Enterprises routinely need to track competitor activity, industry trends, and market signals across news sources, press releases, job postings, and public filings. This information is inherently ephemeral because a competitor’s product launch announcement published yesterday is invisible to any LLM. By building a RAG pipeline that continuously ingests and indexes targeted public web sources, organizations can deploy analysts-in-the-loop systems that answer strategic queries based on fresh, real-world evidence rather than stale parametric knowledge.

Why Web Search Integration Is Not Enough

On the side of the architecture, the retrieval part of RAG systems can be built in different ways. The common ones are two:

• Creating a knowledge base of documents (like a plethora of PDFs and similar).

• Retrieving the needed information from the live web.

The knowledge base of documents is generally something that companies have already built over the years. So, let’s focus on the web retrieval part—which is also the most interesting one for us, scraping professionals.

Today, every commercial LLM has a web search tool integrated into it. So, at first glance, it seems to be the right tool to use for creating RAG systems. After all, if the model can query the live web, doesn’t that solve the knowledge cutoff problem entirely? The answer is: partially, and only for a narrow subset of enterprise needs. This is because the web search integration and RAG are fundamentally different mechanisms that solve different problems. Below are the main differences between the two:

• Web search operates on public web content: First of all, a web search tool cannot reach your internal knowledge base, your proprietary documentation, your CRM history, or your company’s legal contracts. Sure, some companies put their internal knowledge on the web with restricted access, but this encompasses only a few cases. Also, the truth is that no company actually puts all its documentation on a website, even if with restricted access (think of legal contracts, for example).

• Web search is non-deterministic and uncontrolled by design: When an LLM issues a search query, the results it retrieves depend on the search engine’s ranking algorithm, which is commercially influenced and optimized for general web traffic, not for your specific domain. In a RAG system, instead, you control the corpus. In other words, you decide what documents are indexed, how they are chunked, how they are weighted, and when they are refreshed. This level of governance is not optional in regulated industries where compliance is a requirement. Indeed, an enterprise cannot defend an AI-generated legal or financial output that was grounded in whatever Google happened to rank first. Also, no one guarantees that the web search is actually able to read the content on a website that is highly protected. In other words, very often happens that the web search tool can’t overcome the anti-bot systems on a target website, and the model simply hallucinates the response to your query (and you don’t even notice!).

• The retrieval mechanism itself is categorically different: Web search tools rely on keyword-based and PageRank-style relevance signals. RAG retrieval, by contrast, operates in a dense semantic vector space. This means that it finds documents that are conceptually relevant to a query even when they share no exact keywords with it. For domain-specific queries, this distinction in retrieval precision directly impacts the quality of the generated response.

• Web search tools introduce latency, rate limits, and external dependencies: A RAG system backed by an internal vector store is a self-contained, auditable, scalable infrastructure component. A system that depends on a web search tool at every call is coupled to external services, subject to availability failures, and difficult to reproduce or audit

• Web search tools don’t solve the hallucination problem: Web search tools increase the probability that relevant information surfaces in the context window, but if the search query is poorly formed, if the right page isn’t indexed, or if the model misreads the retrieved snippet, hallucination remains entirely possible. RAG, when implemented with rigorous chunking strategies, high-quality embeddings, and re-ranking layers, provides a far more controlled and reliable grounding mechanism.

In short, web search tools make general-purpose LLMs more useful for casual queries. This means that an LLM equipped with a web search tool is useful to respond to basic queries that rely on data recently published on the web. However, they can’t replace RAG systems for any organization that requires precision, privacy, control, and auditability over wide and specific knowledge.

Check the TWSC YouTube Channel

Web Scraping as the Data Acquisition Layer for RAG

The limitations of web search tools presented in the previous paragraph are structural. The solution to overcome them is to use web scraping as the retrieval layer for your RAG pipelines. This is because by using web scraping, you are not querying an external search engine and accepting whatever it returns. You get direct, programmatic access to the content of any publicly accessible web page, on your own schedule, under your own control.

So, let’s see how scraping the web changes the dynamic of retrieving content from the web for RAG purposes.

Full Control Over the Knowledge Corpus

When you build a RAG pipeline fed by a web scraper, you are the one defining the boundaries of your knowledge base. You decide which domains, subdomains, and URL patterns are in scope. You decide how frequently each source is re-crawled, which pages are included, which are filtered out, and which are prioritized during indexing. None of this is possible with web search integration, where the retrieval scope is, by definition, the entire public web as ranked by an opaque external algorithm.

This level of corpus governance is a prerequisite for any enterprise deployment where the knowledge base must be auditable, bounded, and defensible.

Also, using web scraping as the retrieval layer doesn’t mean that you have to code and manage your own scraping infrastructure—which could cause overheads as well as a huge amount of investments. The scraping industry today is mature enough to offer you:

• Pre-scraped datasets whose data can be refreshed based on your needs.

• Scraping infrastructure as-a-service. In other words, you tell the provider the content to scrape, and their professionals do the job for you, providing you with ready-for-analytics data.

Accessing Content That Search Engines Cannot Deliver

Public search engines index a fraction of the publicly accessible web, and they do so on their own timeline and according to their own crawl priorities. Pages that were published recently, that exist on domains with low authority, that are dynamically rendered via JavaScript, or that are structured in ways that resist standard crawling are routinely underrepresented or absent from search engine indexes entirely.

A purpose-built web scraper is not subject to any of these constraints. It can be engineered to handle JavaScript-rendered single-page applications using headless browser automation tools such as Playwright or Puppeteer, capturing the fully hydrated DOM rather than the bare HTML shell that a naive HTTP request would return. It can navigate paginated content, follow internal links recursively to achieve full site coverage, and process non-HTML formats.

Now, you can also think that a plausible solution is to put a target URL into the prompt and ask the LLM to retrieve the content via the web search tool. But due to the nature of the web search tool, this is not a reliable solution, because, as discussed previously:

• The web search tool may not be able to retrieve the content if the domain is highly protected by anti-bots.

• The use of a web search tool is non-deterministic due to how LLMs work. This means that the LLM can respond without actually retrieving the content. Or, in some cases, the LLM can respond, but its response can be a mix of content retrieved from the target URL and other content that the web search tool has retrieved from the web. In other words, the LLM can autonomously decide to retrieve the content, for a given query, from the target URL you passed into the prompt, and from other content that the web search tool retrieves from the web, based on the query.

But for RAG pipelines that need comprehensive, deep coverage of a specific web domain, the right degree of control over the crawling behavior is a fundamental feature, often a non-negotiable one.

Need help with your scraping project?

Deterministic, Reproducible Retrieval

One of the most underappreciated properties of a scraping-fed RAG system is reproducibility. When a user queries a RAG system backed by a web-scraped and version-controlled vector store, the retrieval behavior is deterministic: the same query against the same index returns the same results. This makes the system testable, debuggable, and auditable in ways that a live-search-grounded system categorically is not.

In a search-integrated LLM, the retrieved content is ephemeral. Specifically speaking, the content exists only in the context window of a single inference call. This means that it cannot be logged in a structured way, and may differ entirely if the same query is issued five minutes later because the search ranking shifted. In a scraping-fed RAG system, instead, every document in the index has a known origin, a known ingestion timestamp, and a known embedding. If an output is questioned, you can trace it back to the exact chunk, the exact source page, and the exact crawl that produced it. This traceability is the operational backbone of trustworthy enterprise AI.

Freshness on Your Terms

The knowledge cutoff problem that cripples base LLMs does not disappear with a static RAG corpus. A vector store that was indexed six months ago and never refreshed is, from a knowledge currency standpoint, equivalent to a model with a six-month-old training cutoff. Web scraping solves this by making freshness a configurable, automated pipeline.

A production RAG scraping pipeline typically implements a tiered crawl schedule calibrated to the update frequency of each source: high-volatility sources like breaking news feeds, regulatory update pages, earnings release portals, and product changelog endpoints are re-crawled on a schedule ranging from hourly to daily. Lower-volatility sources may be re-crawled weekly or triggered by change detection signals. When new content is detected, only the affected chunks are re-embedded and re-indexed, making incremental updates computationally efficient even at scale.

This architecture gives enterprises something that neither a base LLM nor a search-integrated model can offer: a knowledge base with a freshness SLA. This is a contractual guarantee that the information grounding the system’s outputs is no older than a defined threshold.

Clean, Structured Text as a First-Class Output

Web scraping, when implemented correctly, does not simply dump raw HTML into the processing pipeline. A production-grade scraping pipeline extracts clean, structured, semantically meaningful text from the web.

This matters enormously for chunking quality, and by extension, for retrieval quality. Recall that the integrity of every chunk is bounded by the integrity of the raw text it was derived from. A scraper that faithfully extracts clean, well-structured prose gives the downstream chunker the structural signals it needs to produce semantically coherent chunks. A scraper that emits raw HTML soup, or that fails to distinguish between primary content and navigational boilerplate, contaminates the entire corpus with noise that no chunking strategy can fully recover from.

Furthermore, this is another point of distinction with web search tools. In another article here at the web scraping club, we already covered the importance of feeding LLMs in markdown, rather than in HTML. Based on this, consider that:

• Every scraping pipeline can be managed to return data in any format you prefer. It can be a structured JSON, markdown, or anything else.

• Web search tools go directly to “read” the HTML of the target web pages. This means that the LLM responds based also on footers, banners, and anything in between. And this can lead the model to confusion as it responds to users’ queries.

Conclusion

RAG is an architectural commitment and, as such, it must be preceded by a deliberate choice to build a system where outputs are grounded in a controlled and continuously refreshed knowledge base. But that commitment is only as strong as the pipeline feeding it.

Web search integration tools are not that pipeline. This is because they are optimized for general-purpose queries, not for the precision, governance, and reproducibility that enterprises demand.

Web scraping, on the other hand, gives you full ownership over what enters the knowledge base, how fresh it is, how it is structured, and how it can be traced back when an output is questioned.

The next step is to start thinking of your scraping infrastructure not as a standalone data collection tool, but as the retrieval backbone of an intelligent system.

So, let us know in the comments: are you using web scraping for RAG purposes?

Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.

Share