<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[The Web Scraping Club]]></title><description><![CDATA[News, solutions and interviews about web scraping.
In this substack you will find weekly content about:
- Web Scraping techniques
- Interviews with key people in the industry
- Anti bot infos and counter measures
- Real world examples and code]]></description><link>https://www.scraping.club</link><image><url>https://substackcdn.com/image/fetch/$s_!gJt2!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F1e343ec9-7946-4440-8c00-57209a1d99a1_1024x1024.png</url><title>The Web Scraping Club</title><link>https://www.scraping.club</link></image><generator>Substack</generator><lastBuildDate>Thu, 19 Nov 2026 02:23:23 GMT</lastBuildDate><atom:link href="https://www.scraping.club/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[The Web Scraping Club SRL]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[pier@thewebscraping.club]]></webMaster><itunes:owner><itunes:email><![CDATA[pier@thewebscraping.club]]></itunes:email><itunes:name><![CDATA[Pierluigi Vinciguerra]]></itunes:name></itunes:owner><itunes:author><![CDATA[Pierluigi Vinciguerra]]></itunes:author><googleplay:owner><![CDATA[pier@thewebscraping.club]]></googleplay:owner><googleplay:email><![CDATA[pier@thewebscraping.club]]></googleplay:email><googleplay:author><![CDATA[Pierluigi Vinciguerra]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Reading StockX’s internal API with the Decodo Web Data API]]></title><description><![CDATA[Justifying my LEGO addiction with the Decodo Web Data API test]]></description><link>https://www.scraping.club/p/reading-stockxs-internal-api-decodo-web-data-api</link><guid isPermaLink="false">https://www.scraping.club/p/reading-stockxs-internal-api-decodo-web-data-api</guid><dc:creator><![CDATA[Pierluigi Vinciguerra]]></dc:creator><pubDate>Wed, 18 Nov 2026 18:02:38 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/691e0801-ada5-4517-911f-09ce5556731d_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>LEGO sets are one of the main passions of my youth (and not only). I have a couple of them in my actual office, tons of them still in my parents&#8217; house. In case there&#8217;s someone working at LEGO in the audience, a sponsorship paid in bricks would be seriously taken into consideration. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vxCY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vxCY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg 424w, https://substackcdn.com/image/fetch/$s_!vxCY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg 848w, https://substackcdn.com/image/fetch/$s_!vxCY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!vxCY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!vxCY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg" width="1456" height="1941" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1941,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:278659,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/225339262?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!vxCY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg 424w, https://substackcdn.com/image/fetch/$s_!vxCY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg 848w, https://substackcdn.com/image/fetch/$s_!vxCY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!vxCY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ccf209e-019d-4aaa-8817-d1bc9acb5a6a_1536x2048.jpeg 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The LEGO set in my office</figcaption></figure></div><p>This week I also bought my 1-year-old son his first DUPLO set. I was returning from the Extract Summit and, as every parent who stayed far from his child, felt guilty about it. </p><p>Marketing teams working for the shops in the airport know this feeling very well, so they put toys at every corner, and I almost fell for it.  I&#8217;ve found a nice DUPLO set, around 29 EUR, but it was too big to fit in my backpack, and since I needed to travel with Ryanair, I preferred not to trigger the well-known money thirst of their ground personnel when they see extra baggage (sorry guys, I know you&#8217;re incentivized to behave like that).<br>Anyway, I didn&#8217;t buy it; I shared the link to the game on the LEGO website with my wife, and there it was costing 25 EUR. But I was no longer feeling guilty, so I didn&#8217;t purchase it. </p><p>Then a couple of days after, I saw the same toy inside a supermarket, at 17 EUR. Now it&#8217;s no longer a guilty feeling driving me, but the fact that I&#8217;ve found a bargain. </p><p>Game bought, child happy with his new colorful train, and life goes on as good as always. </p><p>So, when I was thinking about how to test the new <a href="https://decodo.com/deals/free-plan-web-data-api?utm_medium=affiliate&amp;utm_source=The+Web+scraping+club+srl&amp;promoter_id=15855491&amp;fpr=3966225">Decodo Web Data API</a>, I suddenly found an inspiration in creating a small and naive price comparison tool for LEGO sets, to answer a specific question: is there still room for LEGO resellers, buying from one website and selling to another one?</p><p>Trying to answer this question could be a good battleground for the Web Data API, so I&#8217;ve selected two websites to compare data from: eBay and StockX. Both of them are protected by anti-bots, and they&#8217;re a great use case for some of the features I wanted to test more. <br>But first, let&#8217;s see the product in details.</p><h2>What the Decodo Web Data API is</h2><p>The Web Data API is one base URL, <code>https://data.decodo.com/v1</code>, with one API key sent as a bearer token. It has four endpoints.</p><ul><li><p><code>/scrape</code> takes a URL and returns the page as HTML, Markdown, a screenshot, or the list of XHR calls.</p></li><li><p><code>/search</code> takes a query and returns a Google results page as JSON.</p></li><li><p><code>/map</code> takes a URL and returns the URLs of the website, without downloading them.</p></li><li><p><code>/crawl</code> follows the links from a starting URL and downloads the pages.</p></li></ul><p>I used the first three for this article and left <code>/crawl</code> out of the test. </p><p>Before we start, just keep in mind that this product replaces the older Decodo Web Scraping API and Fast Search API. </p><p>The price is per 1,000 successful requests and depends on two choices: the proxy pool (<code>standard</code> or <code>premium</code>) and JavaScript rendering. On the free and $19 plans, according to the <a href="https://help.decodo.com/docs/web-data-api-billing">billing page</a>, a standard request costs $0.50 per 1,000, a premium one $1.00, and a premium one with JavaScript $1.50. The default pool is premium. <code>/search</code> costs $0.75 per 1,000.</p><p>You can try it without paying. There&#8217;s a <a href="https://decodo.com/deals/free-plan-web-data-api?utm_medium=affiliate&amp;utm_source=The+Web+scraping+club+srl&amp;promoter_id=15855491&amp;fpr=3966225">free plan</a> with all four endpoints and no credit card required. According to the product page, it gives you $5 of credit, or up to 10K requests every month, with a rate limit of 10 requests per second. For reference, all the tests of this article took about 1,800 calls. My dashboard shows a spend of $1.76 for the scraped websites, so the whole work fits in the free credit.</p><p>For all the tests I wrote a thin client around <code>requests</code>. It adds the bearer header, retries the failed calls and logs every call (status, latency, bytes) to a JSONL file. This is the whole <code>client.py</code>, and all the other scripts of this article import it.</p><pre><code><code>"""Minimal Decodo Web Data API client (https://data.decodo.com/v1).

Every call is appended to artifacts/calls.jsonl with status, latency and payload size,
so the article numbers come from logged runs.
"""

import json
import os
import random
import time
from pathlib import Path

import requests
from dotenv import load_dotenv

HERE = Path(__file__).parent
load_dotenv(HERE / ".env")

API = "https://data.decodo.com/v1"
RETRYABLE = {408, 429, 500, 502, 503, 504, 524}
ARTIFACTS = HERE / "artifacts"
LOG = ARTIFACTS / "calls.jsonl"
ARTIFACTS.mkdir(exist_ok=True)


class WebDataError(RuntimeError):
    pass


def _headers() -&gt; dict:
    key = os.environ.get("DECODO_WEB_DATA_API_KEY", "").strip()
    if not key:
        raise WebDataError("DECODO_WEB_DATA_API_KEY is not set")
    return {
        "Authorization": f"Bearer {key}",
        "Content-Type": "application/json",
        "Accept": "application/json",
    }


def _log(entry: dict) -&gt; None:
    with LOG.open("a") as f:
        f.write(json.dumps(entry) + "\n")


def call(method: str, path: str, payload: dict | None = None, attempts: int = 3) -&gt; dict:
    for attempt in range(1, attempts + 1):
        t0 = time.time()
        try:
            resp = requests.request(method, f"{API}{path}", json=payload, headers=_headers(), timeout=180)
        except (requests.Timeout, requests.ConnectionError) as exc:
            _log({"ts": t0, "path": path, "payload": payload, "status": type(exc).__name__, "attempt": attempt})
            if attempt == attempts:
                raise WebDataError(f"{path} failed after {attempts} attempts: {exc}") from exc
            time.sleep(2**attempt + random.random())
            continue
        elapsed = round(time.time() - t0, 2)
        try:
            body = resp.json()
        except ValueError:
            body = {"raw": resp.text[:1000]}
        first = {}
        if isinstance(body, dict) and isinstance(body.get("results"), list) and body["results"]:
            first = body["results"][0] if isinstance(body["results"][0], dict) else {}
        _log({
            "ts": t0, "path": path, "payload": payload, "status": resp.status_code,
            "attempt": attempt, "elapsed_s": elapsed, "bytes": len(resp.content),
            "task_id": first.get("task_id"), "target_status": first.get("status_code"),
            "api_status_code": body.get("status_code") if isinstance(body, dict) else None,
            "error": None if resp.ok and "results" in body else resp.text[:300],
        })
        # A failed scrape comes back as HTTP 200 with {"status": "failed", "status_code": 613, ...}
        failed = isinstance(body, dict) and "results" not in body and body.get("status") == "failed"
        if resp.ok and not failed:
            return body
        if failed:
            if attempt == attempts:
                raise WebDataError(f"{path} -&gt; scrape failed {body.get('status_code')}: {body.get('message')}")
            time.sleep(1 + random.random())
            continue
        if resp.status_code not in RETRYABLE or attempt == attempts:
            raise WebDataError(f"{path} -&gt; {resp.status_code}: {resp.text[:500]}")
        time.sleep(2**attempt + random.random())
    raise WebDataError("unreachable")


def scrape(url: str | None = None, attempts: int = 3, **params) -&gt; dict:
    payload = dict(params)
    if url:
        payload["url"] = url
    return call("POST", "/scrape", payload, attempts=attempts)


def map_site(url: str, attempts: int = 2, **params) -&gt; dict:
    return call("POST", "/map", {"url": url, **params}, attempts=attempts)


def search(query: str, attempts: int = 2, **params) -&gt; dict:
    return call("POST", "/search", {"query": query, **params}, attempts=attempts)</code></code></pre><p>I had to add the <code>"status": "failed"</code> part after a first run, as you&#8217;ll see when I talk about eBay results. With the client in place, a first call looks like this.</p><pre><code><code>from client import scrape

body = scrape("https://ip.decodo.com")
result = body["results"][0]
print(result["status_code"], result["type"], len(result["content"]))
# 200 raw 49377</code></code></pre><h2>The xhr parameter on StockX</h2><p>The first page I wanted to extract data from is the list of <a href="https://stockx.com/brands/lego?category=collectibles&amp;order=DESC&amp;sort=highest_bid">LEGO collectibles on StockX</a>, sorted by highest bid. StockX is a marketplace with bids and asks, just like a stock exchange, so for every product it knows the highest bid, the lowest ask, and the last sale. Seems a great way to understand the market price for a LEGO set.</p><p>The request is the same <code>/scrape</code> call as before, with one more field.</p><pre><code><code>{
  "url": "https://stockx.com/brands/lego?category=collectibles&amp;order=DESC&amp;sort=highest_bid",
  "xhr": true,
  "geo": "Italy"
}</code></code></pre><p>With <code>xhr</code> set to true, the <code>content</code> field of the response becomes an array, with one object for every call the page made. Each object has <code>url</code>, <code>method</code>, <code>status_code</code>, <code>request_headers</code>, <code>request_payload</code>, <code>response_headers</code> and <code>response_body</code>.</p><p>Just like when you open your network tab in the browser,  most of the requests you&#8217;ll see are what you&#8217;d expect from a modern website: the cookie banner, Datadog, ad networks, a Cloudflare challenge call. In the middle there were six POST calls to <code>https://gateway.stockx.com/api/graphql</code>, and one of them had a response body of 122 KB. Its <code>request_payload</code> says the operation is <code>getDiscoveryData</code>, and the response contains the 40 products of the page.</p><pre><code><code>{
  "title": "LEGO Marvel X-Men: The X-Mansion Set 76294",
  "model": "Set 76294",
  "condition": "New",
  "market": {
    "state": {
      "highestBid": {"amount": 353},
      "lowestAsk": {"amount": 384}
    },
    "statistics": {
      "last90Days": {"averagePrice": 469, "salesCount": 242},
      "lastSale": {"amount": 506}
    }
  }
}</code></code></pre><p>So yes, with one parameter you get the internal API responses of a protected website and its requests too. In the payload I could read the variables of the GraphQL query (country, currency, page index, filters, sort) and the persisted query hash.</p><p>Some interesting points about the API usage: </p><ul><li><p>The billing confirms that <code>xhr</code> renders the page anyway. When I filter my dashboard by requests with JavaScript enabled, all the 14 StockX calls are there, and only 2 of them had <code>headless</code> in the payload. So you should count an <code>xhr</code> request at the JavaScript rate, which is $1.50 per 1,000 on the smaller plans.</p></li><li><p>The second is that there&#8217;s no filter. You get every call, analytics included, and you look for yours in the list. There&#8217;s a browser action called <code>fetch_resource</code> that takes a regex and returns only the first call that matches. I didn&#8217;t use it, because I wanted to see everything.</p></li><li><p>The third is the role of <code>geo</code>. My first request had no <code>geo</code>, the exit IP was in Turkey, and the GraphQL variables said <code>country: TR, currency: USD</code>. With <code>geo: "Italy"</code> they became <code>country: IT, currency: EUR</code>. So the proxy location decides which market StockX shows you, and for a price comparison with eBay Italy I need the Italian one.</p></li></ul><h3>Ten pages of the catalog</h3><p>The code that reads a page is short. <code>BASE</code> is the URL of the sorted list and <code>GEO</code> is &#8220;Italy&#8221;. We ask for the page, we look for the <code>getDiscoveryData</code> call in the list, and we parse its body.</p><pre><code><code>def fetch_page(page: int, tries: int = 3) -&gt; dict:
    url = BASE if page == 1 else f"{BASE}&amp;page={page}"
    last = "no attempt"
    for attempt in range(1, tries + 1):
        t0 = time.time()
        try:
            body = scrape(url, attempts=1, xhr=True, geo=GEO)
        except WebDataError as exc:
            last = str(exc)[:200]
            continue
        calls = body["results"][0]["content"]
        for c in calls if isinstance(calls, list) else []:
            if "gateway.stockx.com/api/graphql" not in c.get("url", ""):
                continue
            req = json.loads(c.get("request_payload") or "{}")
            if req.get("operationName") != "getDiscoveryData":
                continue
            data = json.loads(c["response_body"]).get("data") or {}
            browse = data.get("browse")
            if not browse:
                continue
            return {
                "page": page, "attempt": attempt, "elapsed_s": round(time.time() - t0, 1),
                "xhr_calls": len(calls), "variables": req["variables"], "browse": browse,
            }
        last = f"getDiscoveryData not in {len(calls)} captured calls"
    return {"page": page, "error": last}</code></code></pre><p>Then a second function turns the GraphQL response into flat rows. The set number is in the <code>model</code> field (&#8221;Set 76294&#8221;), and the prices are under <code>market</code>.</p><pre><code><code>def rows(result: dict) -&gt; list[dict]:
    out = []
    for pos, edge in enumerate(result["browse"]["results"]["edges"], 1):
        n = edge["node"]
        state = (n.get("market") or {}).get("state") or {}
        stats = (n.get("market") or {}).get("statistics") or {}
        m = re.search(r"\d{4,7}", n.get("model") or "") or re.search(r"\b\d{4,7}\b", n.get("title") or "")
        out.append({
            "page": result["page"], "pos": pos, "set_number": m.group(0) if m else None,
            "model": n.get("model"), "title": n.get("title"), "url_key": n.get("urlKey"),
            "condition": n.get("condition"), "is_ad": bool(edge.get("isAd")),
            "highest_bid": (state.get("highestBid") or {}).get("amount"),
            "lowest_ask": (state.get("lowestAsk") or {}).get("amount"),
            "last_sale": (stats.get("lastSale") or {}).get("amount"),
            "sales_90d": (stats.get("last90Days") or {}).get("salesCount"),
            "avg_90d": (stats.get("last90Days") or {}).get("averagePrice"),
        })
    return out</code></code></pre><p>I ran it on the first 10 pages. All 10 returned the GraphQL response, and one of them needed a second attempt. A call took between 21 and 82 seconds, with most of them around 25.</p><p>The result is 400 products (397 different set numbers) with highest bid, lowest ask, last sale and the sales of the last 90 days, all with &#8220;New&#8221; conditions. </p><p>Curious to know which is the most expensive? <a href="https://stockx.com/lego-star-wars-cloud-city-set-10123">A vintage Star Wars theme set, Cloud City, with a bid of 1,821 euros.</a></p><h2>eBay Italy and the browser check</h2><p>For every set number, I wanted the first page of an eBay.it search to read price and condition from the result cards, without opening the detail pages. This means 397 searches, and I started with the cheapest configuration.</p><p>In my first probes the standard proxy pool failed, with and without JavaScript. The premium pool without JavaScript returned the results page in about 3 seconds. So I launched the full run with premium and no JS, and here I found the two most important things of this test.</p><h3>A failed scrape is an HTTP 200</h3><p>When Decodo can&#8217;t get a page, the HTTP status of the API response is still 200. The error is in the body.</p><pre><code><code>{
  "status": "failed",
  "status_code": 613,
  "message": "We were not able to scrape the target. You can try the following: 1) Switch between JS and non-JS rendering, 2) Retry the request, 3) Switch to a different geo-location.",
  "task_id": "7528388964445613057"
}</code></code></pre><p>My client crashed on the first one, because it looked for <code>results</code> in a response that had none. If you check only <code>response.ok</code>, you&#8217;ll make the same mistake. The good news is that a 613 isn&#8217;t billed. </p><h3>A block page is a success</h3><p>The second thing is worse, in my opinion, but it&#8217;s not a fault of the tool but a trick website behavior. In many responses, the status was 200, <code>results</code> was there, and the content was a page of 13 KB with the title &#8220;Ci scusiamo per l&#8217;interruzione&#8221; (sorry for the interruption). It&#8217;s the browser check that eBay shows before the real page. For the API this is a success, and the <a href="https://help.decodo.com/docs/web-data-api-response-codes">response codes page</a> says that a 200 is billed.</p><p>So I added my own check on the content, with up to 5 attempts for every search. These are the numbers of the full run: 874 calls for 397 searches.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1l6y!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1l6y!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png 424w, https://substackcdn.com/image/fetch/$s_!1l6y!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png 848w, https://substackcdn.com/image/fetch/$s_!1l6y!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png 1272w, https://substackcdn.com/image/fetch/$s_!1l6y!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1l6y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png" width="1456" height="658" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:658,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:185287,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/225339262?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!1l6y!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png 424w, https://substackcdn.com/image/fetch/$s_!1l6y!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png 848w, https://substackcdn.com/image/fetch/$s_!1l6y!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png 1272w, https://substackcdn.com/image/fetch/$s_!1l6y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa2f40241-cc9c-4273-b5ec-d80db629e52d_3200x1446.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Only 40% of the calls returned data, and 48 searches completed the browser check on all 5 attempts. At the list price of $1.00 per 1,000, the 408 check pages are about 40 cents, so the money is a small problem. The bigger one is the data: without a check on the content, almost half of my responses would have been empty pages marked as good, which means you&#8217;ll be billed for them. Not a big issue for this test, but keep it in mind if you plan to use the API on a larger scale.</p><h3>JavaScript rendering fixes it</h3><p>The error message of the 613 suggests to switch between JS and non-JS rendering, so I tried <code>headless: "html"</code> on the 48 missing searches. All 48 came back with the listings.</p><p>But at that point I had made hundreds of calls, and the share of 613 errors had grown during the run. To be sure that the difference came from the rendering, I ran a control: 24 searches without JS and the same 24 with JS, one after the other, at the same concurrency.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!O64H!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!O64H!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png 424w, https://substackcdn.com/image/fetch/$s_!O64H!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png 848w, https://substackcdn.com/image/fetch/$s_!O64H!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png 1272w, https://substackcdn.com/image/fetch/$s_!O64H!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!O64H!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png" width="1456" height="623" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:623,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:163943,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/225339262?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!O64H!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png 424w, https://substackcdn.com/image/fetch/$s_!O64H!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png 848w, https://substackcdn.com/image/fetch/$s_!O64H!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png 1272w, https://substackcdn.com/image/fetch/$s_!O64H!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8717d8cc-6bec-4f43-b8f5-727ece23a6a2_3200x1370.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://bundle.scraping.club/&quot;,&quot;text&quot;:&quot;Scraping Bundle - 20k+$ in tools&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://bundle.scraping.club/"><span>Scraping Bundle - 20k+$ in tools</span></a></p><div><hr></div><p>With the rendering, 23 calls out of 24 returned the real page. Then I ran all 397 searches again, only with JS and with up to 3 attempts. It took 393 calls: 378 real pages, 11 browser checks, and 4 errors, with a median page time of 5 seconds, and the run ended in 8.5 minutes with 6 workers. At the end of the loop, 395 of 397 searches had their page.</p><p>The 4 errors were something new: an HTTP 429 that arrives in 0.2 seconds with the message &#8220;Requests for this page type are temporarily limited. Please try again later.&#8221;, so I was rate-limited.</p><p>The dashboard lets you filter the requests by JavaScript rendering. So at the end I compared its numbers for ebay.it with the real result pages in my log, over all the eBay calls of the session.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HAA6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HAA6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png 424w, https://substackcdn.com/image/fetch/$s_!HAA6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png 848w, https://substackcdn.com/image/fetch/$s_!HAA6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png 1272w, https://substackcdn.com/image/fetch/$s_!HAA6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HAA6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png" width="1456" height="672" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:672,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:240621,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/225339262?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HAA6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png 424w, https://substackcdn.com/image/fetch/$s_!HAA6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png 848w, https://substackcdn.com/image/fetch/$s_!HAA6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png 1272w, https://substackcdn.com/image/fetch/$s_!HAA6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2f48a65f-0072-4ebf-8240-48d7511e54b5_3200x1478.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><h2>Parsing the results</h2><p>The last piece is the parser. Decodo has target templates with ready parsers for websites like Amazon, Google, Walmart, and TikTok, but eBay isn&#8217;t in the list. I also tried <code>parse: true</code> on the eBay URL, and the API answered with a validation error. So I asked Claude to build the boring part, and it used BeautifulSoup, something that I&#8217;d never use, but that&#8217;s it; it&#8217;s not the core of the article. A good old Python script with XPATH selector would have been way more readable.</p><pre><code><code>PRICE = re.compile(r"EUR\s*([\d.]+,\d{2})")


def eur(text: str) -&gt; list[float]:
    return [float(p.replace(".", "").replace(",", ".")) for p in PRICE.findall(text or "")]


def parse(html: str) -&gt; dict:
    soup = BeautifulSoup(html, "lxml")
    heading = soup.select_one(".srp-controls__count-heading")
    listings, seen = [], set()
    for card in soup.select("li.s-card"):
        title_el = card.select_one(".s-card__title .su-styled-text") or card.select_one(".s-card__title")
        price_el = card.select_one(".s-card__price")
        title = title_el.get_text(" ", strip=True) if title_el else ""
        lid = card.get("data-listingid")
        if not price_el or title == "Shop on eBay" or lid in seen:
            continue
        seen.add(lid)
        prices = eur(" ".join(p.get_text(" ", strip=True) for p in card.select(".s-card__price")))
        sub = card.select_one(".s-card__subtitle")
        subtitle = sub.get_text(" ", strip=True) if sub else ""
        attrs = [a.get_text(" ", strip=True) for a in card.select(".s-card__attribute-row")]
        shipping = next((a for a in attrs if "consegna" in a.lower() or "spedizione" in a.lower()), None)
        listings.append({
            "id": lid, "title": title,
            "condition": subtitle.split("|")[0].strip() or None,
            "seller_type": subtitle.split("|")[1].strip() if "|" in subtitle else None,
            "price": prices[0] if prices else None,
            "price_max": prices[-1] if len(prices) &gt; 1 else None,
            "price_text": price_el.get_text(" ", strip=True),
            "shipping_text": shipping,
            "shipping": (eur(shipping) or [0.0 if "gratis" in shipping.lower() else None])[0] if shipping else None,
            "from": next((a[3:] for a in attrs if a.startswith("da ")), None),
            "auction": any("offert" in a.lower() and "proposta" not in a.lower() for a in attrs),
        })
    count_heading = heading.get_text(" ", strip=True) if heading else None
    m = re.match(r"([\d.]+)\+?\s+risultat", count_heading or "")
    total = int(m.group(1).replace(".", "")) if m else None
    # eBay pads short result lists with looser matches; keep only the cards covered by the result count
    if total is not None:
        listings = listings[:total]
    return {"count_heading": count_heading, "total_results": total, "listings": listings,
            "challenge": "Ci scusiamo per l'interruzione" in (soup.title.get_text() if soup.title else "")}</code></code></pre><p>And this is the function of the final run, with the check on the content. <code>PLAN</code> is a list of three <code>{"headless": "html"}</code>, one for every attempt, and <code>ebay</code> is the module with the parser above. A response counts only if the page has the results heading of eBay.</p><pre><code><code>def run(set_number: str) -&gt; dict:
    path = OUT / f"{set_number}.json"
    if path.exists():
        return json.loads(path.read_text())
    url = f"https://www.ebay.it/sch/i.html?_nkw=lego+{set_number}&amp;_sacat=19006&amp;_ipg=240"
    for attempt, extra in enumerate(PLAN, 1):
        mode = "js" if extra else "no_js"
        t0 = time.time()
        r = parsed = None
        try:
            body = scrape(url, attempts=1, proxy_pool="premium", geo="Italy", **extra)
            r = body["results"][0]
            parsed = ebay.parse(r["content"])
            outcome = "challenge" if parsed["challenge"] else ("page" if parsed["count_heading"] else "other")
        except WebDataError as exc:
            outcome = "613" if "613" in str(exc) else "error"
        elapsed = round(time.time() - t0, 1)
        with lock, ATTEMPTS_LOG.open("a") as f:
            f.write(json.dumps({"set": set_number, "attempt": attempt, "mode": mode, "outcome": outcome, "s": elapsed}) + "\n")
        if outcome == "page":
            out = {"set_number": set_number, "url": url, "status_code": r["status_code"], "task_id": r["task_id"],
                   "attempts": attempt, "mode": mode, "elapsed_s": elapsed, "html_bytes": len(r["content"]), **parsed}
            path.write_text(json.dumps(out, ensure_ascii=False))
            return out
    return {"set_number": set_number, "error": f"no result page in {len(PLAN)} attempts"}</code></code></pre><h2>Comparing the prices</h2><p>My first query was &#8220;lego set 10123&#8221;, and to be honest the results were useless. A search like this on eBay returns mostly spare parts, minifigures and instruction manuals, all with the set number in the title. The cheapest &#8220;new&#8221; listing for the Cloud City was 1.43 euros.</p><p>So in the final run I changed two things. In the URL, I added the category of the complete sets (<code>_sacat=19006</code>, &#8220;Set e pacchetti completi&#8221;) and 240 results per page. Sellers put spare parts in that category too, so I added a filter on the title. I keep a card only if the title has the set number, no other set number, and none of the words that mark a partial item (ref, minifig, istruzioni, LED, and so on). Again Claude came with a boring but effective solution.</p><pre><code><code>PARTIAL = re.compile(
    r"\b(ref|minifig\w*|figur\w*|personaggi\w*|istruzion\w*|manual\w*|libretto|anleitung|bauanleitung|instructions?|solo|"
    r"scatola vuota|box vuot\w*|empty box|leerkarton|pezz[oi]|ricamb\w*|parti|part[es]?|teile|adesiv\w*|stickers?|"
    r"aufkleber|led|luci|light\w*|beleuchtung|illuminazione|teca|vetrina|display|acrylic|acrilic\w*|espositore|stand|"
    r"supporto|compatibil\w*|moc|custom|polybag|sacchett\w*|bag|busta|incomplet\w*|senza|poster|depliant)\b", re.I)


def complete_set(title: str, num: str) -&gt; bool:
    if not re.search(rf"(?&lt;!\d){num}(?!\d)", title):
        return False
    others = [n for n in re.findall(r"(?&lt;!\d)\d{4,7}(?!\d)", title) if n != num and not n.startswith(("19", "20"))]
    return not others and not PARTIAL.search(title)</code></code></pre><p>Out of 53,604 cards, the filter keeps 18,217. For every set I take the median price by condition, because the minimum is still dirty. For the Millennium Falcon 75192, the cheapest &#8220;new&#8221; listing that passes the filter is a 29.90 euros chess board from the set, while the median of the 91 new listings is 1,000 euros.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://substack.thewebscraping.club/p/consulting&quot;,&quot;text&quot;:&quot;Need help with your scraping project?&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://substack.thewebscraping.club/p/consulting"><span>Need help with your scraping project?</span></a></p><div><hr></div><p>Just to draw some numbers, a new set on eBay Italy is listed about 30% above the last sale on StockX, and a used one about 30% below it. The lowest ask on StockX is often far from the last sale. For example, the Colosseum (set 10276) has a lowest ask of 2,258 euros and a last sale of 1,181.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Iubi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Iubi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png 424w, https://substackcdn.com/image/fetch/$s_!Iubi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png 848w, https://substackcdn.com/image/fetch/$s_!Iubi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png 1272w, https://substackcdn.com/image/fetch/$s_!Iubi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Iubi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png" width="1456" height="895" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:895,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:296553,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/225339262?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Iubi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png 424w, https://substackcdn.com/image/fetch/$s_!Iubi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png 848w, https://substackcdn.com/image/fetch/$s_!Iubi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png 1272w, https://substackcdn.com/image/fetch/$s_!Iubi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5919e300-69e2-4c90-8ddc-8957fcf8e5e0_3200x1966.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>I want to be clear about the limits of this comparison. On eBay we read the price the seller asks, without shipping, while the last sale of StockX is a price somebody paid. The title filter is a list of keywords and some partial items still pass. And for 77 sets eBay has more than 240 results, so we see only the first 240 in its default order.<br>Last but not least, this comparison is really naive, do not buy LEGO sets in production!</p><h2>The other endpoints: /map and /search</h2><p>After being happy for building my pricing comparison, I&#8217;ve just realized that I&#8217;ve used only the <code>/scrape</code> endpoint and not all the other ones, so let&#8217;s be creative.</p><p>What if we want to know what LEGO asks for the sets that are still in production? To find them, I used <code>/map</code> on the Italian LEGO store.</p><h3>/map to find the product pages</h3><p><code>/map</code> reads the sitemap of a website and the links of the page you give it, and returns the list of URLs. My first call, on <code>https://www.lego.com/it-it</code> with a filter on the product paths, returned zero links. The response had a <code>warning</code> field that explains why: &#8220;sitemap index lists 341 child sitemaps; 19 fetched (at most 25 sitemap files per map)&#8221;. This limit of 25 files isn&#8217;t in the docs, and on a big website it means that the sitemap route doesn&#8217;t reach the products.</p><p>The other route worked well. With <code>sitemap: "skip"</code>, <code>/map</code> reads only the links of the page. One call on the home page returned the 47 theme pages (Star Wars, Technic, Icons and so on). Then I made one call for every theme page and kept only the product URLs (<code>JS</code> is false here, I didn&#8217;t need the rendering).</p><pre><code><code>home = map_site("https://www.lego.com/it-it", sitemap="skip", geo="Italy")
themes = sorted({l["url"].split("?")[0] for l in home["results"][0]["content"]["links"]
                 if re.search(r"/it-it/themes/[^/]+$", l["url"].split("?")[0])})


def one(url):
    t0 = time.time()
    params = {"sitemap": "skip", "select_paths": ["^/it-it/product/"], "geo": "Italy"}
    if JS:
        params["JS_rendering"] = True
    try:
        r = map_site(url, attempts=2, **params)["results"][0]
        return {"theme": url, "status_code": r["status_code"], "s": round(time.time() - t0, 1),
                "links": [l["url"].split("?")[0] for l in r["content"]["links"]]}
    except WebDataError as exc:
        return {"theme": url, "error": str(exc)[:200], "s": round(time.time() - t0, 1), "links": []}</code></code></pre><p>The 47 calls took 26 seconds with 4 workers, with no errors and no JavaScript. They returned 551 product URLs, and every URL ends with the set number (<code>/it-it/product/millennium-falcon-75192</code>). Of the 397 StockX sets, 48 are in this list, so they&#8217;re still on sale at the official store. The real number can be higher, because I read only the first page of every theme.</p><h3>Two formats in one call</h3><p>For the 48 product pages I went back to <code>/scrape</code>, with a feature I hadn&#8217;t tried yet. If you add <code>?type=raw,markdown</code> to the endpoint, the response contains two results, the HTML and the Markdown of the same page.</p><pre><code><code>def jsonld_price(html: str):
    for block in re.findall(r'&lt;script[^&gt;]+application/ld\+json[^&gt;]*&gt;(.*?)&lt;/script&gt;', html, re.S):
        try:
            data = json.loads(block)
        except ValueError:
            continue
        for item in data if isinstance(data, list) else [data]:
            offers = item.get("offers") if isinstance(item, dict) else None
            offer = offers[0] if isinstance(offers, list) and offers else offers
            if isinstance(offer, dict) and offer.get("price") is not None:
                return float(offer["price"]), offer.get("priceCurrency"), (offer.get("availability") or "").rsplit("/", 1)[-1]
    return None, None, None


def one(item):
    num, url = item
    t0 = time.time()
    try:
        body = call("POST", "/scrape?type=raw,markdown",
                    {"url": url, "proxy_pool": "premium", "geo": "Italy", "markdown": True}, attempts=3)
    except WebDataError as exc:
        return {"set": num, "url": url, "error": str(exc)[:200]}
    by_type = {r.get("type"): r for r in body["results"]}
    html = (by_type.get("raw") or {}).get("content") or ""
    md = (by_type.get("markdown") or {}).get("content") or ""
    price, currency, availability = jsonld_price(html)
    m = re.search(r"(\d{1,3}(?:\.\d{3})*,\d{2})\s*&#8364;", md)
    p = stockx[num]
    return {"set": num, "url": url, "s": round(time.time() - t0, 1), "types": sorted(by_type), "html_chars": len(html),
            "md_chars": len(md), "retail": price, "currency": currency, "availability": availability,
            "first_price_in_markdown": m.group(1) if m else None, "title": p["title"],
            "stockx_highest_bid": p["highest_bid"], "stockx_lowest_ask": p["lowest_ask"], "stockx_last_sale": p["last_sale"] or None}</code></code></pre><p>In the code, <code>matches</code> is the list of the 48 sets with their URLs and <code>stockx</code> is the catalog we built before. I read the price from the JSON-LD in the HTML, and I used the Markdown to check what an LLM would see. The median page is 1.17 million characters of HTML and 53,000 of Markdown, 22 times smaller. On all the 48 pages the first price in the Markdown was the same as the JSON-LD one.</p><p>The retail price tells something I didn&#8217;t expect. Of these 48 sets, 46 have a last sale on StockX, and only 17 were sold above the retail price. The Millennium Falcon 75192 costs 849.99 euros at the LEGO store, and its last sale on StockX was 804 euros. The Death Star 75419 costs 999.99 and its last sale was 858. So, in this sample, most of the sets still in production sell on StockX below the price of the official store.</p><h3>/search to find who else sells</h3><p><code>/search</code> is the simplest of the three. You send a query and you get the Google results as a flat JSON, with <code>organic</code>, <code>people_also_ask</code>, <code>related_searches</code> and a few other keys. I used it to check if eBay and StockX were the right sources. For the 20 sets with the highest bid I searched &#8220;lego 10123 prezzo&#8221; (and so on) with <code>geo: "Italy"</code>, and counted the domains in the organic results.</p><pre><code><code>def one(r):
    t0 = time.time()
    try:
        body = search(f"lego {r['set_number']} prezzo", geo="Italy", locale="it-it")
    except WebDataError as exc:
        return {"set": r["set_number"], "error": str(exc)[:200]}
    return {"set": r["set_number"], "s": round(time.time() - t0, 1), "body": body}


domains = Counter()
for r in ok:
    organic = r["body"].get("organic") or []
    for d in {urlparse(o["link"]).netloc.removeprefix("www.") for o in organic}:
        domains[d] += 1</code></code></pre><p>All 20 calls answered, with a median of 1.3 seconds and 8 to 10 organic results each.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!kX-m!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!kX-m!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png 424w, https://substackcdn.com/image/fetch/$s_!kX-m!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png 848w, https://substackcdn.com/image/fetch/$s_!kX-m!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png 1272w, https://substackcdn.com/image/fetch/$s_!kX-m!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!kX-m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png" width="1456" height="925" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:925,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:184901,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/225339262?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!kX-m!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png 424w, https://substackcdn.com/image/fetch/$s_!kX-m!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png 848w, https://substackcdn.com/image/fetch/$s_!kX-m!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png 1272w, https://substackcdn.com/image/fetch/$s_!kX-m!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3f5e65b7-5f01-4c77-a7ca-9593d5340694_3200x2034.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><p>eBay and StockX are the first two, so the choice was fine. If I wanted to extend the comparison, the next sources would be Amazon, Subito (the Italian classifieds website), and Trovaprezzi, an Italian pricing comparison website.</p><h2>Final considerations</h2><p>The <code>xhr</code> parameter is the reason I&#8217;d use this API. Reading the internal API of StockX took one field in a JSON payload, and the same call gave me the requests too, with headers and payloads. The cost is time (around 25 seconds for a page), and you get every call without a filter. In the dashboard, the 14 StockX calls cost 2 cents in total, at the JavaScript rate.</p><p>A cost-wise way of using it could be to let your agent write a scraper based on the call that the API made, the payload, and results, and then check if you could bypass the bot protection on the APIs (if there are, since it often happens that a website is protected but its endpoints are not).</p><p>On eBay the product works with JavaScript rendering, and you need to know it before you start. Without it, in my run, almost half of the calls returned a block page marked as a success. My advice is the same I&#8217;d give for any scraping API. Check <code>results</code> before you read it, and validate the content of every page with a rule of your own (a selector, a title, a minimum size).</p><p><code>/map</code> is fast when it reads the links of a page. On the sitemap side it stops at 25 files, which is a problem on big websites. <code>/search</code> did what it says. The multi-format response is a small feature, but it saved me a second call on every page.</p><p>All the code you need is in this article. It runs on Python 3.11 with <code>requests</code>, <code>python-dotenv</code>, <code>beautifulsoup4</code> and <code>lxml</code>, which were already in our environment, and it reads a <code>DECODO_WEB_DATA_API_KEY</code> from a <code>.env</code> file next to <code>client.py</code>.</p><div><hr></div><p>Do you think the <a href="https://decodo.com/deals/free-plan-web-data-api?utm_medium=affiliate&amp;utm_source=The+Web+scraping+club+srl&amp;promoter_id=15855491&amp;fpr=3966225">Decodo Web Data API</a> would be useful in your scraping routines or your colleague&#8217;s? If so, share this article <a href="https://decodo.com/deals/free-plan-web-data-api?utm_medium=affiliate&amp;utm_source=The+Web+scraping+club+srl&amp;promoter_id=15855491&amp;fpr=3966225">and the API page</a>.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/p/reading-stockxs-internal-api-decodo-web-data-api?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.scraping.club/p/reading-stockxs-internal-api-decodo-web-data-api?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><div><hr></div><p></p>]]></content:encoded></item><item><title><![CDATA[Playwright MCP vs chrome-devtools-mcp: Clearing Up the Confusion]]></title><description><![CDATA[Clarifying the main differences between the two tools and understanding which is better for web scraping and browser automation]]></description><link>https://www.scraping.club/p/playwright-mcp-vs-chrome-devtools-mcp</link><guid isPermaLink="false">https://www.scraping.club/p/playwright-mcp-vs-chrome-devtools-mcp</guid><dc:creator><![CDATA[Antonello Zanini]]></dc:creator><pubDate>Sun, 15 Nov 2026 19:15:57 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/16f3c4d3-d75d-4e9b-ad2c-3fb9db5333ab_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>If you follow online communities, you&#8217;ve probably seen someone dismiss <code>chrome-devtools-mcp</code> as a copy of Playwright MCP. I understand the reaction. Both are MCP servers, both let AI agents interact with a browser, and both appear in many of the same setup guides.</p><p>But they have some fundamental differences, and those differences become more apparent when you use them for real-world tasks&#8230;</p><p>In this article, I&#8217;ll explore what each tool offers, where their capabilities overlap, where they differ, and how to decide which one to use, or whether using both makes sense.</p><h2>A Quick Overview of Playwright MCP and chrome-devtools-mcp</h2><p><a href="https://github.com/microsoft/playwright-mcp">Playwright MCP</a> is Microsoft&#8217;s MCP server built on top of Playwright. Instead, <code>chrome-devtools-mcp</code> (also known as Chrome DevTools for agents) is Google&#8217;s MCP server built on the CDP (<a href="https://chromedevtools.github.io/devtools-protocol/">Chrome DevTools Protocol</a>), relying on Puppeteer to automate browser interactions.</p><p>Although both equip AI agents with tools for browser automation, they serve different purposes. Playwright MCP focuses on exposing Playwright&#8217;s browser automation capabilities, making it well-suited for controlling browsers and executing automated workflows. In contrast, <code>chrome-devtools-mcp</code> combines browser automation with deeper insights into Chrome&#8217;s underlying behavior, including performance monitoring, network activity, and SEO analysis.</p><p>In the rest of this post, I&#8217;ll explore both tools in detail, compare their capabilities, and explain where each one fits best.</p><div class="callout-block" data-callout="true"><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/B5JPeJ">Decodo&#8217;s Web Data API</a></strong> turns any site into clean, structured data your models can actually use.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/B5JPeJ" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png" width="716" height="286.4" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:280,&quot;width&quot;:700,&quot;resizeWidth&quot;:716,&quot;bytes&quot;:105634,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;text&quot;:&quot;Activate your free plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/B5JPeJ"><span>Activate your free plan</span></a></p></div><div><hr></div><h2>An Introduction to Playwright MCP</h2><p>Now that you have a high-level understanding of Playwright MCP vs <code>chrome-devtools-mcp</code>, let me introduce you to Playwright MCP.</p><h3>What Playwright MCP Is</h3><p>Playwright MCP is an MCP server from Microsoft that exposes Playwright&#8217;s API as tools that AI agents can call. It was released in March 2025 and quickly became one of the most popular MCP servers, with over 38k GitHub stars at the time of writing.</p><p>By default, it runs as a <a href="https://modelcontextprotocol.io/specification/draft/basic/transports/stdio">local process over stdio</a>, but you can also start it as an HTTP server with the <code>--port</code> flag. That comes in handy when the browser has to run on a machine with no GUI, or when several clients share one server.</p><p><strong>Further reading</strong>:</p><ul><li><p><em><a href="https://playwright.dev/docs/getting-started-mcp">Official docs</a></em></p></li><li><p><em><a href="https://github.com/microsoft/playwright-mcp">GitHub repository</a></em></p></li><li><p><em><a href="https://scrape.do/blog/playwright-mcp/">Playwright MCP Deep Dive: The Ultimate Guide with Examples</a></em></p></li></ul><h3>Key Features</h3><p>Playwright MCP&#8217;s main capabilities include:</p><ul><li><p><strong>LLM-friendly page interaction</strong>: Provides structured page data based on <a href="https://playwright.dev/docs/getting-started-mcp#accessibility-snapshots">Playwright&#8217;s accessibility tree</a>, allowing AI agents to interact with web pages without requiring a vision model.</p></li><li><p><strong>Deterministic element targeting</strong>: References in page snapshots help agents identify and interact with specific elements across calls, reducing the ambiguity associated with screenshot-based targeting.</p></li><li><p><strong>Cross-browser support</strong>: Supports browser sessions in Chromium, Firefox, WebKit, and Microsoft Edge.</p></li><li><p><strong>Session management</strong>: Can start a fresh browser session or, when configured, connect to an existing session.</p></li><li><p><strong>Connection flexibility</strong>: Can launch a local browser or connect to a remote browser through a CDP endpoint.</p></li><li><p><strong>Opt-in tool groups</strong>: Its tools are organized into groups, such as network, storage, DevTools, vision, PDF, and testing. You can enable only the groups you need to reduce <a href="https://dev.to/thedailyagent/mcp-tool-overload-why-more-tools-make-your-agent-worse-5a49">tool overhead</a>, a common concern when working with MCP servers.</p></li><li><p><strong>Recording and video capture</strong>: Supports tracing, video recording with chapter markers, and recording user interactions, making it easier to review and debug automated sessions.</p></li></ul><h3>Main Tools</h3><p>The table lists the most useful tools exposed by the Playwright MCP:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HDe4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HDe4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png 424w, https://substackcdn.com/image/fetch/$s_!HDe4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png 848w, https://substackcdn.com/image/fetch/$s_!HDe4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png 1272w, https://substackcdn.com/image/fetch/$s_!HDe4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HDe4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png" width="1456" height="1088" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1088,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The main Playwright MCP toolsThe main Playwright MCP tools&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The main Playwright MCP toolsThe main Playwright MCP tools" title="The main Playwright MCP toolsThe main Playwright MCP tools" srcset="https://substackcdn.com/image/fetch/$s_!HDe4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png 424w, https://substackcdn.com/image/fetch/$s_!HDe4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png 848w, https://substackcdn.com/image/fetch/$s_!HDe4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png 1272w, https://substackcdn.com/image/fetch/$s_!HDe4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1c00c529-9a3d-47f8-9440-962c541f42b4_1490x1113.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The main Playwright MCP tools</figcaption></figure></div><h3>How Playwright MCP Works</h3><p>Playwright MCP sits between your AI agent and the Playwright engine. In an AI agent scenario, the process works as follows:</p><ol><li><p>The AI agent interprets the user&#8217;s instructions and determines which MCP tools need to be called.</p></li><li><p>The MCP tools call the underlying Playwright API.</p></li><li><p>Playwright <a href="https://www.scraping.club/p/webdriver-vs-cdp-vs-bidi">performs the corresponding actions in the browser via CDP</a>.</p></li></ol><p>First, the agent navigates to a page using <code>browser_navigate</code>. When it needs to interact with the page, it typically calls <code>browser_snapshot</code> to retrieve a structured representation of the page&#8217;s accessibility tree. This references elements on the page using a format like this:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!MBkV!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!MBkV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png 424w, https://substackcdn.com/image/fetch/$s_!MBkV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png 848w, https://substackcdn.com/image/fetch/$s_!MBkV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png 1272w, https://substackcdn.com/image/fetch/$s_!MBkV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!MBkV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png" width="1456" height="766" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:766,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Playwright MCP&#8217;s accessibility tree: sample outputPlaywright MCP&#8217;s accessibility tree: sample output&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Playwright MCP&#8217;s accessibility tree: sample outputPlaywright MCP&#8217;s accessibility tree: sample output" title="Playwright MCP&#8217;s accessibility tree: sample outputPlaywright MCP&#8217;s accessibility tree: sample output" srcset="https://substackcdn.com/image/fetch/$s_!MBkV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png 424w, https://substackcdn.com/image/fetch/$s_!MBkV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png 848w, https://substackcdn.com/image/fetch/$s_!MBkV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png 1272w, https://substackcdn.com/image/fetch/$s_!MBkV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecba4250-6d80-44d6-a3fd-e844693a9592_2598x1366.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Playwright MCP&#8217;s accessibility tree: sample output</figcaption></figure></div><p>The agent then selects a reference and passes it to an interaction tool, such as <code>browser_click</code>. The server resolves the reference to a <a href="https://playwright.dev/docs/locators">Playwright locator</a>, performs the requested action, and returns the updated page state to the agent.</p><p>Using the same workflow, an AI agent can interact with other page elements, such as buttons, links, text fields, and dropdowns, to complete multi-step browser workflows.</p><div><hr></div><div class="callout-block" data-callout="true"><p><em>Tired of getting blocked while scraping the web?<br><strong><a href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club">ScrapingBee</a></strong> handles proxies, browsers, anti-bot systems, and retries so you can focus on your data. Get clean Markdown, JSON, or HTML from the web with up to a 99.9% success rate. ScrapingBee is SOC 2 Type II and GDPR compliant, and trusted by 4,000+ developers.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 424w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 848w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png" width="1456" height="437" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:437,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:128210,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/216891176?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 424w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 848w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club&quot;,&quot;text&quot;:&quot;Discover ScrapingBee&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club"><span>Discover ScrapingBee</span></a></p></div><div><hr></div><h2>A Beginner&#8217;s Guide to chrome-devtools-mcp</h2><p>After exploring Playwright MCP, it&#8217;s time to dig into <code>chrome-devtools-mcp</code>.</p><h3>What Is chrome-devtools-mcp?</h3><p><code>chrome-devtools-mcp</code> is the MCP server for Google Chrome DevTools, maintained by the Chrome DevTools team on GitHub. It entered public preview in September 2025 and has grown rapidly since then, gaining over 54k GitHub stars.</p><p>Like Playwright MCP, it communicates over stdio. It can also connect to an existing Chrome instance through a browser URL or WebSocket endpoint, rather than always launching a new browser instance.</p><p><strong>Further reading</strong>:</p><ul><li><p><a href="https://developer.chrome.com/docs/devtools/agents">Official page</a></p></li><li><p><a href="https://github.com/ChromeDevTools/chrome-devtools-mcp">GitHub repository</a></p></li></ul><h3>Key Features</h3><p>Chrome DevTools for agents provides many features, such as:</p><ul><li><p><strong>Performance insights</strong>: Records performance traces and analyzes them using Chrome DevTools&#8217; analysis engine, returning actionable insights such as <a href="https://developer.chrome.com/docs/performance/insights/lcp-breakdown">Largest Contentful Paint (LCP) breakdowns</a>.</p></li><li><p><strong>Advanced debugging</strong>: Inspects network requests and retrieves JavaScript console messages, including source-mapped stack traces to help identify errors.</p></li><li><p><strong>Reliable browser automation</strong>: Uses Puppeteer to execute browser actions such as clicking elements, typing text, and navigating between pages.</p></li><li><p><strong>Lighthouse audits</strong>: Runs accessibility, SEO, and best-practices audits through a single tool call, without requiring a separate process.</p></li><li><p><strong>Memory snapshots</strong>: Captures V8 heap snapshots to disk, helping developers investigate memory usage and identify potential memory leaks.</p></li><li><p><strong>Live Chrome attachment</strong>: Connects to an existing Chrome instance, allowing agents to work with an already-open browser session.</p></li><li><p><strong>Emulation controls</strong>: Configures CPU and network throttling, viewport dimensions, user agent, geolocation, and color scheme to simulate different devices and environments.</p></li><li><p><strong>Slim mode</strong>: Offers a reduced set of three tools for navigation, script execution, and screenshots, helping minimize context usage.</p></li></ul><h3>Main Tools</h3><p>Discover the most useful tools provided by the MCP server:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ReMk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ReMk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png 424w, https://substackcdn.com/image/fetch/$s_!ReMk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png 848w, https://substackcdn.com/image/fetch/$s_!ReMk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png 1272w, https://substackcdn.com/image/fetch/$s_!ReMk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ReMk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png" width="1456" height="1172" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1172,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The main chrome-devtools-mcp toolsThe main chrome-devtools-mcp tools&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The main chrome-devtools-mcp toolsThe main chrome-devtools-mcp tools" title="The main chrome-devtools-mcp toolsThe main chrome-devtools-mcp tools" srcset="https://substackcdn.com/image/fetch/$s_!ReMk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png 424w, https://substackcdn.com/image/fetch/$s_!ReMk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png 848w, https://substackcdn.com/image/fetch/$s_!ReMk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png 1272w, https://substackcdn.com/image/fetch/$s_!ReMk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ac07c55-8034-41c0-8112-be012fb8bc61_1490x1199.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The main chrome-devtools-mcp tools</figcaption></figure></div><h3>How Does Chrome DevTools for Agents Work</h3><p>For browser automation tasks such as clicking, typing, and navigating, Chrome DevTools for agents relies on Puppeteer through the CDP, following an approach similar to Playwright MCP (which is where the confusion between the two tools stems from).</p><p>For all other features, it reuses the analysis engines that power <a href="https://developer.chrome.com/docs/devtools">Chrome DevTools</a>:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!BUSU!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BUSU!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png 424w, https://substackcdn.com/image/fetch/$s_!BUSU!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png 848w, https://substackcdn.com/image/fetch/$s_!BUSU!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png 1272w, https://substackcdn.com/image/fetch/$s_!BUSU!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!BUSU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png" width="1456" height="924" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:924,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The &#8220;Lighthouse&#8221; view from the Chrome's DevTools panelThe &#8220;Lighthouse&#8221; view from Chrome's DevTools panel&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The &#8220;Lighthouse&#8221; view from the Chrome's DevTools panelThe &#8220;Lighthouse&#8221; view from Chrome's DevTools panel" title="The &#8220;Lighthouse&#8221; view from the Chrome's DevTools panelThe &#8220;Lighthouse&#8221; view from Chrome's DevTools panel" srcset="https://substackcdn.com/image/fetch/$s_!BUSU!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png 424w, https://substackcdn.com/image/fetch/$s_!BUSU!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png 848w, https://substackcdn.com/image/fetch/$s_!BUSU!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png 1272w, https://substackcdn.com/image/fetch/$s_!BUSU!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0bb39cc6-9fec-416c-a897-04742b57c9a6_1576x1000.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The &#8220;Lighthouse&#8221; view from Chrome&#8217;s DevTools panel</figcaption></figure></div><p>This reliance on Chrome-specific technologies also explains why <code>chrome-devtools-mcp</code> supports only Google Chrome and <a href="https://developer.chrome.com/docs/automation-and-testing/chrome-for-testing">Chrome for Testing</a>.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://bundle.scraping.club/&quot;,&quot;text&quot;:&quot;Scraping Bundle - 20k+$ in tools&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://bundle.scraping.club/"><span>Scraping Bundle - 20k+$ in tools</span></a></p><div><hr></div><h3>When to Use Playwright MCP</h3><p>Playwright MCP is especially useful when you want an AI agent to interact with dynamic and/or highly interactive websites. Other use cases are also supported.</p><h3>Testing Flows Across Browsers</h3><p>If your automation needs to work across Chromium-based browsers, Firefox, and Safari, Playwright MCP has you covered. Use the <code>--browser</code> flag to select the browser engine, while keeping the same set of tools across browser configurations.</p><h3>Automating Multi-Step Workflows</h3><p>Most browser automation scenarios involve multiple steps. Playwright MCP provides tools that let an AI agent inspect the current page, understand what&#8217;s happening, and decide what to do next. This makes it well suited to workflows such as navigating a site, filling out forms, handling <a href="https://www.scraping.club/p/handling-infinite-scrolling-python-js">infinite scrolling scenarios</a>, and completing multi-step wizards.</p><h3>Inspecting Network Requests</h3><p>Dynamic websites often load data through AJAX or other network requests. Inspecting these requests can be a <a href="https://www.scraping.club/p/apis-in-web-scraping">convenient way to retrieve structured data directly</a>, without relying solely on information rendered on the page. Playwright MCP supports this through tools such as <code>browser_network_requests</code> and <code>browser_network_request</code>.</p><h3>Preserving Cookies Between Runs</h3><p>Some automation workflows are easier (or even only possible!) when specific cookies or local storage values are already present, such as consent preferences or session data. With <code>--caps=storage</code>, an agent can use <code>browser_storage_state</code> to save cookies and local storage to a file, then restore that state in a later run with <code>browser_set_storage_state</code>.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><h3>When to Use Chrome DevTools MCP</h3><p><code>chrome-devtools-mcp</code> is a strong choice when you need the kind of debugging and performance insights you&#8217;d normally get by opening Chrome DevTools yourself.</p><h3>Automating Browser Workflows</h3><p><code>chrome-devtools-mcp</code> provides browser interaction tools powered by Puppeteer, allowing an AI agent to navigate pages, click elements, fill out forms, and interact with web applications. However, it supports only Chromium-based browsers and Firefox.</p><h3>Investigating Slow Pages</h3><p>When a page feels slow, guessing rarely helps. Start a performance trace with <code>performance_start_trace</code>, reproduce the issue, and stop the trace with <code>performance_stop_trace</code>. The resulting trace includes named performance insights that help identify potential bottlenecks.</p><p>For example, if LCP (<a href="https://web.dev/articles/lcp">Largest Contentful Paint</a>) is poor, the agent can call <code>performance_analyze_insight</code> to examine the LCP breakdown and identify which phase is taking the most time.</p><p>Before measuring, you can also employ the <code>emulation</code> tools to throttle CPU and network speed, simulating a mid-range mobile device rather than your fast development machine. These are capabilities that Playwright MCP doesn&#8217;t offer.</p><h3>Diagnosing Console Errors</h3><p>The <code>list_console_messages</code> tool retrieves browser console output and can include stack traces that map back to the page&#8217;s source code when <a href="https://developer.mozilla.org/en-US/docs/Glossary/Source_map">source maps</a> are available. This helps agents investigate JavaScript errors, failed operations, and other issues that may prevent a page from working as expected.</p><p>Combined with network inspection tools, console messages can also help narrow down whether a problem stems from client-side code or failed network requests.</p><h3>Running Accessibility and SEO Audits</h3><p><a href="https://developer.chrome.com/docs/lighthouse/overview">Lighthouse reports</a> can be useful, but generating them manually often means switching between tools. The <code>lighthouse_audit</code> tool lets an AI agent run audits directly as part of its workflow, covering areas such as accessibility, SEO, and best practices.</p><h3>Investigating Memory Leaks</h3><p>Memory leaks can be difficult to detect until a page becomes sluggish or unresponsive. The <code>take_memory_snapshot</code> tool captures a V8 heap snapshot and saves it to a specified path for further analysis.</p><p>This low-level memory investigation capability goes beyond Playwright MCP&#8217;s core browser automation tools.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://substack.thewebscraping.club/p/consulting&quot;,&quot;text&quot;:&quot;Need help with your scraping project?&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://substack.thewebscraping.club/p/consulting"><span>Need help with your scraping project?</span></a></p><div><hr></div><h3>Can Playwright MCP and Chrome DevTools MCP Be Used Together?</h3><p>Yes, but how you combine them matters more than the fact that you can!</p><p>The obvious setup, which involves installing both servers and letting each launch its own browser, can be the wrong choice. Sure, it works for independent, stateless tasks, but each server operates in its own browser session. Cookies, local storage, open tabs, and in-progress workflows don&#8217;t automatically carry over.</p><p>For example, if Playwright MCP opens an e-commerce page and clicks &#8220;Load more&#8221; five times, a separately launched Chrome DevTools MCP browser on the same URL will still see the initially loaded products. It won&#8217;t inherit the state of the page controlled by Playwright MCP.</p><p>A more useful setup is to have both MCP servers connect to the same Chrome (or Firefox, but not Safari, as it isn&#8217;t supported by both) instance. After all, both tools speak CDP, a browser can accept multiple connections over the same remote debugging port, and both tools can connect to a CDP endpoint. So, that&#8217;s possible!</p><p><strong>Note</strong>: Multiple connections to the same CDP endpoint don&#8217;t guarantee that both servers can control the browser concurrently without conflicts, so validate the configuration before relying on it.</p><p>With this setup, both servers can act on the same tabs, cookies, and session, and the agent can pick the right tool for each step. Playwright MCP can drive the workflow. Once the page is in the state you care about, Chrome DevTools MCP can profile it, run an audit, take a heap snapshot, or read source-mapped console errors. Nothing needs to be re-created, because both servers are working with the same live page.</p><p>For a detailed comparison, refer to the full coverage table below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!NwjP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!NwjP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg 424w, https://substackcdn.com/image/fetch/$s_!NwjP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg 848w, https://substackcdn.com/image/fetch/$s_!NwjP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!NwjP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!NwjP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg" width="1297" height="2160" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:2160,&quot;width&quot;:1297,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Playwright MCP vs chrome-devtools-mcp&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Playwright MCP vs chrome-devtools-mcp" title="Playwright MCP vs chrome-devtools-mcp" srcset="https://substackcdn.com/image/fetch/$s_!NwjP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg 424w, https://substackcdn.com/image/fetch/$s_!NwjP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg 848w, https://substackcdn.com/image/fetch/$s_!NwjP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!NwjP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7f05d637-5fd0-428e-9823-737c8fac01a0_1297x2160.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Playwright MCP vs chrome-devtools-mcp</figcaption></figure></div><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>Final Thoughts</h2><p>In this article, I&#8217;ve explained why Playwright MCP and <code>chrome-devtools-mcp</code> may look similar at first glance but have distinct strengths. You&#8217;ve now seen what each tool offers and where each one excels.</p><p>To be honest, most of Chrome DevTools MCP&#8217;s additional capabilities are more useful for testing, debugging, performance analysis, and SEO audits rather than for web scraping or general browser automation. For those tasks, Playwright MCP may be the better choice (just as <a href="https://www.scraping.club/p/web-scraping-from-0-to-hero-our-first">Playwright is one of the best options for web scraping!</a>)</p><p>That said, access to Chrome&#8217;s console messages and debugging tools can still be valuable. That&#8217;s one reason it can make sense to connect both MCP servers to the same browser session, allowing the agent to use each tool when it fits the task. With remote browsers, however, this setup can be more challenging unless the cloud browser provider supports persistent sessions and allows both servers to connect to the same browser instance.</p><p>I hope this article has cleared up some of the confusion and helped you understand which tool is best suited to your needs. If you have any questions or suggestions, feel free to leave a comment below. Until next time!</p><div><hr></div><p><em>Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/p/playwright-mcp-vs-chrome-devtools-mcp?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.scraping.club/p/playwright-mcp-vs-chrome-devtools-mcp?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><div><hr></div><h2>FAQ</h2><h3>Does Chrome DevTools for agents come with a CLI?</h3><p>Yes. Alongside the MCP server, <code>chrome-devtools-mcp</code> ships a command-line interface for users (or agents) that prefer shell commands. It&#8217;s documented in the project&#8217;s docs, and the project also bundles a skill to help agents use it.</p><h3>Does Playwright MCP come with a CLI?</h3><p>No, Playwright MCP is only an MCP server. Microsoft publishes a separate package, <code>@playwright/cli</code>, which gives coding agents concise shell commands and skills. It&#8217;s a different product from Playwright MCP and from the Playwright test runner, so check the package name before you install.</p><h3>Do Playwright MCP and Chrome DevTools MCP support connection to remote browsers?</h3><p>Yes, both can connect to a remote browser over CDP. Playwright MCP uses <code>--cdp-endpoint</code>, and <code>chrome-devtools-mcp</code> uses <code>--wsEndpoint</code> or <code>--browserUrl</code>. Remote stealth browsers typically handle proxies and anti-bot measures on their side, helping your AI-powered automation avoid being blocked. </p>]]></content:encoded></item><item><title><![CDATA[How teams actually deal with blocks, how many failures go unnoticed, and what it all costs
]]></title><description><![CDATA[Numbers and ideas from the Dataimpulse survey]]></description><link>https://www.scraping.club/p/how-teams-deal-with-blocks-scraping</link><guid isPermaLink="false">https://www.scraping.club/p/how-teams-deal-with-blocks-scraping</guid><dc:creator><![CDATA[Olia Liudko]]></dc:creator><pubDate>Wed, 11 Nov 2026 08:40:35 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!SrvP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>This is a guest post written by the <a href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov">DataImpulse</a> team; all the numbers and ideas are drawn from a survey of its users.</em></p><div><hr></div><p>46.7% of users surveyed for Apify&#8217;s State of Web Scraping Report noted that they face more difficulties while scraping protected websites. One of the main reasons is AI - its development has caused scrapers to become more sophisticated, and naturally, anti-bot systems evolve too. This makes scraping teams constantly look for solutions to deal with blocks.</p><p><a href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov">DataImpulse</a>, a proxy provider, conducted its own survey among its users to find out what errors occur the most often now and what actually helps get data.</p><p>In the survey, which took place in October 2026, representatives of different occupations participated. Solo specialists are the largest group (42%), data-based SaaS company employees (19%), and only 4% belong to in-house teams of big companies. The most often named use cases are price and goods scraping for e-commerce (31%), social media scraping (25%), and data gathering for LLMs and AI training (16%).</p><h2>What blocks look like in practice</h2><p>Blocks do not always look like instantly denied access. Moreover, the most dangerous are those that teams do not notice - not those that stop a scraper immediately.</p><p>Error 403 Forbidden is the most obvious version and the most honest answer. A server directly notifies that a request was denied. Usually, 403 happens when a WAF (Cloudflare, Akamai, Imperva) filters a request even before it reaches an app. The reasons often lie lower than the HTTP level: TLS fingerprint (JA3/JA4) that does not match the claimed User-Agent header, the order of HTTP/2-frames, which is typical for libraries like requests or httpx, or the poor reputation of an IP subnet. 403 itself is easy to spot while monitoring the process, so it rarely goes unnoticed. Understanding why it happened is another story, though.</p><p>Error 429 Too many requests is often misinterpreted. Teams react to it by increasing the number of IPs in use and aggressively rotating them, though modern anti-bot detectors consider not only IP rate limits, but also sessions, cookies, device fingerprinting, and even behavioral patterns. A situation where IP changes and browser fingerprint remains the same only makes the error happen faster, as a system sees the same client with different addresses and switches from rate limiting to full block. The majority of scrapers ignore a Retry-After header.</p><p>CAPTCHA (reCAPTCHA, hCaptcha, Cloudflare Turnstile) happens when an anti-bot sees a user as suspicious, but not enough to block immediately. The problem is that such pages often return a 200 status code. If a parser checks only the status code, it successfully scrapes a blank page.</p><p>JS challenge is an intermediate step between an access permit and a CAPTCHA. Cloudflare Managed Challenge, Akamai&#8217;s sensor scripts (sensor_data), DataDome, or HUMAN (PerimeterX) execute code in a browser, which collects hundreds of signals: navigator properties, Canvas and WebGL rendering, event timing, automation signals like navigator.webdriver, or CDP artifacts. Without a real JS engine, this step is impossible, and a headless browser without disguise passes it less often now.</p><p>A 200 Soft block is the main reason for failed requests that go unnoticed. A website returns a valid HTML page, but with wrong data: an empty list of goods, prices without discounts, an abridged catalog, redirects to the main page, or data that is purposely falsified, so-called data poisoning. HTTP status monitoring shows a 100% success rate, but the dataset obtained is useless. Detection of such blocks is only possible by validating content: checking schemas, fill rate, and response size anomalies. Comparing the run to previous ones also helps.</p><p>Results of DataImpulse&#8217;s survey support the theory: 67% of respondents named error 403, CAPTCHA, JS challenge, and 200 soft block as the most common problems. Rate limiting comes second - 31% of pros dealt with error 429 or slow responses without an error code.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SrvP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png 424w, https://substackcdn.com/image/fetch/$s_!SrvP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png 848w, https://substackcdn.com/image/fetch/$s_!SrvP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png 1272w, https://substackcdn.com/image/fetch/$s_!SrvP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SrvP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png" width="1456" height="796" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:796,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!SrvP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png 424w, https://substackcdn.com/image/fetch/$s_!SrvP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png 848w, https://substackcdn.com/image/fetch/$s_!SrvP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png 1272w, https://substackcdn.com/image/fetch/$s_!SrvP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F91d0fe00-e938-4da7-870d-58c369687725_2048x1120.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>There are also other errors that occur less often. Timeouts or lost connections happen to  20% of respondents, redirects to a login page or consent page happen to 15%, and 502/503 (server errors) happen to 12%. 15% of specialists faced a 407 Proxy Authentication Required error, though this error is about proxy setup and usually happens due to typos, wrong credentials, and other small inconsistencies. Only 11% of respondents say that they rarely experience blocks.</p><h3>How long does a workaround remain effective?</h3><p>Nothing works forever. Anti-bot solution providers issue updates non-stop. Akamai regularly changes versions of a sensor script and obfuscation, Cloudflare updates challenge logic without any announcement, and DataDome and HUMAN train their models using fresh traffic, including your scraper&#8217;s. The more widely used a workaround is, the faster its signature gets to detectors.</p><p>It is important to distinguish between two scenarios:</p><ul><li><p>A clean break - the success rate falls almost to 0 in a matter of hours. It is noticeable, so teams react right away. Among DataImpulse&#8217;s users, 65% noticed something was wrong on the same day.</p></li><li><p>Slow fall - the percentage of successful requests decreases gradually because an anti-bot system first flags a user, collects data, and only after that does it amplify the reaction. Retry rate grows, as well as proxy spending and the number of soft blocks, but no alerts go off as there is no sudden drop below the threshold.</p></li></ul><p>The second scenario is the most expensive one. Between the moment when the workaround stops being useful and the moment a team notices, weeks may pass. This whole time, a scraping infrastructure uses resources and returns biased data.</p><p>The survey shows that 22% of respondents noticed mistakes only within a week. Almost one in ten respondents learned that the pipeline does not deliver the right data only after a week or longer. About 3% knew about the problem after more than a month.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!S0Zb!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb38f84e-7b71-40a4-b81b-756e88e35f95_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!S0Zb!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb38f84e-7b71-40a4-b81b-756e88e35f95_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!S0Zb!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb38f84e-7b71-40a4-b81b-756e88e35f95_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!S0Zb!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb38f84e-7b71-40a4-b81b-756e88e35f95_2048x1152.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!S0Zb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb38f84e-7b71-40a4-b81b-756e88e35f95_2048x1152.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/db38f84e-7b71-40a4-b81b-756e88e35f95_2048x1152.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!S0Zb!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb38f84e-7b71-40a4-b81b-756e88e35f95_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!S0Zb!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb38f84e-7b71-40a4-b81b-756e88e35f95_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!S0Zb!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb38f84e-7b71-40a4-b81b-756e88e35f95_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!S0Zb!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdb38f84e-7b71-40a4-b81b-756e88e35f95_2048x1152.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>So the metric that really proves a scraper&#8217;s health is not &#8220;the website returns a 200 status code&#8221;. It is &#8220;we see the same data real users see&#8221;.</p><h3>How teams deal with scraping blocks</h3><p>Often, after a block occurs, teams follow a classic scenario, trying to solve the problem quickly to avoid wasting resources and time. The irony is that the reasons behind the blocks are different, and standard steps taken without investigation only delay the fix.</p><p>So, often the first step would be to retry sending a request. This helps with random glitches like timeouts, 502/503 errors caused by an overloaded origin server, and temporary rate limiting. If a block happened because a bot was detected by its fingerprint, a retry is at the very least useless - your request will be denied again. Or worse (and more real): aggressive retries without backoffs and delays trigger protection mechanisms further and may lead to a ban.</p><p>Changing the IP is another method that solves the problem only if it was an address that was blocked. If an anti-bot identifies a client by its TLS fingerprint, cookies, or device fingerprint, IP rotation changes nothing; it only burns out the proxy pool, as more IPs become associated with a particular bot&#8217;s activities.</p><p>One more method is changing a proxy type. Datacenter addresses are cheap, but often blocked by the majority of WAFs. Only 15% of respondents use them. Residential IPs are more costly, but have a better reputation. They help fit the normal usage patterns and have a good ratio of price and benefits, so they are the most popular type - 79% of scraping pros use rotating residential proxies and 44% use sticky residential proxies (for some questions, respondents could choose more than one answer, so the general share of residential proxies is more than 100%). Mobile proxies are rarely blocked because the IP via a CGNAT operator is used by thousands of real users, so if such an address gets blocked, it affects a lot of ordinary people. The price is relatively high, though - that is why only 17% of people use them. 15% of users also said that they rely on cloud-based solutions or VPS addresses.</p><p>Switching between sticky and rotating sessions is another widespread solution. Whether it helps depends on how a website verifies a user. Rotating sessions are suitable for mass data collection from independent pages, and they are used by 59% of respondents. Sticky sessions are necessary when an anti-bot issues a token after passing a challenge (cf_clearance, _abck, datadome): a token is tied to an IP, and changing an IP mid-session makes the token invalid. That is why 40% of users rely on them. Mismatching IPs, cookies, and fingerprints within the same session is one of the main reasons for &#8220;unexplained&#8221; blocks. At the same time, only 8% of users claimed that switching to sticky sessions truly enhanced the success rate.</p><p>Fingerprinting is where the majority of blocks happen. It is about coherence of the whole stack: TLS (JA3/JA4), HTTP/2-parameters, headers and their order, JS browser environment (Canvas, timezone, language, etc.), and also whether all of it matches the IP&#8217;s location. Anti-bots look for inconsistencies: Chrome running on Windows with a TLS fingerprint specific to Python, Kyiv timezone with a Texas IP address. Tools like curl_cffi, patched browsers, and anti-detect solutions fix some of these inconsistencies, but they need constant monitoring and updates.</p><p>Unblocker APIs (Web Unlocker, Zyte API, ScraperAPI, etc.) are often the last resort. Teams pay for successful requests and hand over all the struggles to data providers. It is faster and somewhat predictable, but expensive at scale, creates dependency on a vendor, and still never explains why blocks happen. When an unlocker is unable to deal with a particular domain, there is no internal expertise or solution. Only 7% of DataImpulse users tried this method.</p><p>The summary is simple: before changing the proxy or session type, it is necessary to understand at what level detection happens: IP, fingerprinting, JS environment, or behavior patterns. The reason determines the solution.</p><h3>How many retries it takes for one successful run</h3><p><a href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov">DataImpulse</a> asked its respondents about websites that were the hardest to scrape. 31% of respondents said that the first request goes to waste in only 5% of cases; for another 31%, it happened in more than 15% of cases. 27% of users reported 5-15% unsuccessful first requests. At the same time, 10% of respondents do not have such statistics.</p><p>If we talk about how many requests it takes to get one usable data record, it is even more interesting. Only 11% of respondents were lucky on the first try. 32% of users said it takes them 2 requests on average; another 32% needed 3-5 requests and 11% used 6 or more. In other words, on a protected website it takes 2-3 attempts to get data.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!TAB9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef0e8576-e47c-4cf0-80fa-80ec6acd67ac_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!TAB9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef0e8576-e47c-4cf0-80fa-80ec6acd67ac_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!TAB9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef0e8576-e47c-4cf0-80fa-80ec6acd67ac_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!TAB9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef0e8576-e47c-4cf0-80fa-80ec6acd67ac_2048x1152.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!TAB9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef0e8576-e47c-4cf0-80fa-80ec6acd67ac_2048x1152.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ef0e8576-e47c-4cf0-80fa-80ec6acd67ac_2048x1152.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!TAB9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef0e8576-e47c-4cf0-80fa-80ec6acd67ac_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!TAB9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef0e8576-e47c-4cf0-80fa-80ec6acd67ac_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!TAB9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef0e8576-e47c-4cf0-80fa-80ec6acd67ac_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!TAB9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fef0e8576-e47c-4cf0-80fa-80ec6acd67ac_2048x1152.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Note that the data looks a bit contradictory. Not so many users reported obvious denials, but for most of them it still takes several attempts to get a usable data record. One of the possible explanations is that some responses are technically successful, but impossible to use. It may be due to soft blocks, incomplete pages, or data from the wrong regions.</p><h3>What really works and what turns out to be useless</h3><p>For this question, users could choose several answers. For half the respondents, switching to residential proxies was a way out. Fingerprinting solutions work too: 26% of respondents praised headless browsers and 25% -  fingerprint and TLS emulation. 18% of pros shared that precise geolocation was a game-changer, and 17% said mobile proxies worked.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!b2Y4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F012771ca-a735-4de6-9a79-ff81c6ab1e0e_2048x1436.png 424w, https://substackcdn.com/image/fetch/$s_!b2Y4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F012771ca-a735-4de6-9a79-ff81c6ab1e0e_2048x1436.png 848w, https://substackcdn.com/image/fetch/$s_!b2Y4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F012771ca-a735-4de6-9a79-ff81c6ab1e0e_2048x1436.png 1272w, https://substackcdn.com/image/fetch/$s_!b2Y4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F012771ca-a735-4de6-9a79-ff81c6ab1e0e_2048x1436.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!b2Y4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F012771ca-a735-4de6-9a79-ff81c6ab1e0e_2048x1436.png" width="1456" height="1021" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/012771ca-a735-4de6-9a79-ff81c6ab1e0e_2048x1436.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1021,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!b2Y4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F012771ca-a735-4de6-9a79-ff81c6ab1e0e_2048x1436.png 424w, https://substackcdn.com/image/fetch/$s_!b2Y4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F012771ca-a735-4de6-9a79-ff81c6ab1e0e_2048x1436.png 848w, https://substackcdn.com/image/fetch/$s_!b2Y4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F012771ca-a735-4de6-9a79-ff81c6ab1e0e_2048x1436.png 1272w, https://substackcdn.com/image/fetch/$s_!b2Y4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F012771ca-a735-4de6-9a79-ff81c6ab1e0e_2048x1436.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Some popular fixes turned out to be not so effective. Reducing speed and the number of parallel threads worked for 8% of users; CAPTCHA solvers for 5%, switching to another provider for 5%, and an API unblocker for 5%.</p><h3>What happens after &#8220;success&#8221;</h3><p>For the majority of pipelines, a request is successful if it returns status code 200 and the parser does not crash. From here, data often gets into a database without any validation. Here, a number of glitches accumulate.</p><p>Experienced teams validate content on several levels. The first one - structural: whether key selectors are present, response size is typical, and challenge markers or CAPTCHAs are present. Second - data schemas: required fields, types, allowed ranges (like price is not 0 and not 10 times higher than the market median). Third one - statistical: fill rate across fields, the number of goods in a category compared to the previous run, and sudden changes. If a fill rate drops from 98% to 70%, it almost always means a soft block or layout change, not real changes on a website.</p><p>Even with all of this, some typical production problems still go unnoticed during the basic validation because the data looks correct.</p><ul><li><p>Mismatching geo. Residential proxies that belong to another region or incorrect targeting - and a website shows different goods, prices, and delivery options.</p></li><li><p>Wrong currency. Prices shown in dollars instead of euros, which lands in the currency field as a number.</p></li><li><p>Outdated prices. Response from a CDN cache or an old version of a page provided by a website purposely.</p></li><li><p>Empty pages with a 200 status code. &#8220;No goods were found&#8221; instead of a catalog - and such a response may pass as a legitimate result.</p></li></ul><p>86% of DataImpulse&#8217;s respondents dealt with at least one scenario. 33% of them noticed missing pages or goods. 57% of users learned about the problem from alerts or auto-monitoring. Basic validation worked for only 18% of cases. 14% of users learned about the problem via another team, 9% through their clients, and 13% accidentally.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Pw6q!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd7fe21-e1b2-45ff-a6ad-3a5a52b321b1_2048x956.png 424w, https://substackcdn.com/image/fetch/$s_!Pw6q!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd7fe21-e1b2-45ff-a6ad-3a5a52b321b1_2048x956.png 848w, https://substackcdn.com/image/fetch/$s_!Pw6q!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd7fe21-e1b2-45ff-a6ad-3a5a52b321b1_2048x956.png 1272w, https://substackcdn.com/image/fetch/$s_!Pw6q!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd7fe21-e1b2-45ff-a6ad-3a5a52b321b1_2048x956.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Pw6q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd7fe21-e1b2-45ff-a6ad-3a5a52b321b1_2048x956.png" width="1456" height="680" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ecd7fe21-e1b2-45ff-a6ad-3a5a52b321b1_2048x956.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:680,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Pw6q!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd7fe21-e1b2-45ff-a6ad-3a5a52b321b1_2048x956.png 424w, https://substackcdn.com/image/fetch/$s_!Pw6q!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd7fe21-e1b2-45ff-a6ad-3a5a52b321b1_2048x956.png 848w, https://substackcdn.com/image/fetch/$s_!Pw6q!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd7fe21-e1b2-45ff-a6ad-3a5a52b321b1_2048x956.png 1272w, https://substackcdn.com/image/fetch/$s_!Pw6q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fecd7fe21-e1b2-45ff-a6ad-3a5a52b321b1_2048x956.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3>What do blocks cost</h3><p>Proxy bills rarely show the price of blocks. Price per GB may be low, but when it takes 3 requests to get a response, the real price of a data unit is three times higher. Add to that challenge pages and browser rendering - all of which also consume traffic.</p><p>Not all scraping teams count it this way. Only 24% of respondents track accurate prices for usable records and 22% track approximate prices. 51% of users only account for price per GB or 1 request, meaning they track the price of resources, but not results.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!x1n6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c865f73-cb9b-4413-b177-884d279c6356_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!x1n6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c865f73-cb9b-4413-b177-884d279c6356_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!x1n6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c865f73-cb9b-4413-b177-884d279c6356_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!x1n6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c865f73-cb9b-4413-b177-884d279c6356_2048x1152.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!x1n6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c865f73-cb9b-4413-b177-884d279c6356_2048x1152.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3c865f73-cb9b-4413-b177-884d279c6356_2048x1152.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!x1n6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c865f73-cb9b-4413-b177-884d279c6356_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!x1n6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c865f73-cb9b-4413-b177-884d279c6356_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!x1n6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c865f73-cb9b-4413-b177-884d279c6356_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!x1n6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3c865f73-cb9b-4413-b177-884d279c6356_2048x1152.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Time is another thing blocks consume. 36% of DataImpulse&#8217;s users say that they spend less than 10% of working hours dealing with blocks; 28% of users invest from 10 to 25% of their time. At the same time, 12% of respondents sacrifice 25-50% of their time to blocks and 6% - even more than half their business hours. In other words, one out of five teams waste at least 25% of their time not collecting data, but fighting for access to it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ei4q!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a274c81-fdd7-4f15-9085-0b9a43eb2b74_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!ei4q!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a274c81-fdd7-4f15-9085-0b9a43eb2b74_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!ei4q!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a274c81-fdd7-4f15-9085-0b9a43eb2b74_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!ei4q!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a274c81-fdd7-4f15-9085-0b9a43eb2b74_2048x1152.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ei4q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a274c81-fdd7-4f15-9085-0b9a43eb2b74_2048x1152.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/5a274c81-fdd7-4f15-9085-0b9a43eb2b74_2048x1152.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=wsc_guestpost_nov&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!ei4q!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a274c81-fdd7-4f15-9085-0b9a43eb2b74_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!ei4q!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a274c81-fdd7-4f15-9085-0b9a43eb2b74_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!ei4q!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a274c81-fdd7-4f15-9085-0b9a43eb2b74_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!ei4q!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5a274c81-fdd7-4f15-9085-0b9a43eb2b74_2048x1152.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h3>To summarize</h3><p>Blocks often do not stop a scraper, but quietly spoil data. Residential proxies solve the IP reputation problem, but success depends on the whole stack: fingerprinting, sessions, geolocation. So, status code 200 OK and low price per GB are good, but the real metrics are whether the data you get matches what real users see and the cost per usable record.</p>]]></content:encoded></item><item><title><![CDATA[CSS Selectors vs. XPath: What’s Better for Web Scraping?]]></title><description><![CDATA[Neither syntax saves a brittle selector. Here&#8217;s what actually survives a website redesign.]]></description><link>https://www.scraping.club/p/css-selectors-vs-xpath-scraping</link><guid isPermaLink="false">https://www.scraping.club/p/css-selectors-vs-xpath-scraping</guid><dc:creator><![CDATA[Federico Trotta]]></dc:creator><pubDate>Mon, 09 Nov 2026 10:30:37 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/cdb34154-639a-4b2c-b4c9-94beb204b6b3_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>CSS selectors and XPath can both find the same product card on the same web page, and for most scrapers they return the same data. So &#8220;Which one is better?&#8221; is not the correct question to ask.</p><p>Three questions settle the choice between the two:</p><ul><li><p>What identifies the record you want?</p></li><li><p>What do you need back?</p></li><li><p>Does your tool support the expression you&#8217;re about to write (which is a different question from whether a browser supports it)?</p></li></ul><p>When those three leave you a genuine choice, the rule of thumb is the following:</p><ul><li><p>Start with CSS selectors: They match the way you already look at a page, class matching is built into the syntax, and they handle ordinary extraction with less to read.</p></li><li><p>Reach for XPath per query rather than per project: When the rule depends on text, on moving up or back through the tree, or on filtering before positioning.</p></li></ul><p>This article starts with the tree that CSS selectors and XPath expressions query, covers each language&#8217;s syntax against the same running example, compares them directly, and finishes with the question that matters more than either syntax: &#8220;What makes a selector survive a website redesign?&#8221;</p><p>Let&#8217;s get into this!</p><div><hr></div><div class="callout-block" data-callout="true"><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/B5JPeJ">Decodo&#8217;s Web Data API</a></strong> turns any site into clean, structured data your models can actually use.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/B5JPeJ" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png" width="716" height="286.4" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:280,&quot;width&quot;:700,&quot;resizeWidth&quot;:716,&quot;bytes&quot;:105634,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;text&quot;:&quot;Activate your free plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/B5JPeJ"><span>Activate your free plan</span></a></p></div><div><hr></div><h2>What Is the DOM?</h2><p>Before you can choose between CSS selectors and XPath expressions, you need to understand what they do at a high level.</p><p>Both languages query a parsed document tree, and the shape of that tree decides what your expressions can find. In a browser, that tree is exposed through the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model">Document Object Model (DOM)</a>: A programming interface for accessing and modifying a document&#8217;s elements, attributes, and text. Thanks to this, a product card on a website becomes an element with other elements nested inside it. Its name becomes text within one of those elements. Its destination URL becomes an attribute on a link.</p><p>So, you&#8217;re dealing with elements, nesting, attributes, and text. And that mapping is the entire vocabulary you&#8217;re about to query with. For example, consider a catalog containing two products:</p><pre><code><code>&lt;section id="catalog"&gt;
  &lt;article class="product" data-product-id="p101"&gt;
    &lt;h2 class="product-name"&gt;
      &lt;a href="/products/trail-backpack"&gt;Trail Backpack&lt;/a&gt;
    &lt;/h2&gt;
    &lt;p class="price"&gt;&#8364;79.00&lt;/p&gt;
  &lt;/article&gt;

  &lt;article class="product" data-product-id="p102"&gt;
    &lt;h2 class="product-name"&gt;
      &lt;a href="/products/camping-lantern"&gt;Camping Lantern&lt;/a&gt;
    &lt;/h2&gt;
    &lt;p class="price"&gt;&#8364;24.00&lt;/p&gt;
  &lt;/article&gt;
&lt;/section&gt;</code></code></pre><p>The element relationships are:</p><ul><li><p><strong>Parent:</strong> The <code>&lt;section&gt;</code> is the parent of both <code>&lt;article&gt;</code> elements. A parent sits exactly one level above its child.</p></li><li><p><strong>Child:</strong> Each <code>&lt;article&gt;</code> has an <code>&lt;h2&gt;</code> and a <code>&lt;p&gt;</code> as element children.</p></li><li><p><strong>Sibling:</strong> The two <code>&lt;article&gt;</code> elements share the same parent, so they&#8217;re siblings. Within each product card, the <code>&lt;h2&gt;</code> and <code>&lt;p&gt;</code> are also siblings.</p></li><li><p><strong>Descendant:</strong> Each <code>&lt;a&gt;</code> is a descendant of its <code>&lt;h2&gt;</code>, its <code>&lt;article&gt;</code>, and the <code>&lt;section&gt;</code>. A descendant can sit one or several levels below an ancestor.</p></li><li><p><strong>Ancestor:</strong> From the link&#8217;s perspective, its enclosing <code>&lt;h2&gt;</code>, <code>&lt;article&gt;</code>, and <code>&lt;section&gt;</code> are ancestors.</p></li></ul><p>Below is a schema to help you visualize the structure:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Oaq2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Oaq2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!Oaq2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!Oaq2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!Oaq2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Oaq2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png" width="1456" height="971" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:971,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A simplified DOM subtree for a product card, showing element relationships and text nodes by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A simplified DOM subtree for a product card, showing element relationships and text nodes by Federico Trotta" title="A simplified DOM subtree for a product card, showing element relationships and text nodes by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!Oaq2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png 424w, https://substackcdn.com/image/fetch/$s_!Oaq2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png 848w, https://substackcdn.com/image/fetch/$s_!Oaq2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png 1272w, https://substackcdn.com/image/fetch/$s_!Oaq2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F50fd2711-770b-4f95-b15c-63e660f6970e_1536x1024.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A simplified DOM subtree for a product card, showing element relationships and text nodes</figcaption></figure></div><p>These relationships describe elements, but the DOM also contains text nodes. For instance, the words <code>Trail Backpack</code> form a text node inside the first <code>&lt;a&gt;</code>. Whitespace between tags can also create text nodes, which matters when you distinguish an element&#8217;s children from all its child nodes.</p><div><hr></div><div class="callout-block" data-callout="true"><p style="text-align: center;"><strong><a href="https://dashboard.datafuel.ai/en/register?ref=GN4DENZK">Datafuel</a></strong> is The Fuel for Web Scraping &amp; AI Agents, with proxies and scraping APIs.</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://dashboard.datafuel.ai/en/register?ref=GN4DENZK" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aHRN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58937c54-1895-48dc-9526-fb4f398e6624_336x96.png 424w, https://substackcdn.com/image/fetch/$s_!aHRN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58937c54-1895-48dc-9526-fb4f398e6624_336x96.png 848w, https://substackcdn.com/image/fetch/$s_!aHRN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58937c54-1895-48dc-9526-fb4f398e6624_336x96.png 1272w, https://substackcdn.com/image/fetch/$s_!aHRN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58937c54-1895-48dc-9526-fb4f398e6624_336x96.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!aHRN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58937c54-1895-48dc-9526-fb4f398e6624_336x96.png" width="440" height="125.71428571428571" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/58937c54-1895-48dc-9526-fb4f398e6624_336x96.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:96,&quot;width&quot;:336,&quot;resizeWidth&quot;:440,&quot;bytes&quot;:2715,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://dashboard.datafuel.ai/en/register?ref=GN4DENZK&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223138976?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58937c54-1895-48dc-9526-fb4f398e6624_336x96.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!aHRN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58937c54-1895-48dc-9526-fb4f398e6624_336x96.png 424w, https://substackcdn.com/image/fetch/$s_!aHRN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58937c54-1895-48dc-9526-fb4f398e6624_336x96.png 848w, https://substackcdn.com/image/fetch/$s_!aHRN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58937c54-1895-48dc-9526-fb4f398e6624_336x96.png 1272w, https://substackcdn.com/image/fetch/$s_!aHRN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58937c54-1895-48dc-9526-fb4f398e6624_336x96.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://dashboard.datafuel.ai/en/register?ref=GN4DENZK&quot;,&quot;text&quot;:&quot;Discover Datafuel Scraping API&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://dashboard.datafuel.ai/en/register?ref=GN4DENZK"><span>Discover Datafuel Scraping API</span></a></p></div><div><hr></div><h2>XPath: Syntax, Expressions, and Practical Usage</h2><p><a href="https://www.w3.org/TR/1999/REC-xpath-19991116/">XPath, short for XML Path Language</a>, is the language to reach for when a record is defined by where it sits. An expression describes a path through the document tree, and each step along that path says where to look and which nodes to keep. That buys you two things CSS selectors don&#8217;t have: Movement upward and backward through the tree, and conditions on text.</p><p>For example, with respect to the HTML shown in the previous chapter, you can start from the catalog section, find its product cards, and select the links inside their headings:</p><pre><code><code>//section[@id='catalog']/article/h2/a</code></code></pre><p>In the above snippet:</p><ul><li><p><code>//section</code> finds section elements anywhere in the document.</p></li><li><p>The predicate <code>[@id='catalog']</code> keeps the ones whose <code>id</code> is <code>catalog</code>.</p></li><li><p><code>/article/h2/a</code> follows direct-child relationships from each survivor down to the links.</p></li></ul><p>Predicates are anything inside the square brackets. They&#8217;re how XPath filters what a step found, and most real-world expressions contain several of them.</p><p>Beyond that, the working vocabulary is small:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xqho!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xqho!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png 424w, https://substackcdn.com/image/fetch/$s_!xqho!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png 848w, https://substackcdn.com/image/fetch/$s_!xqho!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png 1272w, https://substackcdn.com/image/fetch/$s_!xqho!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xqho!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png" width="1200" height="566.2087912087912" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/01182985-2461-416c-ac3e-291c22e330c3_1623x766.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:687,&quot;width&quot;:1456,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:154925,&quot;alt&quot;:&quot;XPath&#8217;s vocabulary by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223138976?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-large" alt="XPath&#8217;s vocabulary by Federico Trotta" title="XPath&#8217;s vocabulary by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!xqho!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png 424w, https://substackcdn.com/image/fetch/$s_!xqho!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png 848w, https://substackcdn.com/image/fetch/$s_!xqho!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png 1272w, https://substackcdn.com/image/fetch/$s_!xqho!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01182985-2461-416c-ac3e-291c22e330c3_1623x766.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">XPath&#8217;s vocabulary</figcaption></figure></div><p>If you&#8217;d like a gentler walkthrough of these building blocks first, <a href="https://www.scraping.club/p/xpath-css-selectors-web-scraping">XPath and CSS selectors in web scraping</a> covers them step by step.</p><p>Now, let&#8217;s make the catalog a little more realistic. Adding a stock flag, a second class, a shipping note, and a product name split across two nodes gives us enough structure to exercise every row of that table:</p><pre><code><code> &lt;section id="catalog"&gt;
  &lt;article class="product featured"
           data-product-id="p101"
           data-stock="available"&gt;
    &lt;h2 class="product-name"&gt;
      &lt;a href="/products/trail-backpack"&gt;
        &lt;span&gt;Trail&lt;/span&gt; Backpack
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;p class="price" data-currency="EUR"&gt;&#8364;79.00&lt;/p&gt;
    &lt;p class="shipping"&gt;Free shipping&lt;/p&gt;
  &lt;/article&gt;

  &lt;article class="product"
           data-product-id="p102"
           data-stock="unavailable"&gt;
    &lt;h2 class="product-name"&gt;
      &lt;a href="/products/camping-lantern"&gt;Camping Lantern&lt;/a&gt;
    &lt;/h2&gt;
    &lt;p class="price" data-currency="EUR"&gt;&#8364;24.00&lt;/p&gt;
    &lt;p class="shipping"&gt;Shipping calculated at checkout&lt;/p&gt;
  &lt;/article&gt;
&lt;/section&gt;</code></code></pre><p>Selecting the available products takes a single attribute predicate as follows:</p><pre><code><code>//article[@data-stock='available']</code></code></pre><p>With the expression above, you select the first article. If you want a specific product ID, combine conditions using <code>and</code>:</p><pre><code><code>//article[@data-stock='available' and @data-product-id]</code></code></pre><p>Note that <code>@data-product-id</code> on its own only proves the attribute exists, not that it holds a usable value.</p><p>Class matching is where XPath pays for its power. An HTML <code>class</code> attribute, in fact, holds several whitespace-separated tokens, and both obvious expressions get it wrong:</p><ul><li><p><code>@class='product'</code> compares the entire attribute, so it misses the first article, whose value is <code>product featured</code>.</p></li><li><p><code>contains(@class, 'product')</code> tests a substring, so it will happily return <code>product-summary</code> next to your product cards.</p></li></ul><p>This means that matching a complete token takes the longer form, which is the following one:</p><pre><code><code> //article[
  contains(concat(' ', normalize-space(@class), ' '), ' product ')
]</code></code></pre><p>This pads the attribute with spaces and collapses its whitespace. This way, <code>' product '</code> only matches a whole token, selects both articles, and nothing that merely contains the string.</p><h3>Advantages of XPath</h3><ul><li><p><strong>It navigates upward as well as downward</strong>: Parent and ancestor axes let you find a product card through a nested link or label.</p></li><li><p><strong>It combines several kinds of conditions</strong>: Predicates can test attributes, text, position, and nested structure in the same expression, such as selecting available products that contain a particular link.</p></li><li><p><strong>It expresses relationships between nodes</strong>: Sibling axes let you find a value beside a known label or a shipping notice after a price, rather than relying only on the target element&#8217;s attributes.</p></li><li><p><strong>It can select the values you need directly</strong>: Expressions ending in <code>/@href</code> or <code>/text()</code> select attributes or text nodes when the library exposes those results. String functions can also return a scalar value instead of an element.</p></li><li><p><strong>It suits relationship-driven extraction and XML</strong>: XPath is useful when a document&#8217;s structure provides better anchors than its classes or IDs. Its namespace-aware node tests also support XML extraction when you configure the required namespace bindings.</p></li></ul><h3>Disadvantages of XPath</h3><ul><li><p><strong>Long absolute paths are fragile</strong>: A path tied to every container between <code>/html</code> and a field can break when a site inserts, removes, or rearranges intermediate elements.</p></li><li><p><strong>Complex expressions take effort to maintain</strong>: Several nested predicates and axes can make a query difficult to review or debug. Splitting extraction into a record-selection step and smaller field queries often makes it clearer.</p></li><li><p><strong>Features depend on the version and implementation</strong>: XPath 2.0 functions and library-specific extensions aren&#8217;t available everywhere. Browser-native evaluation generally provides XPath 1.0, so you can&#8217;t assume it supports regex functions or other newer features.</p></li><li><p><strong>Namespaces require explicit handling</strong>: In XPath 1.0, an unprefixed element name matches elements in no namespace. Namespaced XML, including documents with a default namespace, usually requires a prefix bound through the evaluator.</p></li><li><p><strong>Small syntax differences can change the result</strong>: <code>text()</code> isn&#8217;t the same as an element&#8217;s full string value, <code>node()</code> includes more than elements, and <code>[1]</code> depends on its position and axis. These expressions can return valid but unexpected results, so checking the matched nodes matters as much as checking the syntax.</p></li></ul><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://bundle.scraping.club/&quot;,&quot;text&quot;:&quot;Scraping Bundle - 20k+$ in tools&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://bundle.scraping.club/"><span>Scraping Bundle - 20k+$ in tools</span></a></p><div><hr></div><h2>CSS Selectors: Syntax, Expressions, and Practical Usage</h2><p>CSS selectors are the default language for scraping HTML, and for a mundane reason: They match the way you already look at a page. Indeed, if you can open <a href="https://developer.chrome.com/docs/devtools">DevTools</a> on your browser, click an element, and read <code>article.product</code>, you can already write the selector that finds it.</p><p>With this syntax, a selector matches tag names, classes, IDs, attributes, and the relationships between elements. To find the links inside product headings:</p><pre><code><code>#catalog article.product h2 a</code></code></pre><p>Here:</p><ul><li><p><code>#catalog</code> matches the element whose ID is <code>catalog</code>.</p></li><li><p><code>article.product</code> matches <code>article</code> elements carrying the <code>product</code> class.</p></li><li><p>The spaces describe descendant relationships, so <code>h2 a</code> finds links anywhere inside those headings even if another element sits between the heading and the link.</p></li></ul><p>Note that one property shapes every other decision in this section: Standard CSS selects elements, and nothing else. It returns nodes, not the attribute values or text strings you ultimately want. So, once you have a link, retrieving its <code>href</code> or its text is a separate operation performed by your scraping tool&#8217;s API.</p><p>In this context, note that there are libraries that bolt on extraction syntax. Those extensions are convenient, but they aren&#8217;t CSS, and they don&#8217;t move between tools. This means that whichever tool you&#8217;re using determines how much of your extraction code is portable - and the tool-by-tool comparison later in this article covers exactly how far that stretches.</p><p>For now, let&#8217;s only focus on the patterns that cover nearly all routine extraction in the table below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HJzq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HJzq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png 424w, https://substackcdn.com/image/fetch/$s_!HJzq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png 848w, https://substackcdn.com/image/fetch/$s_!HJzq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png 1272w, https://substackcdn.com/image/fetch/$s_!HJzq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HJzq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png" width="727.9861450195312" height="620.5400615834935" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:797,&quot;width&quot;:935,&quot;resizeWidth&quot;:727.9861450195312,&quot;bytes&quot;:118937,&quot;alt&quot;:&quot;CSS&#8217;s syntax by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223138976?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-normal" alt="CSS&#8217;s syntax by Federico Trotta" title="CSS&#8217;s syntax by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!HJzq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png 424w, https://substackcdn.com/image/fetch/$s_!HJzq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png 848w, https://substackcdn.com/image/fetch/$s_!HJzq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png 1272w, https://substackcdn.com/image/fetch/$s_!HJzq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb97decb0-6582-49e9-96ce-ea0f652884d1_935x797.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">CSS&#8217;s syntax</figcaption></figure></div><p>Two rows earn more attention than the others:</p><ul><li><p><code>[attr~="value"]</code> is the token matcher that XPath has to rebuild from three nested function calls, and <code>.class</code> gives you the same behaviour for free.</p></li><li><p><code>:has()</code> is the only relational escape hatch in the list, and it&#8217;s also the least portable.</p></li></ul><p>To make some examples, let&#8217;s make the catalog realistic again, this time by adding a product image to each card:</p><pre><code><code>&lt;section id="catalog"&gt;
  &lt;article class="product featured"
           data-product-id="p101"
           data-stock="available"&gt;
    &lt;a class="product-image" href="/products/trail-backpack"&gt;
      &lt;img src="/images/trail-backpack.jpg" alt="Trail Backpack"&gt;
    &lt;/a&gt;
    &lt;h2 class="product-name"&gt;
      &lt;a href="/products/trail-backpack"&gt;
        &lt;span&gt;Trail&lt;/span&gt; Backpack
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;p class="price" data-currency="EUR"&gt;&#8364;79.00&lt;/p&gt;
    &lt;p class="shipping"&gt;Free shipping&lt;/p&gt;
  &lt;/article&gt;

  &lt;article class="product"
           data-product-id="p102"
           data-stock="unavailable"&gt;
    &lt;a class="product-image" href="/products/camping-lantern"&gt;
      &lt;img src="/images/camping-lantern.jpg" alt="Camping Lantern"&gt;
    &lt;/a&gt;
    &lt;h2 class="product-name"&gt;
      &lt;a href="/products/camping-lantern"&gt;Camping Lantern&lt;/a&gt;
    &lt;/h2&gt;
    &lt;p class="price" data-currency="EUR"&gt;&#8364;24.00&lt;/p&gt;
    &lt;p class="shipping"&gt;Shipping calculated at checkout&lt;/p&gt;
  &lt;/article&gt;
&lt;/section&gt;</code></code></pre><p>Select the repeated records first with the following:</p><pre><code><code>#catalog &gt; article.product</code></code></pre><p>That matches both product cards:</p><ul><li><p><code>.product</code> tests a complete class token, so it matches <code>class="product"</code> and <code>class="product featured"</code> alike without also matching <code>class="product-summary"</code>.</p></li></ul><p>From there, each field has its own query:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XLkD!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XLkD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png 424w, https://substackcdn.com/image/fetch/$s_!XLkD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png 848w, https://substackcdn.com/image/fetch/$s_!XLkD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png 1272w, https://substackcdn.com/image/fetch/$s_!XLkD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!XLkD!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png" width="1200" height="311.0112359550562" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:346,&quot;width&quot;:1335,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:63632,&quot;alt&quot;:&quot;CSS queries examples by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223138976?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-large" alt="CSS queries examples by Federico Trotta" title="CSS queries examples by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!XLkD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png 424w, https://substackcdn.com/image/fetch/$s_!XLkD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png 848w, https://substackcdn.com/image/fetch/$s_!XLkD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png 1272w, https://substackcdn.com/image/fetch/$s_!XLkD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca93a171-eb6c-40c0-9621-1ef90425ba27_1335x346.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">CSS query examples</figcaption></figure></div><p>Run those five against the first card, and you get <code>Trail Backpack</code>, <code>&#8364;79.00</code>, <code>/products/trail-backpack</code>, <code>/images/trail-backpack.jpg</code> and <code>p101</code>:</p><ul><li><p>Three of the five are attributes;</p></li><li><p>Two are text;</p></li><li><p>None of those values came from the selector itself.</p></li></ul><p>Two further assumptions are embedded in the name: The code has to include descendant text, because <code>Backpack</code> and <code>Trail</code> live in different nodes, and it should normalize whitespace. The link and image paths are relative, and reading an attribute does not resolve them; so, joining them against the page URL is a step you write yourself.</p><p>This is also where the scoping decision should be made deliberately when you&#8217;re about to start a new scraping project. It&#8217;s easier to select the cards, then query within each one &#8212; <code>.product-name &gt; a</code> finds that card&#8217;s name link when your library evaluates against the card&#8217;s subtree. However, collecting all names and all prices separately and zipping them by position gives the same output today, but misaligns every later record when one product lacks a price.</p><p>Also, choosing the right relationship makes these queries more precise. For instance, compare the following:</p><pre><code><code>.product .price 

.product &gt; .price</code></code></pre><p>Both match here, but they mean different things:</p><ul><li><p>The first says &#8220;any price element below a product card&#8221;.</p></li><li><p>The second says &#8220;the price element that belongs directly to this card&#8221;.</p></li></ul><p>If a card later embeds a recommended product with its own price, the descendant version collects that nested price too. The child version excludes it, and would instead stop matching if the site wrapped the main price in a new container.</p><p>Neither failure is hypothetical: Both can happen on real listing pages. So, which one you can tolerate depends on what identifies the field, so state that and stop.</p><p>As another example, note that attribute conditions narrow the record set without touching field extraction at all:</p><pre><code><code>article.product[data-stock="available"]</code></code></pre><p>So, if you also want to require a product ID, add an attribute-existence condition as follows:</p><pre><code><code>article.product\[data-stock="available"\][data-product-id]</code></code></pre><p>Like the equivalent XPath condition, <code>[data-product-id]</code> checks whether the attribute exists, not whether it contains a useful value. You still need to validate the extracted ID.</p><h3>Advantages of CSS Selectors</h3><ul><li><p><strong>The syntax is familiar</strong>: If you&#8217;ve worked with HTML and stylesheets, selectors such as <code>.product</code>, <code>#catalog</code>, and <code>[data-stock="available"]</code> already describe elements in a recognizable way.</p></li><li><p><strong>Common queries stay concise</strong>: Expressions such as <code>.product-name &gt; a</code> make routine extraction easy to read and review without lengthy class-matching predicates.</p></li><li><p><strong>They cover everyday extraction needs</strong>: Classes, IDs, attributes, and combinators handle many product listings, article pages, tables, and other repeated structures.</p></li><li><p><strong>They integrate with common tools</strong>: Browser APIs such as <code>[querySelectorAll()](https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll)</code> accept CSS selectors, as do popular scraping and automation libraries, although their supported features and return types differ.</p></li><li><p><strong>They can avoid unnecessary structural dependencies</strong>: A selector anchored to a stable attribute or meaningful class can keep working when unrelated containers elsewhere on the page change.</p></li></ul><h3>Disadvantages of CSS Selectors</h3><ul><li><p><strong>It can&#8217;t look backward and it can&#8217;t read text</strong>: There&#8217;s no general way to move from a matched link to its parent or an arbitrary ancestor, and no standard selector for &#8220;elements containing this string&#8221;. <code>:has()</code> covers part of the first gap by selecting an enclosing element based on its contents; the second gap means filtering text in application code or reaching for tool-specific extensions that won&#8217;t travel.</p></li><li><p><strong>Text-based matching needs another approach</strong>: Standard CSS doesn&#8217;t provide a general selector for elements containing a particular text string. You&#8217;ll usually filter through your application code or use tool-specific extensions that aren&#8217;t portable.</p></li><li><p><strong>Complex relationships can become hard to follow</strong>: Several nested pseudo-classes and combinators can make a selector difficult to maintain, especially when it encodes a page&#8217;s layout rather than a stable identifying feature.</p></li><li><p><strong>Advanced syntax isn&#8217;t equally portable</strong>: A selector that works in a browser may fail in a library with a smaller supported subset. Library-specific extraction extensions also won&#8217;t necessarily work in browser query APIs.</p></li><li><p><strong>Small assumptions can produce incorrect matches</strong>: Exact class-attribute comparisons miss elements with additional classes, positional pseudo-classes count siblings rather than arbitrary result sets, generated classes can change, and broad descendant selectors can include fields from nested records.</p></li></ul><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><h2>XPath vs. CSS Selectors: In-Depth Comparison</h2><p>CSS selectors and XPath can handle many of the same scraping tasks, but they describe those tasks differently:</p><ul><li><p>CSS usually starts with an element&#8217;s identifying features, such as its class or attributes.</p></li><li><p>XPath combines node tests with paths and predicates, giving you more control over navigation and filtering.</p></li></ul><p>This section compares them on five axes: whether the same selection ends up readable, what each can navigate to, what each returns, what your tool actually supports, and how fast either one evaluates.</p><h3>Equivalent Selections, Different Grammar</h3><p>Selecting available product cards is straightforward in both syntaxes:</p><pre><code><code># CSS  
article.product[data-stock="available"]  

# XPath  
//article[  
  contains(concat(' ', normalize-space(@class), ' '), ' product ')  
  and @data-stock='available'  
]  </code></code></pre><p>Both select the same articles, but:</p><ul><li><p>CSS expresses the class condition more naturally because class matching is built into its syntax.</p></li><li><p>XPath needs a longer condition to tell the <code>product</code> class apart from values such as <code>product-summary</code>.</p></li></ul><p>That difference shrinks as soon as the task depends on text or relationships rather than classes. For instance, to find a product through its name, XPath expresses the condition directly:</p><pre><code><code>//article[.//a[normalize-space(.)='Trail Backpack']]  </code></code></pre><p>Instead, standard CSS can&#8217;t test that text value. You&#8217;d select candidate cards and filter their text in application code, or use a tool-specific text-matching feature.</p><p>Also, when the condition depends on a descendant&#8217;s <em>attributes</em> instead, a supported <code>:has()</code> selector keeps the query concise:</p><pre><code><code>article.product:has(a[href="/products/trail-backpack"])  </code></code></pre><p>So the more readable option depends on what identifies the record:</p><ul><li><p>CSS reads better when classes and attributes provide clear anchors.</p></li><li><p>XPath reads better when you need to describe text conditions, ancestors, or relationships between several nodes.</p></li></ul><p>The table below shows the two syntaxes side by side against the same catalog. The expressions assume document-level evaluation unless a row specifies a product-card context:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!8v8c!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8v8c!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png 424w, https://substackcdn.com/image/fetch/$s_!8v8c!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png 848w, https://substackcdn.com/image/fetch/$s_!8v8c!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png 1272w, https://substackcdn.com/image/fetch/$s_!8v8c!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8v8c!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png" width="1200" height="453.2967032967033" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:550,&quot;width&quot;:1456,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:134950,&quot;alt&quot;:&quot;CSS Vs XPath syntax side-by-side by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223138976?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-large" alt="CSS Vs XPath syntax side-by-side by Federico Trotta" title="CSS Vs XPath syntax side-by-side by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!8v8c!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png 424w, https://substackcdn.com/image/fetch/$s_!8v8c!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png 848w, https://substackcdn.com/image/fetch/$s_!8v8c!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png 1272w, https://substackcdn.com/image/fetch/$s_!8v8c!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe7f6ab73-29f2-4ac9-b1cc-54c419dc5dfa_1593x602.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">CSS Vs XPath syntax side-by-side</figcaption></figure></div><p>Read the rows as a cost curve rather than a scorecard. The top half is where CSS is shorter, the middle rows are roughly even, and the shipping paragraph is where XPath&#8217;s verbosity becomes obvious.</p><h3>Navigation: The Direction CSS Can&#8217;t Go</h3><p>Both syntaxes describe descendants, direct children, and following siblings. XPath adds explicit axes for parents, ancestors, and preceding siblings, and that&#8217;s a capability difference rather than a spelling difference.</p><p>If you already hold a link as your context node, you can select its nearest enclosing article with:</p><pre><code><code>ancestor::article[1]  </code></code></pre><p>There&#8217;s no equivalent standard CSS expression that traverses upward from that link. You can sometimes rewrite the task as <code>article:has(...)</code>, but that selects candidate articles based on a relationship rather than navigating from the link itself. Alternatively, your library may expose a parent-traversal method outside its selector interface entirely.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://substack.thewebscraping.club/p/consulting&quot;,&quot;text&quot;:&quot;Need help with your scraping project?&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://substack.thewebscraping.club/p/consulting"><span>Need help with your scraping project?</span></a></p><div><hr></div><h3>What Comes Back: Elements, Attributes and Strings</h3><p>The distinction between selector syntax and library functionality matters as much when extracting values:</p><ul><li><p>Standard CSS selects elements, so retrieving a link&#8217;s <code>href</code> requires another API operation.</p></li><li><p>XPath can select <code>@href</code> directly or return a string through a function, but only if the calling API accepts that result type.</p></li></ul><p>What&#8217;s interesting to note is that <a href="https://scrapy.org/">Scrapy</a>, the framework that has <a href="https://www.scraping.club/p/scrapy-ten-years-of-scraping-framework">stayed relevant for over a decade</a>, exposes both approaches:</p><pre><code><code># Scrapy with CSS
response.css(".product-name &gt; a::attr(href)").getall()  

# Scrapy with XPath
response.xpath(  
    "//*[contains(concat(' ', normalize-space(@class), ' '), "  
    "' product-name ')]/a/@href"  
).getall()  </code></code></pre><p>Here <code>::attr(href)</code> is Scrapy&#8217;s extraction extension, not standard CSS. So, you can&#8217;t paste it into <code>document.querySelectorAll()</code> and expect it to work. Likewise, an XPath expression ending in <code>/@href</code> works through Scrapy&#8217;s selector interface, but <a href="https://www.selenium.dev/">Selenium</a>&#8217;s XPath element locator expects elements rather than attribute nodes.</p><h3>Support Varies by Tool</h3><p>As previously introduced in this article, the tools themselves draw the boundaries you&#8217;re working inside. The following table summarizes the landscape of the different possibilities available:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!H-I8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!H-I8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png 424w, https://substackcdn.com/image/fetch/$s_!H-I8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png 848w, https://substackcdn.com/image/fetch/$s_!H-I8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png 1272w, https://substackcdn.com/image/fetch/$s_!H-I8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!H-I8!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png" width="1200" height="447.21030042918454" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:521,&quot;width&quot;:1398,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:109239,&quot;alt&quot;:&quot;CSS Vs XPath per tool by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223138976?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-large" alt="CSS Vs XPath per tool by Federico Trotta" title="CSS Vs XPath per tool by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!H-I8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png 424w, https://substackcdn.com/image/fetch/$s_!H-I8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png 848w, https://substackcdn.com/image/fetch/$s_!H-I8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png 1272w, https://substackcdn.com/image/fetch/$s_!H-I8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa1cafb9f-14c1-4e08-9a02-f9c28ffc5b29_1398x521.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">CSS Vs XPath per tool</figcaption></figure></div><p>A browser&#8217;s support, therefore, doesn&#8217;t establish what every scraping library supports. In other words, a browser may accept <code>:has()</code> while a CSS-to-XPath translator rejects it. Similarly, Beautiful Soup may accept a selector through Soup Sieve, but that doesn&#8217;t mean the same expression works in Scrapy, and a library&#8217;s XPath extension won&#8217;t necessarily work through native browser evaluation.</p><p>Portability also depends on what you&#8217;re querying. Beautiful Soup and Scrapy typically query trees parsed from downloaded HTML. Selenium and Playwright query a browser&#8217;s DOM, which scripts may have modified, and <a href="https://www.scraping.club/p/scraping-javascript-rendered-pages">knowing when a page really needs a headless browser</a> is its own skill.</p><h3>Performance: Measure It, Don&#8217;t Assume It</h3><p>Even if we&#8217;re talking about querying something (the DOM), let&#8217;s state this clearly: Syntax doesn&#8217;t determine speed. A short CSS selector isn&#8217;t automatically faster than a longer XPath expression, and XPath&#8217;s extra navigation options don&#8217;t automatically make it slower.</p><p>Based on that, for routine extraction, you can just write the expression that clearly identifies the right elements in your actual tool. If it stays understandable, handles the document&#8217;s structure, and returns the expected records, you&#8217;ve addressed the part that usually decides whether a scraper holds up. Execution speed becomes a separate question, and you measure it when selector evaluation contributes meaningfully to the scraper&#8217;s workload.</p><h3>Are CSS Selectors Easier to Learn and Use?</h3><p>For common tasks, yes, and the earlier sections have been building that case. An ID uses <code>#</code>, a class uses <code>.</code>, an attribute condition goes inside square brackets, and the same syntax you use in stylesheets and DevTools works in a scraper.</p><p>Conversely, XPath starts from a different point: Paths through the tree, with predicates filtering the nodes along the way. Simple attribute conditions are still easy to follow, but some everyday HTML tasks need more explanation.</p><p>The following expressions select the same elements from the same catalog:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vNnK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vNnK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png 424w, https://substackcdn.com/image/fetch/$s_!vNnK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png 848w, https://substackcdn.com/image/fetch/$s_!vNnK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png 1272w, https://substackcdn.com/image/fetch/$s_!vNnK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!vNnK!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png" width="1200" height="402.5995125913891" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:413,&quot;width&quot;:1231,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:70307,&quot;alt&quot;:&quot;CSS Vs XPath per scraping goal by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223138976?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-large" alt="CSS Vs XPath per scraping goal by Federico Trotta" title="CSS Vs XPath per scraping goal by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!vNnK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png 424w, https://substackcdn.com/image/fetch/$s_!vNnK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png 848w, https://substackcdn.com/image/fetch/$s_!vNnK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png 1272w, https://substackcdn.com/image/fetch/$s_!vNnK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F582a6e4a-26cb-4cfa-b7bb-da182bdc39b5_1231x413.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">CSS Vs XPath per scraping goal</figcaption></figure></div><p>For these six tasks, CSS needs less syntax, and row two is the clearest case: <code>.product</code> is easier to recognize than the XPath class-token condition, especially when the same condition appears several times in one query.</p><p>Readability is where the advantage gets complicated, though, because adding conditions exposes what each syntax costs. For example, selecting available articles that contain an image is manageable in either:</p><pre><code><code># CSS  
article[data-stock="available"]:has(img[src])  
  
# XPath  
//article[@data-stock='available' and .//img[@src]]  </code></code></pre><p>However:</p><ul><li><p>The CSS expression requires <code>:has()</code> support, which varies by tool.</p></li><li><p>The XPath expression uses a descendant test inside its predicate, which standard XPath 1.0 has supported for decades.</p></li></ul><p>In that case, if you only add one more requirement &#8212; for example, an article containing a link whose normalized text is <code>Trail Backpack</code> &#8212; and XPath keeps every condition in the query, you need the following expression:</p><pre><code><code>//article[  
  @data-stock='available'  
  and .//img[@src]  
  and .//a[normalize-space(.)='Trail Backpack']  
]  </code></code></pre><p>Standard CSS can&#8217;t express that final text condition, so the extraction splits into a selector plus a separate filtering step. At that point, XPath is easier to understand than CSS, because the expression keeps the related conditions together.</p><p>But that doesn&#8217;t mean every condition belongs in one query. If an expression becomes difficult to explain, select the records first and extract their fields separately.</p><h3>XPath Has More Advanced Filtering Logic</h3><p>The previous examples show where XPath offers more than a different spelling for the same selection: Its predicates combine attribute values, text, position, and relationships with other nodes, while its axes navigate beyond descendants and following siblings.</p><p>Now, suppose you need the first available article that contains a price paragraph with a currency attribute:</p><pre><code><code>(  
  //article[  
    @data-stock='available'  
    and ./p[  
      contains(concat(' ', normalize-space(@class), ' '), ' price ')  
      and @data-currency  
    ]  
  ]  
)[1]  </code></code></pre><p>The inner predicate identifies a suitable price paragraph. The outer predicate keeps available articles containing that paragraph. The final <code>[1]</code> selects the first article from the complete filtered result set &#8212; and where that <code>[1]</code> applies is the point worth dwelling on. <code>@data-currency</code> here only proves the attribute exists, so whether the currency and price text are usable remains validation your extraction code owns.</p><p>Now, look at the last row of the learnability table above: <code>#catalog &gt; article:nth-of-type(2)</code> versus <code>//*[@id='catalog']/article[2]</code>. Both count article siblings rather than selecting the second member of an arbitrary result set, and that distinction costs you in CSS. Now, consider this:</p><pre><code><code>#catalog &gt; article.product:nth-of-type(2)  </code></code></pre><p>This selects the second article child, but only if it also has the <code>product</code> class. So, if the first article isn&#8217;t a product card, it still counts toward that position, so the selector matches nothing at all.</p><p>What&#8217;s interesting to consider is that XPath can filter first and position afterwards:</p><pre><code><code>//*[@id='catalog']/article[  
  contains(concat(' ', normalize-space(@class), ' '), ' product ')  
][2]  </code></code></pre><p>Here <code>[2]</code> applies to the articles that passed the class condition. That difference matters as soon as a page mixes product cards with promotional articles, sponsored blocks, or other records sharing a tag name (which most real listing pages do, by the way&#8230;).</p><p>The same directional logic applies to reverse axes. From a price paragraph, the nearest preceding heading sibling is:</p><pre><code><code>preceding-sibling::h2[1]  </code></code></pre><p>Because the axis runs backward, <code>[1]</code> selects the closest matching heading rather than the earliest one in document order. Getting that backwards silently returns the wrong node.</p><p>Text conditions provide the other main distinction, and they connect back to the nested span in the first product name. You can identify that link and then select its enclosing product card:</p><pre><code><code>//a[normalize-space(.)='Trail Backpack']  
  /ancestor::article[  
    contains(concat(' ', normalize-space(@class), ' '), ' product ')  
  ][1]  </code></code></pre><p>As covered earlier, <code>normalize-space(.)</code> includes descendant text, so the nested <code>&lt;span&gt;</code> is accounted for. From the matching link, <code>ancestor::article[...]</code> finds enclosing product articles and <code>[1]</code> keeps the nearest one. That&#8217;s how you locate a record through a recognizable field even when the record itself has no unique identifying attribute (which is a common situation on pages where IDs are generated per session).</p><p>CSS handles some similar tasks with <code>:has()</code>. If you know the link&#8217;s destination, you can select its enclosing product article with:</p><pre><code><code>article.product:has(a[href="/products/trail-backpack"])  </code></code></pre><p>That&#8217;s often more concise than the XPath equivalent. But it tests an attribute rather than the link&#8217;s text, and it selects articles <em>containing</em> a matching link rather than traversing upward from an existing link context.</p><h3>Reliability and Maintainability: Which Selectors Break Less Often?</h3><p>Neither CSS nor XPath is inherently more resistant to website changes. What matters is which details the expression depends on, and how likely those details are to change.</p><p>A redesign can remove anything without touching the underlying product data, so generated class names and IDs can be worse still, changing between builds or even between sessions.</p><p>That&#8217;s why a field-specific attribute provides a better anchor:</p><pre><code><code># CSS  
[data-field="product-name"]  
  
# XPath  
//*[@data-field='product-name']  </code></code></pre><p>That attribute is hypothetical, so its advantage would come from its role as a field identifier, not from the <code>data-</code> prefix itself.</p><p>The same reasoning applies to semantic markup and structural relationships. A heading inside a product card may provide a useful anchor, but selecting every <code>&lt;h2&gt;</code> on the page is far too broad. Combining a stable record boundary with a meaningful field selector gives you better balance:</p><pre><code><code># CSS
article.product .product-name &gt; a  </code></code></pre><p>The same holds in XPath:</p><pre><code><code># XPath
/html/body/main/div/section[@id='catalog']/article/h2/a  </code></code></pre><p>Here&#8217;s a rule of thumb: Anchor to the nearest stable boundary that contains the record, not to the page structure above it. And for a better alternative - when applicable - you can choose to <a href="https://www.scraping.club/p/scraping-with-llms-gpt-vision">go beyond the DOM and LLMs and GPT vision for extracting your web data</a>.</p><p>Overall, the selectors that need less maintenance describe stable record boundaries and meaningful field relationships without depending on unrelated layout details. You can write those selectors in either CSS or XPath: The language helps you express the rule, but choosing and testing the rule is what makes it reliable. And for the times a redesign breaks even a well-chosen rule, <a href="https://www.scraping.club/p/building-self-healing-scrapers-with-gpt">self-healing scrapers built with AI</a> can take some of the repair work off your plate.</p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>When to Use CSS Selectors and When to Use XPath</h2><p>CSS selectors are usually the right starting point for routine HTML extraction: Classes, IDs, attributes, and predictable relationships are often enough to identify product cards and their fields. XPath becomes more useful when selection depends on text, ancestor navigation, or conditions involving several related nodes.</p><p>But the choice also depends on your tool. As introduced previously, a selector that&#8217;s clear and supported in one library may require a workaround in another. The following table summarizes where each approach fits and which exceptions to consider:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zljO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zljO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png 424w, https://substackcdn.com/image/fetch/$s_!zljO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png 848w, https://substackcdn.com/image/fetch/$s_!zljO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png 1272w, https://substackcdn.com/image/fetch/$s_!zljO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zljO!,w_2400,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png" width="1200" height="592.5824175824176" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/df100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:719,&quot;width&quot;:1456,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:128838,&quot;alt&quot;:&quot;When to choose CSS and when XPath: The complete landscape of choices by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223138976?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-large" alt="When to choose CSS and when XPath: The complete landscape of choices by Federico Trotta" title="When to choose CSS and when XPath: The complete landscape of choices by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!zljO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png 424w, https://substackcdn.com/image/fetch/$s_!zljO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png 848w, https://substackcdn.com/image/fetch/$s_!zljO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png 1272w, https://substackcdn.com/image/fetch/$s_!zljO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdf100ad0-bfaa-41d8-800e-f37ecd1c174f_1525x753.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">When to choose CSS and when XPath: The complete landscape of choices</figcaption></figure></div><p>Read the last column as the part that saves debugging time:</p><ul><li><p>Beautiful Soup makes CSS the straightforward option because there is no XPath interface.</p></li><li><p>Scrapy exposes both, so you can choose per query, and it&#8217;s easy to try by <a href="https://www.scraping.club/p/create-your-first-scraper-with-scrapy">building your first Scrapy scraper</a>.</p></li><li><p>Selenium and Playwright support both through element-finding interfaces, but their locator behavior isn&#8217;t identical to querying a static parsed document. The <a href="https://www.scraping.club/p/playwright-vs-selenium">Selenium vs Playwright comparison</a> shows where they differ in practice.</p></li></ul><p>Team familiarity belongs in the decision too. That&#8217;s because forcing a relationship-heavy query into CSS plus several application-level workarounds can make the extraction harder to understand than a relative XPath expression. In simple words, the syntax your maintainers can reason about confidently is the one that will still be correct in six months.</p><h2>Conclusion</h2><p>Neither CSS nor XPath is better for web scraping a priori. The better one is whichever states your selection rule with the fewest assumptions about the page. And before either argument matters, three practical considerations usually decide: What identifies the record, which result type you need, and what your specific tool supports.</p><p>And whichever language you write in, neither protects you from a brittle anchor. A selector tied to presentation classes, generated IDs, or every container between <code>&lt;html&gt;</code> and your target field will break no matter how elegant its syntax. What survives a redesign is an expression anchored to a stable record boundary and a meaningful field relationship - a rule you chose deliberately and tested against the pages you actually scrape. So the answer to the question in the title isn&#8217;t CSS or XPath: it&#8217;s whichever of them depends on as little as possible.</p><p>So, let us know in the comments: Which of the two do you prefer?</p><div><hr></div><p><em>Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/p/css-selectors-vs-xpath-scraping?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.scraping.club/p/css-selectors-vs-xpath-scraping?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Introducing the Scraping Bundle plan: €189 for over €22,000 of value]]></title><description><![CDATA[The Scraping Bundle is live: 11 partner offers, one plan. And more are coming.]]></description><link>https://www.scraping.club/p/scraping-bundle-discount-codes</link><guid isPermaLink="false">https://www.scraping.club/p/scraping-bundle-discount-codes</guid><dc:creator><![CDATA[Pierluigi Vinciguerra]]></dc:creator><pubDate>Fri, 06 Nov 2026 10:29:57 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!S_F3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>If you&#8217;ve been reading these pages for a while, you know how much I care about giving you real value. With my team, we work to keep the best of this newsletter free, thanks to our sponsors who cover the bills, and to the supporters who go deep with The Lab articles.</p><p>Today I&#8217;m launching a new paid plan. More than a plan, it&#8217;s a gift to you from our partners: <strong><a href="https://bundle.scraping.club/">the TWSC Scraping Bundle</a></strong>.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!S_F3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!S_F3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png 424w, https://substackcdn.com/image/fetch/$s_!S_F3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png 848w, https://substackcdn.com/image/fetch/$s_!S_F3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png 1272w, https://substackcdn.com/image/fetch/$s_!S_F3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!S_F3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png" width="1245" height="1671" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1671,&quot;width&quot;:1245,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:265807,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223746023?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!S_F3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png 424w, https://substackcdn.com/image/fetch/$s_!S_F3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png 848w, https://substackcdn.com/image/fetch/$s_!S_F3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png 1272w, https://substackcdn.com/image/fetch/$s_!S_F3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F35965744-25a4-4976-a09e-ca58a2b4d36e_1245x1671.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>A few examples:</p><ul><li><p><strong>Massive</strong>: 3 TB of proxies, free for six months. <strong>$10,000 value</strong>.</p></li><li><p><strong>Byteful</strong>: 3 TB of residential proxies over three months. <strong>$5,850 value</strong>.</p></li><li><p><strong>Datafuel</strong>: the full suite, scraping API plus residential, mobile, and ISP proxies. <strong>&#8364;2,000 value</strong>.</p></li><li><p><strong>Kameleo</strong>: six months of the Business plan for anti-detect browsing. <strong>&#8364;1,794 value</strong>.</p></li></ul><p>These are just four of eleven (more will come in the next months), and you can see every offer and its value at <strong><a href="https://bundle.scraping.club/">bundle.scraping.club</a></strong>.</p><p>Given their value, several of these offers have a limited number of seats. Each partner sets a cap, and once those are claimed, that offer closes for the year. <strong>The earlier you join, the more of the bundle is still on the table for you.</strong></p><p>&#8364;189 for the year against &#8364;22,610 of tools. I&#8217;ll let you do the math.</p><p>And so the decision is safe on your side: claim your offers, and if the bundle hasn&#8217;t been worth more than &#8364;189 to you within 30 days, write to me, and I&#8217;ll refund you in full.</p><h3>How it works</h3><ol><li><p>Subscribe to the TWSC Scraping Bundle plan on Substack.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://www.scraping.club/subscribe" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PAo6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png 424w, https://substackcdn.com/image/fetch/$s_!PAo6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png 848w, https://substackcdn.com/image/fetch/$s_!PAo6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png 1272w, https://substackcdn.com/image/fetch/$s_!PAo6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!PAo6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png" width="1113" height="720" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:720,&quot;width&quot;:1113,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:130975,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://www.scraping.club/subscribe&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223746023?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!PAo6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png 424w, https://substackcdn.com/image/fetch/$s_!PAo6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png 848w, https://substackcdn.com/image/fetch/$s_!PAo6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png 1272w, https://substackcdn.com/image/fetch/$s_!PAo6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80c0a897-bba5-48b0-b60c-1eb52dd6a2ac_1113x720.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div></li><li><p>Go to <strong><a href="https://bundle.scraping.club/">bundle.scraping.club</a></strong> and enter the same email, then the 6-digit code we send you.</p></li><li><p>Click Claim on the offers you want. Depending on the offer, the partner gets in touch within 48 hours, or you go straight to their page.</p></li></ol><p>New partners join throughout the year, and Bundle members hear about them first, usually while the first, limited batch of seats is still open. Join now, and you lock today&#8217;s price: every partner added later comes at no extra cost.</p><p>Two rules to follow:</p><ul><li><p>Most offers are reserved for <strong>new customers</strong> of each partner, and some are for companies only. To get the most out of the bundle, subscribe with your company email.</p></li><li><p><strong>Already a paid subscriber</strong>? Write me, and I&#8217;ll move you to the bundle with your company email, without losing the days you&#8217;ve already paid for.</p></li></ul><p></p><p>Claim the limited offers while the seats last.</p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://bundle.scraping.club/&quot;,&quot;text&quot;:&quot;Get the bundle&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://bundle.scraping.club/"><span>Get the bundle</span></a></p><p></p><p>P.S. Questions about an offer or the plan? Just write me at <strong>pier@thewebscraping.club</strong></p>]]></content:encoded></item><item><title><![CDATA[Crawl of Fame - November 2026]]></title><description><![CDATA[In between events, deobfuscation and new offers.]]></description><link>https://www.scraping.club/p/crawl-of-fame-november-2026</link><guid isPermaLink="false">https://www.scraping.club/p/crawl-of-fame-november-2026</guid><dc:creator><![CDATA[Pierluigi Vinciguerra]]></dc:creator><pubDate>Wed, 04 Nov 2026 10:43:07 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/991fd201-31cc-48ce-8d26-bdd3a9e5fd1d_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Welcome back to the Crawl of Fame; November is full of scraping events for me, and interesting readings on something I&#8217;m working on too.</p><div><hr></div><div class="callout-block" data-callout="true"><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/B5JPeJ">Decodo&#8217;s Web Data API</a></strong> turns any site into clean, structured data your models can actually use.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/B5JPeJ" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png" width="716" height="286.4" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:280,&quot;width&quot;:700,&quot;resizeWidth&quot;:716,&quot;bytes&quot;:105634,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;text&quot;:&quot;Activate your free plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/B5JPeJ"><span>Activate your free plan</span></a></p></div><div><hr></div><h3>Milan, 5th of November - Spritz and Data</h3><p>Together with Geonode, we&#8217;re organizing a Networking Aperitivo in Milan, Italy.<br>There will be a short talk (around 10 minutes), and then we&#8217;ll network and have drinks together. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://luma.com/5c4v7zjk" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!w9OA!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaea466c-7701-45c7-844d-0af2bd545070_1122x1402.png 424w, https://substackcdn.com/image/fetch/$s_!w9OA!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaea466c-7701-45c7-844d-0af2bd545070_1122x1402.png 848w, https://substackcdn.com/image/fetch/$s_!w9OA!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaea466c-7701-45c7-844d-0af2bd545070_1122x1402.png 1272w, https://substackcdn.com/image/fetch/$s_!w9OA!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaea466c-7701-45c7-844d-0af2bd545070_1122x1402.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!w9OA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaea466c-7701-45c7-844d-0af2bd545070_1122x1402.png" width="1122" height="1402" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/caea466c-7701-45c7-844d-0af2bd545070_1122x1402.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1402,&quot;width&quot;:1122,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:1471113,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://luma.com/5c4v7zjk&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223380777?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaea466c-7701-45c7-844d-0af2bd545070_1122x1402.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!w9OA!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaea466c-7701-45c7-844d-0af2bd545070_1122x1402.png 424w, https://substackcdn.com/image/fetch/$s_!w9OA!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaea466c-7701-45c7-844d-0af2bd545070_1122x1402.png 848w, https://substackcdn.com/image/fetch/$s_!w9OA!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaea466c-7701-45c7-844d-0af2bd545070_1122x1402.png 1272w, https://substackcdn.com/image/fetch/$s_!w9OA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaea466c-7701-45c7-844d-0af2bd545070_1122x1402.png 1456w" sizes="100vw"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>If you&#8217;re around and want to join, <a href="https://luma.com/5c4v7zjk">here&#8217;s the registration link</a>.</p><h3>Dublin, 10th and 11th of November - Extract Summit</h3><p>The next week I&#8217;ll be <a href="https://www.extractsummit.io/why">at the Extract Summit by Zyte</a>, where I&#8217;ll share my two cents on the stage at 10:30 on day 2, with my presentation &#8220;Antibots, Explained by a Machine. How LLMs changed the cat-and-mouse game&#8221;.</p><p><a href="https://www.extractsummit.io/agenda">Here&#8217;s the full agenda</a>:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!S90d!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!S90d!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png 424w, https://substackcdn.com/image/fetch/$s_!S90d!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png 848w, https://substackcdn.com/image/fetch/$s_!S90d!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png 1272w, https://substackcdn.com/image/fetch/$s_!S90d!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!S90d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png" width="1456" height="1576" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1576,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:2341127,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/223380777?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!S90d!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png 424w, https://substackcdn.com/image/fetch/$s_!S90d!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png 848w, https://substackcdn.com/image/fetch/$s_!S90d!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png 1272w, https://substackcdn.com/image/fetch/$s_!S90d!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F506e2922-5884-42d8-9ca0-d3b4aa651f1f_1482x1604.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>After these two events, I&#8217;ll pause for a while to focus on new things I&#8217;m working on with The Web Scraping Club. </p><h2>Scraping Bundle: an 189 USD subscription for a 15k USD advantage</h2><p>One of the things I&#8217;ll release very soon, probably this Friday, is the new&nbsp;<strong>Scraping Bundle Subscription:&nbsp;</strong>a newsletter subscription that lets you claim 15K USD in benefits with selected partners (proxy companies, scraping APIs, CAPTCHA solvers, and anti-detect browsers). </p><p>The idea came from <a href="https://www.lennysproductpass.com/">the Lenny&#8217;s product pass</a> from <span class="mention-wrap" data-attrs="{&quot;name&quot;:&quot;Lenny Rachitsky&quot;,&quot;id&quot;:1849774,&quot;type&quot;:&quot;user&quot;,&quot;url&quot;:null,&quot;photo_url&quot;:&quot;https://bucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com/public/images/afba5161-65bb-4d99-8d6b-cce660917fa1_1540x1540.png&quot;,&quot;uuid&quot;:&quot;e3a5f3f0-b1d7-4d96-b03a-ac8addfeac2f&quot;}" data-component-name="MentionToDOM"></span>, but of course focused on the scraping industry.</p><p>I&#8217;ll announce its release with a dedicated issue of the newsletter.</p><h2>Why Is This Fingerprinting Script Creating Bioweapons?</h2><p>This is the title of the latest post <a href="https://nullpt.rs/fingerprinting-bioweapons">from Veritas on his blog</a>, who discovered something interesting. </p><p>Since more and more people are using LLMs to reverse-engineer fingerprinting scripts,&nbsp;<a href="https://www.scraping.club/p/tiktok-scraping-metadata">just as I did in the latest Lab</a>, their creators are trying creative ways to avoid it. <br>In the post, Veritas found out that in some cases, some keywords about creating Bioweapons like <code>computeRicinLethalDosageForWaterSupply</code> . Those are names of functions that are thought to trigger LLMs&#8217; guardrails.</p><p>A couple of other links worth watching from the same article:</p><ul><li><p><a href="https://fingerprint.garden/#canvas">Fingerprint Garden</a> is a catalog of canvas drawing and fields maintained by Veritas</p></li><li><p><a href="https://tranco-list.eu/">Tranco Top Sites List</a> is a list of the most hardened websites against manipulation, curated by a group of researchers that used a scientific approach for populating it.</p></li></ul><h2>Google joins OpenAI and Anthropic in promising new scraping controls</h2><p>A new robots.txt directive has been proposed to the IETF to prevent LLMs from using websites as training data, by writing<br></p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;plaintext&quot;,&quot;nodeId&quot;:&quot;1f8ea7cd-800f-4646-a9fe-db138b5afa38&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-plaintext">User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/</code></pre></div><p>The proposal, written by Anthropic technical lead for standards Kevin Kelley, is now also backed by OpenAI and Google. <br>This won&#8217;t prevent them from crawling, but it&#8217;s dedicated to training. But how can we measure if this instruction was effective? For crawling, we use Google Analytics, but how can we be sure our website hasn't been scraped and used to train LLMs?<br>Who knows? It seems to me to be useless, created solely to say they&#8217;re compliant with web standards. </p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><p>For this month, that&#8217;s it; see you in December with the next edition of the Crawl of Fame.</p>]]></content:encoded></item><item><title><![CDATA[How Humanization Works in Anti-Detect Browser Automation Tools]]></title><description><![CDATA[Understanding behavioral signals and humanization practices in browser automation]]></description><link>https://www.scraping.club/p/how-humanization-works-web-scraping</link><guid isPermaLink="false">https://www.scraping.club/p/how-humanization-works-web-scraping</guid><dc:creator><![CDATA[Antonello Zanini]]></dc:creator><pubDate>Sun, 01 Nov 2026 22:15:44 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/cfeec646-b9b9-49c0-98d1-9469e5e861c3_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Most anti-detect and stealth browser solutions focus on providing strong browser fingerprints and preventing leaks. In many cases, that&#8217;s enough to avoid immediate blocks based on invisible JavaScript challenges running on the page.</p><p>However, once you start actually interacting with a website through browser automation, you&#8217;re giving anti-bot systems many more signals to analyze. I&#8217;m talking about how you move the mouse, where you click, how often you interact with elements, how you scroll through a page, which links you follow, how quickly you navigate, and more.</p><p>These behavioral cues can also play a role in whether your automation gets blocked. That&#8217;s why it&#8217;s fundamental to consider browser automation humanization techniques that make your automation behave more like a regular human user.</p><p>Here, I&#8217;ll walk you through the main approaches for making automated user interactions more realistic and human-like. I&#8217;ll explain how these methods work, how to implement them, and where AI fits into the picture.</p><h2>What Humanization Is and Why It&#8217;s Required</h2><p>Bypassing anti-bot systems through browser automation starts with spoofing browser and <a href="https://www.scraping.club/p/what-is-device-fingerprint">device fingerprints</a>, addressing CDP detection, and eliminating other potential leaks.</p><p>Still, advanced anti-bot systems don&#8217;t just look at technical fingerprints. They can also analyze how a user behaves while interacting with a page or navigating through a website by collecting a range of behavioral signals to determine whether a visitor is likely human or automated. This is what <a href="https://www.ibm.com/think/topics/user-behavior-analytics">user behavior analysis</a> is all about.</p><p>Mouse movement, typing patterns, scrolling, clicking, navigation, and timing can all become signals. If these interactions look suspiciously robotic, anti-bot systems may respond with a direct block or, more commonly, present a challenge such as a CAPTCHA. Now, here is where browser automation humanization comes in.</p><p>Humanization is the practice of making automated interactions behave more like those of a real human user. In practice, that includes more natural mouse movements, variable typing speed, realistic scrolling, pauses between actions, and less predictable interaction patterns.</p><p>Humanization is usually available as a configurable feature in some <a href="https://www.scraping.club/p/browser-automation-landscape-2025">open-source anti-detect browsers</a>, or built directly into premium stealth cloud browser solutions.</p><div><hr></div><div class="callout-block" data-callout="true"><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/B5JPeJ">Decodo&#8217;s Web Data API</a></strong> turns any site into clean, structured data your models can actually use.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/B5JPeJ" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png" width="716" height="286.4" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:280,&quot;width&quot;:700,&quot;resizeWidth&quot;:716,&quot;bytes&quot;:105634,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;text&quot;:&quot;Activate your free plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/B5JPeJ"><span>Activate your free plan</span></a></p></div><blockquote><div><hr></div></blockquote><h2>Browser Automation Humanization Techniques: Summary Table</h2><p>If you want a quick overview of what humanization practices look like in browser automation, take a look at the summary table below:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!lqXN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!lqXN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png 424w, https://substackcdn.com/image/fetch/$s_!lqXN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png 848w, https://substackcdn.com/image/fetch/$s_!lqXN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png 1272w, https://substackcdn.com/image/fetch/$s_!lqXN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!lqXN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png" width="1456" height="883" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:883,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Browser automation humanization techniques&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Browser automation humanization techniques" title="Browser automation humanization techniques" srcset="https://substackcdn.com/image/fetch/$s_!lqXN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png 424w, https://substackcdn.com/image/fetch/$s_!lqXN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png 848w, https://substackcdn.com/image/fetch/$s_!lqXN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png 1272w, https://substackcdn.com/image/fetch/$s_!lqXN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4f3ca7ff-7391-4ebd-a34d-d68b28d91eab_1491x904.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Browser automation humanization techniques</figcaption></figure></div><div><hr></div><div class="callout-block" data-callout="true"><p><em><strong><a href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=banner_nov">DataImpulse</a></strong> is trusted by teams running ad verification, web scraping, SERP tracking, and market research. Ethically sourced proxies, globally accessible, and fairly priced.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=banner_nov" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!fIaJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!fIaJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!fIaJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!fIaJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!fIaJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png" width="1024" height="576" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:576,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:562074,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://dataimpulse.com/?utm_source=website&amp;utm_medium=article&amp;utm_campaign=banner_nov&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/219098748?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!fIaJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!fIaJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!fIaJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!fIaJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc881c55-93fa-4bbd-a38c-078b38fb15a9_1024x576.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://dataimpulse.com/&quot;,&quot;text&quot;:&quot;Get Started With DataImpulse&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://dataimpulse.com/"><span>Get Started With DataImpulse</span></a></p></div><div><hr></div><h2>Mouse Movement Humanization Techniques</h2><p>Mouse movement humanization aims to make automated cursor trajectories less rigid and more representative of natural pointer behavior. Instead of moving directly from point A to B, the idea is to model curvature, acceleration, variability, and timing to produce more natural-looking interactions.</p><h3>Implementation Approaches</h3><p>There are several ways to generate human-like mouse paths, ranging from mathematical curves to probabilistic models and machine learning. The most popular are:</p><ul><li><p><strong><a href="https://arxiv.org/abs/1406.2661">GANs (Generative Adversarial Networks)</a></strong>: Learn movement patterns from recorded human cursor data and generate new trajectories that reproduce characteristics of the training data.</p></li><li><p><strong><a href="https://pomax.github.io/bezierinfo/">B&#233;zier curves</a></strong>: Generate smooth paths between points using control points. They&#8217;re relatively simple to implement and are widely used in browser automation projects.</p></li><li><p><strong><a href="https://ben.land/post/2021/04/25/windmouse-human-mouse-movement/">WindMouse</a></strong>: Model cursor movement using a combination of attraction toward the target and stochastic forces, producing curved trajectories with changing velocity.</p></li><li><p><strong><a href="https://www.cs.cmu.edu/~fp/courses/graphics/asst5/catmullRom.pdf">Catmull-Rom splines</a></strong>: Interpolate between a sequence of points, producing smooth paths that pass through the specified control points to avoid simply connecting them with straight segments.</p></li></ul><p>Now, the path itself is only part of the equation. Velocity and timing also matter when modeling pointer behavior. For this reason, you may also need:</p><ul><li><p><strong><a href="https://www.nngroup.com/articles/fitts-law/">Fitts&#8217;s law</a></strong>: States that the time required to rapidly move to a target area is a function of the distance to the target and the size of the target.</p></li><li><p><strong><a href="https://blog.hirnschall.net/perlin-noise/">Perlin / Gaussian noise</a></strong>: Noise can be introduced into an otherwise smooth trajectory to add small variations and imperfections.</p></li><li><p><strong>Pre- and post-action pauses</strong>: Small pauses before and after interactions can make sequences less mechanically uniform. Avoid one fixed delay everywhere.</p></li></ul><h3>Interesting Projects for Further Reading</h3><p>If you want to explore the topic further, these projects provide useful implementations and references:</p><ul><li><p><em><a href="https://github.com/Xetera/ghost-cursor">Ghost Cursor</a></em>: JavaScript library for generating human-like cursor paths and mouse interactions in Puppeteer or on any 2D plane.</p></li><li><p><em><a href="https://github.com/oxylabs/OxyMouse">OxyMouse</a></em>: Python library implementing B&#233;zier, Gaussian, Perlin, and custom mouse-movement algorithms.</p></li><li><p><em><a href="https://github.com/riflosnake/HumanCursor">HumanCursor</a></em>: Python package for generating cursor movements with variable speed, acceleration, and curvature.</p></li><li><p><em><a href="https://github.com/AsfhtgkDavid/windmouse">WindMouse</a></em>: Python implementation of the WindMouse algorithm with configurable movement dynamics.</p></li><li><p><em><a href="https://github.com/sarperavci/human_mouse">Human Mouse</a></em>: Python project for human mouse movements via B&#233;zier curves and spline interpolation.</p></li><li><p><em><a href="https://github.com/AB6162/shy-mouse-playwright">Shy Mouse Playwright</a></em>: Playwright/Patchright library for achieving more natural mouse movements and interactions, inspired by HumanCursor.</p></li><li><p><em><a href="https://github.com/CloverLabsAI/human-cursor">Human Cursor</a></em>: Browser automation library focused on generating more human-like cursor movements in Playwright and Patchright.</p></li></ul><h2>Typing Humanization Strategies</h2><p>Keystroke pattern emulation focuses on replicating more authentic typing behavior in automated systems. Rather than instantly inserting text into input fields, it models factors such as inter-key latency, variations in typing speed, correction patterns, and input accuracy to generate typing sequences that more closely resemble those of real users.</p><h3>Implementation</h3><p>Reproducing natural keystroke patterns involves multiple complementary strategies, each targeting different aspects of human typing behavior.</p><p>The most effective implementations combine several of these methods:</p><ul><li><p><strong><a href="https://bookdown.org/danielcavey27/lecture_notes/stochastic-process-and-markov-chains.html">Markov chains and stochastic processes</a></strong>: Model typing as state transitions where each keystroke probability depends on previous character sequences. This technique captures bigram dependencies and word-length effects in a statistically grounded framework.</p></li><li><p><strong><a href="https://www.nngroup.com/articles/response-times-3-important-limits/">WPM (Words-Per-Minute) variance</a></strong>: Implement dynamic timing based on word complexity. Common words accelerate naturally while technical terms slow down, reflecting genuine cognitive load variations.</p></li><li><p><strong><a href="https://github.com/ArslanJajja1/bigram-language-model">Bigram-based acceleration</a></strong>: Frequent letter pairs (like &#8220;th&#8221;, &#8220;er&#8221;, &#8220;in&#8221;) are typed in rapid succession since fingers don&#8217;t need to reposition. Modeling these transitions realistically prevents the mechanical uniformity of constant delays.</p></li><li><p><strong><a href="https://pubmed.ncbi.nlm.nih.gov/33617541/">Fatigue simulation</a></strong>: Progressive typing slowdown accumulates naturally over extended text (e.g., decrease speed by 0.05-0.1% per character). This gradual degradation mimics concentration loss that real users experience during repetitive input.</p></li><li><p><strong><a href="https://www.wikiwand.com/en/Edit_distance">Keyboard distance metrics</a></strong>: Calculate spatial relationships between keys to predict latency. Adjacent keys require minimal repositioning time, while distant keys or shifted characters introduce measurable delays.</p></li></ul><p>Beyond timing mechanics, behavioral authenticity for typing actions requires modeling deliberate imperfection:</p><ul><li><p><strong><a href="https://arxiv.org/html/2606.28090v1">Error injection and correction cycles</a></strong>: Introduce realistic typos at 2-5% frequency, with neighbor-key errors being most common.</p></li><li><p><strong>Pause injection points</strong>: Natural conversations with keyboards include micro-pauses before complex words (averaging 200-300ms) and extended hesitations when encountering specialized terminology.</p></li></ul><h3>Interesting Projects for Further Reading</h3><p>Several open-source implementations provide battle-tested approaches to keystroke humanization:</p><ul><li><p><em><a href="https://github.com/arpan404/bumblebee">Bumblebee</a></em>: A Python library supporting mouse and keyboard automation with configurable behavior profiles (&#8221;natural&#8221;, &#8220;careful&#8221;, etc.). Includes integrated WPM control, typo injection, and optional reinforcement-learning-generated cursor trajectories for coordinated input sequences.</p></li><li><p><em><a href="https://github.com/Lax3n/HumanTyping">HumanTyping</a></em>: Advanced typing simulator built on Markov chain theory, modeling variable speed based on word frequency, automatic bigram acceleration, fatigue progression, neighbor-key errors, character swaps, and delayed error detection.</p></li><li><p><em><a href="https://github.com/MaxHalford/clavier">Clavier</a></em>: A Python project for analyzing typing patterns based on physical keyboard layouts. It can calculate distances between keys, words, and consecutive keystrokes.</p></li><li><p><em><a href="https://github.com/UnMars/human_typer">Human Typer</a></em>: A Python package for simulating human keyboard typing with configurable typing speed and error rates.</p></li></ul><h3>Scroll Humanization Practices</h3><p>Scroll humanization makes automated scrolling resemble typical user interaction. Instead of relying on constant speeds or fixed pixel increments, it accounts for variations in scroll velocity, momentum, scroll distance, and pauses between movements.</p><h3>Implementation</h3><p>Some of the most common ways to model more realistic scrolling behavior are:</p><ul><li><p><strong><a href="https://easings.net/">Easing functions (cubic, quadratic, elastic, etc.)</a></strong>: Employ standardized easing curves to model acceleration and deceleration during scrolling to make it more organic.</p></li><li><p><strong><a href="https://engineering.skroutz.gr/blog/handling-inertial-scroll-in-combination-with-scroll-snapping/">Inertial scrolling simulation</a></strong>: Model physical momentum where scroll velocity gradually decreases over time, mimicking the behavior of touch-screen scrolling or trackpad momentum.</p></li><li><p><strong><a href="https://publish.obsidian.md/twsc-public/Web+Scraping/Articles/bezier-curves-web-scraping">Bezier curves</a></strong>: Generate smooth scroll paths using control points, similar to mouse movement curves, to avoid linear or step-based scrolling patterns.</p></li></ul><p>When modeling realistic scrolling behavior, there are several other factors worth considering:</p><ul><li><p><strong>Content-aware pausing</strong>: Introduce pauses at logical content boundaries, such as headings, images, and section breaks, where users are likely to stop to read or process information.</p></li><li><p><strong>Scroll distance variation</strong>: Real users don&#8217;t typically scroll the same distance on every action. Varying scroll distances helps avoid overly uniform movement patterns.</p></li><li><p><strong>Inter-scroll delays</strong>: Add variable pauses between scroll events to account for the time users spend reading, scanning, or deciding where to scroll next.</p></li></ul><h3>Interesting Projects for Further Reading</h3><p>For more information and practical implementations, take a look at the project below:</p><ul><li><p><em><a href="https://github.com/Alverrt/ghost-scroll">Ghost Scroll</a>:</em> A Puppeteer script that makes automated scrolling behave more like human scrolling by combining smooth movements with randomized scroll patterns.</p></li></ul><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><h3>Other General Humanization Considerations</h3><p>Humanization of automation doesn&#8217;t stop at mouse movement, typing, and scrolling. Beyond these core techniques, several higher-level humanization patterns can help create a more consistent and natural automation experience:</p><ul><li><p><strong>Navigation patterns</strong>: Users don&#8217;t navigate websites linearly. As humans, we may revisit pages, use the back button, open links in new tabs, or change direction based on what we encounter. So, you should simulate that behavior.</p></li><li><p><strong>Focus and cursor behavior</strong>: Real users frequently move their cursor between elements, change focus, and briefly hover over interactive components. Performing a task too mechanically can raise suspicion.</p></li><li><p><strong>Timing and pacing</strong>: Human interactions happen at varying speeds. Introducing realistic pauses between actions, along with variation in task duration, can make automation less uniform.</p></li><li><p><strong>Viewport behavior</strong>: Users generally need to see an element before interacting with it. Make sure that elements you&#8217;re interacting with are visible within the current viewport.</p></li></ul><p>Keep in mind that these behaviors need to work together. If your bot moves the mouse like a human but types like a machine or navigates through pages at an unrealistic speed, the overall behavior may still appear automated.</p><p>Similarly, if interactions are <em>too</em> humanized (e.g., taking excessively long to move the mouse or type), they might become unusual in their own way. Ultimately, you want to find a balanced combination of behaviors that produces a consistent and realistic overall interaction pattern.</p><h2>Can AI Help with Browser Automation Humanization?</h2><p>Yes! As you might expect, AI and machine learning can play a pivotal role in browser automation humanization. Recent research explores how machine learning can be used to reproduce aspects of human browsing behavior, including navigation decisions, cursor trajectories, and more.</p><p>For example, <a href="https://arxiv.org/abs/2510.10666">BrowserAgent</a> uses supervised and rejection fine-tuning to train an agent to interact with websites through actions such as clicking, scrolling, and typing. <a href="https://aclanthology.org/2025.acl-long.697/">WebExperT</a> takes a different approach, focusing on human-like planning and &#8220;fast-and-slow&#8221; reasoning for web navigation.</p><p>AI can also be applied at a lower level. Projects such as <a href="https://github.com/sameelarif/scribe">Scribe</a> use collected mouse-movement data to train models that generate human-like cursor trajectories for browser automation.</p><p>The key distinction is that AI can learn and reproduce behavioral patterns from data, rather than relying entirely on coded humanization rules. This opens the door to more complex and variable interaction patterns across different browsing tasks, but with less deterministic control over the final output.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://substack.thewebscraping.club/p/consulting&quot;,&quot;text&quot;:&quot;Need help with your scraping project?&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://substack.thewebscraping.club/p/consulting"><span>Need help with your scraping project?</span></a></p><div><hr></div><h3>Browser Automation Libraries Supporting Humanization Features</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xdRY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xdRY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png 424w, https://substackcdn.com/image/fetch/$s_!xdRY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png 848w, https://substackcdn.com/image/fetch/$s_!xdRY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png 1272w, https://substackcdn.com/image/fetch/$s_!xdRY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xdRY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png" width="1456" height="752" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:752,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Browser automation libraries with humanization features&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Browser automation libraries with humanization features" title="Browser automation libraries with humanization features" srcset="https://substackcdn.com/image/fetch/$s_!xdRY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png 424w, https://substackcdn.com/image/fetch/$s_!xdRY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png 848w, https://substackcdn.com/image/fetch/$s_!xdRY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png 1272w, https://substackcdn.com/image/fetch/$s_!xdRY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F21e31603-26c2-4e1b-9fc6-8e317eccb79a_2305x1191.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Browser automation libraries with humanization features</figcaption></figure></div><p>For further reading, check out:</p><ul><li><p><em><a href="https://www.scraping.club/p/pydoll-webdriver-scraping">Pydoll: WebDriver-Free Browser Automation in Python</a></em></p></li><li><p><em><a href="https://www.scraping.club/p/invisible-playwright-beat-bot-detection-scraping">Can invisible_playwright Beat Bot Detection? A Hands-On Test and Review</a></em></p></li><li><p><em><a href="https://www.scraping.club/p/is-camoufox-still-effective-scraping">THE LAB #106: Is Camoufox still effective, and do the forks help?</a></em></p></li></ul><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>Conclusion</h2><p>Fingerprint spoofing alone isn&#8217;t enough to stay under the radar of anti-bot systems. For proper browser automation, you also need to make the user interactions look more human-like. After all, moving the cursor directly from point A to B or scrolling by the exact same number of pixels at fixed intervals makes your automation look obviously robotic.</p><p>Humanization strategies help you make automated interactions more natural, from generating more realistic mouse trajectories and varying scrolling behavior to modeling keystroke timing and introducing natural-looking typos and correction patterns.</p><p>You now know what browser automation humanization is all about, how to implement these techniques, and which libraries already provide built-in support for them. Thanks for reading, and see you in the next one!</p><div><hr></div><p><em>Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/p/how-humanization-works-web-scraping?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.scraping.club/p/how-humanization-works-web-scraping?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[The LAB #118: Scrape metadata from TikTok videos]]></title><description><![CDATA[How Hermes Agent and GLM 5.3 reverse-engineered the TikTok app to scrape video metadata]]></description><link>https://www.scraping.club/p/tiktok-scraping-metadata</link><guid isPermaLink="false">https://www.scraping.club/p/tiktok-scraping-metadata</guid><dc:creator><![CDATA[Pierluigi Vinciguerra]]></dc:creator><pubDate>Thu, 29 Oct 2026 11:51:19 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/ccfc2c0a-ad98-42ab-8e52-05ee1684558a_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A few weeks ago, a TikTok dataset published on Hugging Face caught my eye: it contained metadata of 5.6B videos from the platform. That&#8217;s a huge dataset, and the data collection immediately appeared to me to be the most interesting part. Using a browser or an emulator to do so would have been too expensive.  In fact, the write-up from DataSocial explained the general shape of the approach: it was completely headless, with no emulators in the loop. <br>I was able to read it before it disappeared (<a href="https://web.archive.org/web/20261011072759/https://datasocial.ai/writing/scraping-tiktoks-mobile-api?ref=hf#intro">thanks Web Archive for the snapshot</a>) and was intrigued by replicating it, but it didn&#8217;t reveal the ingredients of the secret sauce. They were put behind a paywall costing 1699 USD. Which, I mean, is totally fine, but then I asked myself: &#8220;I have this cool reverse engineering setup, made by Hermes Agent, <a href="https://github.com/morluto/rea">Reverse Engineer Anything repo</a> and GLM 5.3, maybe I can find the answers by myself&#8221;. </p><p>So I did. Or better, we did. Well, most of the work was done by GLM; I just set the direction and checked the results.</p><div><hr></div><div class="callout-block" data-callout="true"><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/B5JPeJ">Decodo&#8217;s Web Data API</a></strong> turns any site into clean, structured data your models can actually use.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/B5JPeJ" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png" width="716" height="286.4" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:280,&quot;width&quot;:700,&quot;resizeWidth&quot;:716,&quot;bytes&quot;:105634,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;text&quot;:&quot;Activate your free plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/B5JPeJ"><span>Activate your free plan</span></a></p></div><div><hr></div><h2>The workstation: a model, an agent, and a tool</h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mt3R!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mt3R!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png 424w, https://substackcdn.com/image/fetch/$s_!mt3R!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png 848w, https://substackcdn.com/image/fetch/$s_!mt3R!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png 1272w, https://substackcdn.com/image/fetch/$s_!mt3R!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!mt3R!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png" width="1456" height="1029" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1029,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:138862,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/222481243?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!mt3R!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png 424w, https://substackcdn.com/image/fetch/$s_!mt3R!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png 848w, https://substackcdn.com/image/fetch/$s_!mt3R!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png 1272w, https://substackcdn.com/image/fetch/$s_!mt3R!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f741847-70ad-4af3-a904-634a9fac2a15_2080x1470.png 1456w" sizes="100vw"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>I have limited experience in reverse engineering, so I needed to delegate most of the work to LLMs while making sure I wasn&#8217;t fooled by them. Western models (ChatGPT, Claude) have very strict guardrails and block any request that resembles a cybersecurity one. For this reason, even in past articles, we used GLM or Manus for our investigation on anti-bots. This time I&#8217;m using GLM 5.3, hosted on OpenRouter, called by Hermes agent. <br>Hermes gives all the remaining tools we need to complete the job and I&#8217;ve integrated them with the Reverse Engineer Anything repo, which turns a general agent into a reverse engineer.<br>It is an open source project that runs as an MCP server, the standard way an agent plugs into external tools. You install it with <code>npx rea-agents setup</code>, pick your agent, approve the changes it proposes, and restart. REA drives the programs this field has always used: Ghidra, Hopper or IDA for native binaries, jadx for Android, adb for talking to a device, TShark for captured packets. Instead of choosing the tool yourself and gluing together the results, the agent asks REA for the pseudocode of a function or the strings in a library; REA runs the right program and returns a clean answer, along with its evidence and limitations, which also helps non-specialists like me. On top of REA and its tools, I also installed two classics like Frida for watching a running app and mitmproxy for reading its traffic. </p><div><hr></div><div class="callout-block" data-callout="true"><p><em>Tired of getting blocked while scraping the web?<br><strong><a href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club">ScrapingBee</a></strong> handles proxies, browsers, anti-bot systems, and retries so you can focus on your data. Get clean Markdown, JSON, or HTML from the web with up to a 99.9% success rate. ScrapingBee is SOC 2 Type II and GDPR compliant, and trusted by 4,000+ developers.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 424w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 848w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png" width="1456" height="437" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:437,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:128210,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/216891176?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 424w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 848w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club&quot;,&quot;text&quot;:&quot;Discover ScrapingBee&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club"><span>Discover ScrapingBee</span></a></p></div><div><hr></div><h2>Why TikTok is different</h2><p>Most of the time, when I find an internal API behind an app, the hard part is already over. You intercept the traffic, you read the request, you replay it from Python, and you are done. The app was just a nicer way to call an endpoint that happily answers anyone who asks. I have written about this before, from <a href="https://substack.thewebscraping.club/p/the-lab-12-reverse-engineering-mobile">reverse-engineering a mobile API</a> to <a href="https://www.scraping.club/p/how-to-scrape-data-from-mobile-apps">pulling data from apps with HTTP Toolkit</a> and <a href="https://substack.thewebscraping.club/p/jwt-tokens-and-api-scraping">chasing JWT tokens</a>.</p><p>TikTok&#8217;s mobile API doesn&#8217;t work like that, but it&#8217;s way more strict. Every request to its private API carries a signature, a short blob the app computes and attaches to prove the request came from a real, untampered copy of the app. The problem is that  if the signature is wrong, the server does not argue but it answers with a normal HTTP 200 and an empty body. No error, no code, nothing to say what you broke. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!X0Ds!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!X0Ds!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png 424w, https://substackcdn.com/image/fetch/$s_!X0Ds!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png 848w, https://substackcdn.com/image/fetch/$s_!X0Ds!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png 1272w, https://substackcdn.com/image/fetch/$s_!X0Ds!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!X0Ds!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png" width="1456" height="623" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4313e078-2816-40d0-a81a-f76058789af5_2080x890.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:623,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:104446,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/222481243?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!X0Ds!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png 424w, https://substackcdn.com/image/fetch/$s_!X0Ds!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png 848w, https://substackcdn.com/image/fetch/$s_!X0Ds!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png 1272w, https://substackcdn.com/image/fetch/$s_!X0Ds!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4313e078-2816-40d0-a81a-f76058789af5_2080x890.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The app is really checking four things at once, and all four have to be right on the same request:</p><ol><li><p>A device identity that TikTok issued, so the request looks like it comes from a device it already knows.</p></li><li><p>The request signature, computed by a native library embedded inside the app.</p></li><li><p>The right regional endpoint for what you are asking, because the API is split by region and route.</p></li><li><p>A network fingerprint that looks like a phone and not a script.</p></li></ol><p>Miss any one, and you get the same empty 200. There is no partial success and no hint about which of the four failed. The consequence is that you cannot trust the status line at all, but the only honest signal is the body: a short or empty body means a gate is wrong, and a real payload means all four passed. </p><p>So the first useful thing to build, before touching any of the gates, is a small judge that determines success based on the content of the response rather than its status code. In the real work, it is a few lines, and this is the one piece of code worth showing because everything else depends on it:</p><pre><code><code>def classify(http_status: int, body: bytes) -&gt; Outcome:
    if http_status != 200:
        return Outcome(False, f"upstream HTTP {http_status}",
                       retry_would_help=(http_status == 429))
    if len(body) &lt; MIN_BODY_BYTES:
        return Outcome(False, f"empty upstream body ({len(body)} bytes)",
                       retry_would_help=True)
    ...</code></code></pre><h2>Starting with the heavy lifting</h2><p>The agent started reading the app without running it: an Android app hides its logic as bytecode that decompiles back to something close to Java, bundled with compiled native libraries that do not decompile to anything readable. The first part names things while the native part hides them. So what Hermes did was to use REA, and in particular Jadx, over the bytecode, so that we could create a map: the name of the signing library, the headers the signature carries, the app ID that turns up everywhere. At least now we have something to have a look at. </p><p>Then it ran the app, on a rooted emulator, with Frida attached so it could watch the signing code from the inside. Frida works by hooking functions in a running app and reading what goes in and out of them, so the natural place to attach is the boundary where the app&#8217;s Java code calls down into its native libraries. If you put a listener there, you can see every call that crosses from the managed side into native code.</p><p>The first real finding was that the signing we wanted to replicate, never crossed this border. <br>Let me explain that better: an Android app runs on two levels, the Java side on top and the compiled native libraries underneath, and the signer has a piece on each. The Java-facing piece, the one that sets the device id and starts the SDK, is called from Java, so the hook sees it. But the signature on an actual data request is made on the other side, by the app&#8217;s own networking stack calling the signing library directly, all inside the native code. For this reason, the hook sitting in the middle never sees it. The work we needed was happening under the floor we were standing on, which is worth knowing early because it kills the obvious plan of hooking the signer and reading its output.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xF9P!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xF9P!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png 424w, https://substackcdn.com/image/fetch/$s_!xF9P!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png 848w, https://substackcdn.com/image/fetch/$s_!xF9P!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png 1272w, https://substackcdn.com/image/fetch/$s_!xF9P!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xF9P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png" width="1456" height="788" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:788,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:150512,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/222481243?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xF9P!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png 424w, https://substackcdn.com/image/fetch/$s_!xF9P!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png 848w, https://substackcdn.com/image/fetch/$s_!xF9P!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png 1272w, https://substackcdn.com/image/fetch/$s_!xF9P!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F780210a3-2a6a-444d-a417-0f5363606b18_2080x1126.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Another problem we faced was the following: to read an app&#8217;s encrypted traffic, you put a proxy in the middle and make the app trust its certificate. TikTok&#8217;s API refuses: it pins its own certificate and rejects anything else before a single request goes out, so the proxy sees nothing. </p><p>The agent&#8217;s first tried the usual textbook move: replace the app&#8217;s certificate check with one that always says yes. The app crashed instantly, so the agent spent an afternoon treating it as a tamper-detection issue. But it was not: a pass through the decompiler showed that the real fix was not to replace the check but to let it run normally and flip only its final verdict from reject to accept. The check does its bookkeeping, the stack stays happy, and the pinning is bypassed anyway. With that, the traffic opened up, and the agent captured a batch of real signed requests.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5lhy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5lhy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png 424w, https://substackcdn.com/image/fetch/$s_!5lhy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png 848w, https://substackcdn.com/image/fetch/$s_!5lhy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png 1272w, https://substackcdn.com/image/fetch/$s_!5lhy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5lhy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png" width="1456" height="791" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:791,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:174974,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/222481243?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!5lhy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png 424w, https://substackcdn.com/image/fetch/$s_!5lhy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png 848w, https://substackcdn.com/image/fetch/$s_!5lhy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png 1272w, https://substackcdn.com/image/fetch/$s_!5lhy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab0e894d-5902-4b11-8171-00e35b694dea_2080x1130.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>After reviewing the traffic, we captured some signatures and wanted to understand whether they could be reused. Of course not.<br>The signature is computed over the exact bytes of the request, so changing a single byte, say to ask for the next page, gets the request rejected. You cannot sign once and paginate. Every request you ever send needs its own fresh signature. </p><p>So we had two options: reimplement the app&#8217;s signing math yourself, which is the exact piece the paid write-up sells and which depends on keys that are not public, or take the app&#8217;s own signing library and run it outside the app. </p><p>We choose the second, using a tool called unidbg, which can load that one native library on its own and run its functions on a plain server, with no phone involved, which produces signatures identical to the app&#8217;s for the simple reason that they come from the app&#8217;s own code. With that running, a request assembled in a script and signed by the borrowed library comes back full of data. Finally, the app is out of the loop.</p><p>The last piece was identity. A signature has to be made for a specific device that TikTok knows about, and the one captured from the emulator eventually ages out. So the agent learned to register fresh identities the way the app does on its first launch, warm them up, and prove each one works before trusting it. </p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://github.com/TheWebScrapingClub/scraping-wiki&quot;,&quot;text&quot;:&quot;Find more in the wiki&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://github.com/TheWebScrapingClub/scraping-wiki"><span>Find more in the wiki</span></a></p><div><hr></div><p>From there it is ordinary engineering: a pool of working identities, the content judge from earlier reading every response, and a rotating residential IP so a blocked request retries from a clean address. The numbers from the real runs are modest, but the shape is the point. One machine, no phone, collecting the same public data the app shows, at a steady rate, and scaling by running more signers in parallel.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HUiK!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HUiK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png 424w, https://substackcdn.com/image/fetch/$s_!HUiK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png 848w, https://substackcdn.com/image/fetch/$s_!HUiK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png 1272w, https://substackcdn.com/image/fetch/$s_!HUiK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HUiK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png" width="1456" height="1250" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1250,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:213674,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/222481243?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!HUiK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png 424w, https://substackcdn.com/image/fetch/$s_!HUiK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png 848w, https://substackcdn.com/image/fetch/$s_!HUiK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png 1272w, https://substackcdn.com/image/fetch/$s_!HUiK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0824f786-d446-4c38-8796-8d8c1ad29572_2080x1786.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><div class="callout-block" data-callout="true"><p><em>&#8220;Data is the new oil.&#8221;<br>You&#8217;ve heard it a hundred times, but do you know who does the drilling?<br>With AI, it&#8217;s more true than ever. Models are trained on data scraped from the web. Agents need browser automation that doesn&#8217;t fall over at the first CAPTCHA. But who&#8217;s doing the heavy lifting? Proxies that actually work.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://luma.com/5c4v7zjk" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6re3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 424w, https://substackcdn.com/image/fetch/$s_!6re3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 848w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1272w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6re3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png" width="318" height="397.3582887700535" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1402,&quot;width&quot;:1122,&quot;resizeWidth&quot;:318,&quot;bytes&quot;:1471113,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://luma.com/5c4v7zjk&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/221688546?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!6re3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 424w, https://substackcdn.com/image/fetch/$s_!6re3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 848w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1272w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>That&#8217;s why we partnered with <strong><a href="https://www.linkedin.com/company/geonode/">Geonode</a></strong> for an aperitivo on 5 November in Milan.<br>No talks to sit through but just people who build scrapers, data pipelines and AI agents for a living, and a drink.<br><br>If you&#8217;re one of them, here&#8217;s the link to join: <strong><a href="https://www.linkedin.com/safety/go/?url=https%3A%2F%2Fluma%2Ecom%2F5c4v7zjk&amp;urlhash=TG6m&amp;mt=_MVhK0tQUX3JJQvNRPrQeq4u7WW0fNOSWRfZa99BOWYnJowsTzf1EnbBN_-HBrEpzjxD1u-cfdNwXHw28VhbwRGPBXuXqZ7OSb9qW7AhN0Eryqhpw3SR_IT3&amp;isSdui=true">https://luma.com/5c4v7zjk</a></strong></em></p></div><div><hr></div><h2>The code behind it, and the proof it ran</h2><p>&#8220;Ok, nice work, but we want the code; otherwise, we won&#8217;t believe you,&#8221; you would say.  I&#8217;m not super convinced about sharing the code; I&#8217;d like to avoid requests to take down the article or even being deindexed. </p><p>The code lives in the Lab&#8217;s repository, at <a href="https://github.com/TheWebScrapingClub/thelab">github.com/TheWebScrapingClub/thelab</a>, directory 118.TikTok. The work is a set of small Python modules, each doing one job, and they fall into two groups. Only the first group is in the repo. The code that actually touches TikTok stays out of it on purpose.</p>
      <p>
          <a href="https://www.scraping.club/p/tiktok-scraping-metadata">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[Testing ScrapeUnblocker on Google and Shopee]]></title><description><![CDATA[Let&#8217;s see how this new scraping API works against hard targets]]></description><link>https://www.scraping.club/p/testing-scrapeunblocker-web-api</link><guid isPermaLink="false">https://www.scraping.club/p/testing-scrapeunblocker-web-api</guid><dc:creator><![CDATA[Pierluigi Vinciguerra]]></dc:creator><pubDate>Tue, 27 Oct 2026 12:36:20 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/5a7aadc6-40b2-40ed-8aa9-5eb83ff89470_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>If you follow these pages or, in general, the industry news, you know for sure that Google is becoming harder to scrape,&nbsp;<a href="https://www.scraping.club/p/google-removed-the-urls-serp">as it also removed the destination URL from its search results</a>. <br>Shopee is a website that's hard to tackle:<a href="https://www.scraping.club/p/why-shopee-is-so-hard-to-scrape">&nbsp;custom bot protection</a>, the need for accounts to use the website, and strict geographic controls. <br>Shein is another hard target, and when talking with other professionals, it&#8217;s sometimes not even economically viable to extract data from there, since it requires too much effort and you need many customers to cover the extraction costs. </p><p>So, when <a href="https://scrapeunblocker.com/?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">ScrapeUnblocker</a> came to me asking for a test, and I saw the chance to extract data from these sources, I immediately wanted to test them, and this is my recap.</p><h2>What ScrapeUnblocker is</h2><p>ScrapeUnblocker is an anti-bot bypass API. You send a URL, and the service chooses the best-performing path to that site. The <a href="https://docs.scrapeunblocker.com/introduction?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">docs</a> describe it as &#8220;dedicated site plugins, real browser rendering, rotating premium proxies, automatic challenge solving&#8221;, and it claims to beat Cloudflare, DataDome, PerimeterX, Akamai and hCaptcha among other protections. You can&#8217;t choose the route yourself: you only see the result.</p><p>The public API is three endpoints on <code>https://api.scrapeunblocker.com:</code></p><ul><li><p><code>/getPageSource</code> returns the HTML of any URL, or the same page as Markdown, or structured JSON.</p></li><li><p><code>/serpApi</code> runs a Google search and returns organic results, ads and the AI Overview as JSON.</p></li><li><p><code>/getImage</code> downloads an image in its original format, for image hosts that block plain requests.</p></li></ul><p>Every call is a <code>POST</code> with the parameters in the query string and the key in the <code>x-scrapeunblocker-key</code> header.</p><p>On top of these, there are 75 plugins listed in the <a href="https://docs.scrapeunblocker.com/plugins/overview?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">plugins page</a>. A plugin is a ready-made scraper for one website: you pass a few fields (a keyword, a market, a product id) and you get parsed JSON back. The list covers e-commerce (Amazon, Walmart, Shein, AliExpress, Taobao, Shopee), travel (Booking, Airbnb, Skyscanner), real estate (Zillow, Idealista, Rightmove), jobs, reviews and a long list of Google products (Shopping, Maps, Trends, News, Hotels). The Shopee plugin is very new, the <a href="https://docs.scrapeunblocker.com/changelog?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">changelog</a> dates it to October 23, three days before my tests.</p><p>The rest of the product is built around the same API key:</p><ul><li><p>an MCP server (<code>npx -y scrapeunblocker-mcp</code>) and a Claude Code plugin, with tools like <code>fetch_markdown</code>, <code>google_search</code>, <code>search_and_read</code> and <code>run_plugin</code>;</p></li><li><p>official SDKs for Python, Node.js, PHP and Ruby (the Python one is <code>pip install scrapeunblocker</code>);</p></li><li><p>Spider Cloud, which hosts your Scrapy project from a git repository and lets you choose, run by run, how much of the crawl goes through the unblocking API;</p></li><li><p>a no-code area where you describe the data you want in plain language, and the results arrive by email as CSV or JSON.</p></li></ul><h3>The features of /getPageSource</h3><p>Even if my tests use <code>/serpApi</code> and two plugins, most of the options live on <code>/getPageSource</code>, so a quick tour is useful. With <code>markdown=true</code> you get the page as Markdown, by default only the main content. With <code>parsed_data=true</code> you get JSON extracted from Schema.org, <code>__NEXT_DATA__</code> or AI-generated rules, plus a <code>page_type</code> (product, listing, article, search results and so on). With <code>get_cookies=true</code> you get the HTML, the cookies of the target site and the country of the exit IP. And with <code>steps</code> you can drive a real browser after the page loads (click, type, scroll, wait, up to about ten actions), using the selectors that <code>list_elements=true</code> gives you.</p><p>The billing rules are simple, and in my opinion they&#8217;re the best part of the docs. One request is one credit, whatever the target needs (there&#8217;s no multiplier for JavaScript rendering). Errors on their side, timeouts, blocks (<code>403</code>) and concurrency refusals (<code>429</code>) are free. When the target itself says a page doesn&#8217;t exist, you get its <code>404</code> or <code>410</code> with an <code>X-Origin-Status</code> header, and that one is billed, because the page was reached and retrying won&#8217;t change the answer.</p><p>My first call was the pricing page itself. While I was reading the docs, a plain <code>curl</code> on <code>scrapeunblocker.com/pricing</code> returned a &#8220;Vercel Security Checkpoint&#8221; page, so it looked like a good first test. </p><p>Through <code>/getPageSource</code> with <code>markdown=true</code> the page came back in 3.5 seconds. One detail: the first 175 lines of the Markdown were the cookie consent banner (Cookiebot), and the prices came after it. So &#8220;main content only&#8221; doesn&#8217;t remove cookie banners, and you have to cut them yourself before passing the text to an LLM.</p><p>Here are the prices I read in that Markdown. They&#8217;re in euros, VAT may apply, and there&#8217;s a free trial of 500 requests without a card.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!31ep!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!31ep!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png 424w, https://substackcdn.com/image/fetch/$s_!31ep!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png 848w, https://substackcdn.com/image/fetch/$s_!31ep!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png 1272w, https://substackcdn.com/image/fetch/$s_!31ep!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!31ep!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png" width="816" height="298" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:298,&quot;width&quot;:816,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:35414,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/222188678?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!31ep!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png 424w, https://substackcdn.com/image/fetch/$s_!31ep!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png 848w, https://substackcdn.com/image/fetch/$s_!31ep!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png 1272w, https://substackcdn.com/image/fetch/$s_!31ep!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F558bd461-d358-4964-b7ff-ba7ead40853c_816x298.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>How I set up the tests</h2><p>Before showing the results, I want to be clear about what we can and can&#8217;t see. ScrapeUnblocker is a black box: we send a request and we get JSON back. We don&#8217;t know which route served it (a browser, a direct fetch, an internal API), and we don&#8217;t see what Google or Shopee answered on the way. So this test measures what you get as a customer, and whether the data is right.</p><p>For this reason, I wrote a small client on plain <code>requests</code> (<code>client.py</code>) that does one attempt per call, with no retries. The official SDK retries <code>429</code>, <code>502</code>, <code>503</code> and <code>504</code> automatically, which is what you want in production, but it would hide the failures from my numbers. Every call is logged with status, latency, size, and the ScrapeUnblocker headers (<code>X-Origin-Status</code>, <code>X-SU-Error-Code</code> and so on) in <code>artifacts/calls.jsonl</code>. All the calls ran in sequence from my laptop in Italy, on October 25, 2026.</p><p>Ten requests per target is a small sample. It&#8217;s enough to see if the tool works and what the data looks like, so read the success rates below as a first check. If you need a number for a contract, you need a few thousand calls.</p><p>This is the smallest working version of the two calls, the same file you find as <code>04_minimal_example.py</code>:</p><pre><code><code>import os

import requests
from dotenv import load_dotenv

load_dotenv()
API = "https://api.scrapeunblocker.com"
HEADERS = {"x-scrapeunblocker-key": os.environ["SCRAPEUNBLOCKER_KEY"]}

serp = requests.post(f"{API}/serpApi", params={"keyword": "assicurazione auto", "proxy_country": "it"},
                     headers=HEADERS, timeout=(10, 125))
serp.raise_for_status()
data = serp.json()
print(data["proxyCountry"], "| organic:", len(data["organic"]), "| ads:", len(data["topAds"]) + len(data["bottomAds"]),
      "| AI Overview:", data["aiOverview"] is not None)
for r in data["organic"][:3]:
    print(r["position"], r["title"][:60], r["url"])

shopee = requests.post(f"{API}/marketplace/shopee-search",
                       params={"q": "iphone case", "country": "sg", "max_results": 3},
                       headers=HEADERS, timeout=(10, 125))
shopee.raise_for_status()
for p in shopee.json()["products"]:
    print(p["position"], p["price"], p["currency"], p["discountPercent"], p["labels"], p["name"][:50])</code></code></pre><p>And this is its output:</p><pre><code><code>IT | organic: 10 | ads: 2 | AI Overview: True
1 Assicurazione Auto: Preventivo RCA Online https://www.unipol.it/veicoli-mobilita/assicurazione-auto
2 Assicurazione auto: la polizza pensata per te da Prima https://www.prima.it/assicurazione-auto
3 Assicurazione auto online: scopri la polizza Linear https://www.linear.it/assicurazione-auto
1 34.9 SGD 41 ['10.25 PRICE: $34.90'] Beats iPhone 17 Drenched Case with MagSafe and Cam
2 73.9 SGD 32 ['10.25 PRICE: $73.90'] Beats iPhone 17 Rugged Case with MagSafe and Camer
3 52.9 SGD 41 ['10.25 PRICE: $52.90'] Beats iPhone 17 Kickstand Case with MagSafe and Ca</code></code></pre><p>The timeout of 125 seconds for the read comes from the <a href="https://docs.scrapeunblocker.com/errors?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">errors page</a>: a call can take up to its time cap (60 seconds for a page) plus up to 60 seconds in the queue when all their capacity is busy.</p><h2>Google SERP</h2><p>For Google I chose ten keywords that should give different kinds of result pages: commercial queries, where you expect ads, informational queries, where you expect an AI Overview, and a couple of local ones. I spread them over nine countries and left one without <code>proxy_country</code>, to see where the default exit is. One page per call (<code>pages_to_check=1</code>), because every page is billed as one request.</p><p>There&#8217;s also a small trap in the list. The <a href="https://docs.scrapeunblocker.com/guides/serp-scraping?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">SERP guide</a> lists 24 countries for <code>/serpApi</code>, and Brazil isn&#8217;t one of them. For a country that isn&#8217;t available, the docs say the call doesn&#8217;t fail: it&#8217;s served from another exit, and the response gets two extra fields, <code>requestedCountry</code> and <code>note</code>. So I added &#8220;seguro de carro&#8221; with <code>proxy_country=br</code> to see the fallback.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!KbME!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KbME!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png 424w, https://substackcdn.com/image/fetch/$s_!KbME!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png 848w, https://substackcdn.com/image/fetch/$s_!KbME!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png 1272w, https://substackcdn.com/image/fetch/$s_!KbME!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!KbME!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png" width="813" height="719" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:719,&quot;width&quot;:813,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:65432,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/222188678?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!KbME!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png 424w, https://substackcdn.com/image/fetch/$s_!KbME!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png 848w, https://substackcdn.com/image/fetch/$s_!KbME!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png 1272w, https://substackcdn.com/image/fetch/$s_!KbME!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e83e322-176f-467f-af8e-7fd9d9c91034_813x719.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>All ten calls returned <code>200</code> at the first attempt, between 3.8 and 8.1 seconds (median 6.3). With no country, the request went out from France. And Brazil wasn&#8217;t a fallback at all: the response says <code>"proxyCountry": "BR"</code>, with no <code>requestedCountry</code> and no <code>note</code>, and the results are Brazilian (santander.com.br, nubank, bbseguros). So the country list in the SERP guide is older than the service.</p><h3>What&#8217;s inside the response</h3><p>Every organic result has four fields, <code>title</code>, <code>url</code>, <code>description</code> and <code>position</code>, and the positions go from 1 to the last result with no gaps. The interesting field is <code>url</code>. Remember the <code>goto</code> wrapping from August: Google rewrites the result links for the clients it classifies as automated, and to get the real address back you need one more request per link. Out of the 89 organic results I collected, none pointed to a Google domain. Every URL is the real destination, like <code>https://www.unipol.it/veicoli-mobilita/assicurazione-auto</code>. I can&#8217;t tell from the outside if ScrapeUnblocker gets the page unwrapped or resolves the links on its side, but as a customer you get the final URLs, with no extra work.</p><p>The AI Overview was there on 8 of 10 queries, also on commercial ones like &#8220;iphone 17 pro price&#8221;. It comes as <code>text</code> and <code>markdown</code>, plus <code>sources</code> (the cited hostnames) and <code>references</code> (the cited pages, with title, URL, source name and snippet). For &#8220;assicurazione auto&#8221; the overview starts with &#8220;Puoi confrontare e calcolare i preventivi per l&#8217;assicurazione auto in modo semplice e gratuito tramite il Preventivatore pubblico IVASS...&#8221; and cites eight sites, from ivass.it to prima.it and genertel.it, with ten references, all with a real URL. If you track how brands appear in AI answers, this is probably the most useful part of the payload.</p><p>Ads appeared on 4 of 10 queries, 11 in total, and only the Brazilian one had ads at the top (4 at the top, 2 at the bottom). The docs say that every paid result ends up in <code>topAds</code> or <code>bottomAds</code>, never in <code>organic</code>, and in my sample no organic URL was an ad link. I&#8217;ll be honest, I expected more ads on &#8220;best running shoes&#8221; in the US. But I can&#8217;t compare with what a real user saw from the same exit at the same time, so I&#8217;m only reporting it.</p><h2>Shopee</h2><p>The Shopee plugin has two endpoints. <code>/marketplace/shopee-search</code> takes a keyword (or a shop id, or a Shopee search URL) and a market, and returns one results page of up to 40 products. <code>/marketplace/shopee-product</code> takes a product URL, or a shop id plus an item id, and returns one product with the profile of the shop. It works on eight markets: Singapore, Indonesia, Malaysia, Thailand, Vietnam, the Philippines, Taiwan and Brazil.</p><p>The <a href="https://docs.scrapeunblocker.com/plugins/shopee?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">plugin page</a> is also clear about what you won&#8217;t get. Stock, units sold, the rating count of a product and its reviews are missing, because Shopee doesn&#8217;t show them to visitors who aren&#8217;t logged in. And in four markets (Indonesia, Thailand, Vietnam and the Philippines) a product comes back with only its title and main image, plus price, rating, labels and shop. The response says which case you&#8217;re in, with <code>dataLevel</code> set to <code>full</code> or <code>meta</code>.</p><p>So I planned ten calls to check both claims. Six searches in six markets, one of them (Malaysia) with <code>sort=sales</code> and a price filter between 100 and 300 ringgit. Then four product calls on the first product of four searches, two in markets the docs call &#8220;full&#8221; (Singapore and Brazil) and two in &#8220;meta&#8221; markets (Indonesia and Thailand). Since the product comes from a search, I can also compare the two answers for the same item.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!l7Lt!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!l7Lt!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png 424w, https://substackcdn.com/image/fetch/$s_!l7Lt!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png 848w, https://substackcdn.com/image/fetch/$s_!l7Lt!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png 1272w, https://substackcdn.com/image/fetch/$s_!l7Lt!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!l7Lt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png" width="820" height="633" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:633,&quot;width&quot;:820,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:71929,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/222188678?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!l7Lt!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png 424w, https://substackcdn.com/image/fetch/$s_!l7Lt!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png 848w, https://substackcdn.com/image/fetch/$s_!l7Lt!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png 1272w, https://substackcdn.com/image/fetch/$s_!l7Lt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F37c6feb2-5776-4ab9-8949-710894b6b8c4_820x633.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Again ten out of ten at the first attempt, between 2.6 and 9.3 seconds (median 4.8). In Lab 110 the same error came back from the app, from an HTTP client and from two real browsers. Here I got 240 products in six currencies in about 25 seconds of total waiting, which is quite a change.</p><h3>What a product looks like</h3><p>The Singapore product is the Beats case sold by the Apple Flagship Store, and it&#8217;s the <code>full</code> level. You get the description, brand, categories, two variations (four colors and four iPhone sizes), 16 models, five images, the estimated shipping days (1) and the listing date. The shop block has the rating (4.93), the number of ratings (309,348), followers (186,823), number of products, response rate, the official shop flag and the date the shop joined Shopee. The Brazilian product adds the attributes, the spec rows the seller filled in, like &#8220;Tipo de Conex&#227;o: Sem fio (Bluetooth ou Wifi)&#8221; and the homologation number.</p><p>The two <code>meta</code> products are exactly what the docs describe: title, one image, price, rating, labels and the shop, and nothing about the listing. So if you need descriptions or variations in Indonesia or Thailand, this plugin doesn&#8217;t give them to you, and it says so in the response.</p><p>For all four items the price, the discount, the rating, the labels and the title were the same in the search and in the product call. There&#8217;s one exception, and it&#8217;s a small one: <code>originalPrice</code>. In Brazil the search said 79.90 and the product said <code>null</code>, and in Thailand it was 279 against <code>null</code>. So if you need the crossed-out price, take it from the search.</p><h3>Two prices to check</h3><p>Two results in the searches looked wrong to me at first. In both cases the explanation was in the JSON.</p><p>The cheapest power bank in Thailand costs 1 baht, with an original price of 198. Its label is &#8220;&#3619;&#3634;&#3588;&#3634;&#3614;&#3636;&#3648;&#3624;&#3625;&#3626;&#3635;&#3627;&#3619;&#3633;&#3610;&#3621;&#3641;&#3585;&#3588;&#3657;&#3634;&#3651;&#3627;&#3617;&#3656;&#8221;, which means &#8220;special price for new customers&#8221;. The docs warn you that <code>price</code> is what Shopee shows to a visitor who isn&#8217;t logged in, and that it can be a promotional or new-customer price. This is the case they mean. If you&#8217;re doing price monitoring, you need <code>labels</code> to decide which prices a normal buyer would pay.</p><p>The second is the price filter. In Malaysia I asked for air fryers between 100 and 300 ringgit, and the response shows that the plugin passed the filter to Shopee&#8217;s own search URL (<code>https://shopee.com.my/search?keyword=air%20fryer&amp;sortBy=sales&amp;minPrice=100&amp;maxPrice=300</code>). One product out of 40 came back at 97 ringgit, with an original price of 250 and the label &#8220;Shopee Lagi Murah&#8221;. So the filter is Shopee&#8217;s, with Shopee&#8217;s logic, and if the range matters for your data you should filter again on your side.</p><p>Two smaller details. In Taiwan, 11 of the 40 products had <code>rating: null</code>. And on 36 of the 209 discounted products in the searches you get <code>discountPercent</code> without <code>originalPrice</code> (the example in the docs has the same shape). So make the proper data quality checks before ingesting these products on your data pipeline, so that it won&#8217;t break in later stages.</p><h2>Shein</h2><p>After Shopee I wanted one more e-commerce target, and Shein was the natural one, since fast-fashion price tracking is one of the use cases its plugin page lists. The <a href="https://docs.scrapeunblocker.com/plugins/shein?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">Shein plugin</a> is smaller than the Shopee one. It has only a search endpoint, <code>/marketplace/shein-search</code>, on the US store, and it returns the first results page, up to 20 products. You can sort by relevance or price and filter by a price range in dollars. The docs also say that Shein limits bursts of searches, so I left three seconds between calls.</p><p>Since there&#8217;s no product endpoint, for the product pages I used <code>/getPageSource</code> with <code>parsed_data=true</code> on the URLs the search returned. This is the generic route, the one you&#8217;d use on any website, so it&#8217;s also a way to compare the plugin with the generic unblocker on the same items. I ran six searches (one sorted by price and one with a price filter) and four product pages, the first product of four of the searches.</p><p>All ten calls returned <code>200</code>. The first search took 53.8 seconds, and every call after it took between 1.7 and 6.7 seconds (the searches around 2 seconds each). I don&#8217;t know why the first one was so slow, and with one cold start in my sample I can&#8217;t say if it happens every time. Every search returned 20 products, with title, price, original price, discount, rating, review count, category, image, sold-out flag and the code of the store that sells the product.</p><p>The sort and the filter showed the same behavior I saw on Shopee. With <code>sort=price_low</code> the prices go up from 3.43 to 4.80, except for one product at 9.08 at position 11. With a filter between 10 and 30 dollars on &#8220;men hoodie&#8221;, one product came back at 9.85 (original price 16.59). Also, <code>reviewCount</code> in the search stops at 1001 for popular products, which is how Shein&#8217;s &#8220;1000+&#8221; ends up in the JSON. The product page gives the exact number (1,468 and 1,680 ratings on two of my items).</p><p>The parsed product pages are rich. You get material, composition, fit, care instructions, neckline, sleeve type and a list of images, all extracted without a line of parsing code on my side. The price is a different story. Only one of the four products had the same price in the search and in the parsed page (12.39 dollars for a mini dress). The other three were higher on the page: 2.91 against 2.71, 13.08 against 10.08 and 17.70 against 12.40.</p><p>So I fetched the raw HTML of the two biggest differences (two more calls) and looked for both numbers. On the kids sneakers, the price the page shows to the user is 12.40, in <code>aria-label="Price $12.40"</code> and in the <code>product:price:amount</code> meta tag, the same as the search. The 17.70 comes from the Schema.org data in the page, where every size has its own offer, with prices from 12.40 to 19.00. So <code>parsed_data</code> took one of those offers instead of the price on the screen.</p><p>The hoodie was stranger. In a few minutes I got three prices for the same product: 10.08 from the search, 13.08 from the parsed page and 12.83 in the HTML I downloaded later. From the outside I can&#8217;t tell why the price moved, only that it did. For Shein, I&#8217;d take the price from the plugin, and I wouldn&#8217;t compare prices from different calls as if they were taken at the same moment.</p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>What it costs</h2><p>The whole test, including the pricing page, the minimal example and the Shein calls, was 35 billed requests. At the pay-as-you-go price of &#8364;1 per 1,000 that&#8217;s &#8364;0.035, and it&#8217;s well inside the 500 free requests of the trial. Since every request costs the same, a Shopee product with variations and shop profile costs like a Google page, which costs like any HTML page.</p><p>To put a number on it, monitoring 10,000 keywords a day on Google, one page each, is about 300,000 requests a month. That fits in the Business plan (&#8364;299 with 440,000 requests included). One Shopee search per keyword in each of the eight markets, for 1,000 keywords a day, is 240,000 requests a month, again in the Business plan. These are just list prices from their page, and if you go above 1.8 million requests a month there&#8217;s an Enterprise plan with custom pricing.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!KbLR!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!KbLR!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png 424w, https://substackcdn.com/image/fetch/$s_!KbLR!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png 848w, https://substackcdn.com/image/fetch/$s_!KbLR!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png 1272w, https://substackcdn.com/image/fetch/$s_!KbLR!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!KbLR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png" width="295" height="308" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:308,&quot;width&quot;:295,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:34940,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/222188678?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!KbLR!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png 424w, https://substackcdn.com/image/fetch/$s_!KbLR!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png 848w, https://substackcdn.com/image/fetch/$s_!KbLR!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png 1272w, https://substackcdn.com/image/fetch/$s_!KbLR!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F88e57f5b-4fb4-44e3-ae3c-5ee30590a5c3_295x308.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p></p><h2>How the API compares with the Oxylabs Web API</h2><p>A few days ago, for another Tool of the Week, I tested the <a href="https://oxylabs.io/products/web-api?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">Oxylabs Web API</a>, and the two products answer the same need: one key, and the web as JSON or Markdown. But the endpoints are designed in a different way, so I put them side by side. The Oxylabs column comes from its docs as of October 19 and from the endpoint list I downloaded on October 20 with <code>GET /v1/scrapers</code>.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!qgT8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!qgT8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png 424w, https://substackcdn.com/image/fetch/$s_!qgT8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png 848w, https://substackcdn.com/image/fetch/$s_!qgT8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png 1272w, https://substackcdn.com/image/fetch/$s_!qgT8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!qgT8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png" width="816" height="836" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:836,&quot;width&quot;:816,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:140334,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/222188678?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!qgT8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png 424w, https://substackcdn.com/image/fetch/$s_!qgT8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png 848w, https://substackcdn.com/image/fetch/$s_!qgT8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png 1272w, https://substackcdn.com/image/fetch/$s_!qgT8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8346d1af-a049-46bb-a18f-ff6492a765cd_816x836.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The biggest difference, in my opinion, is how much you decide. With Oxylabs you choose if a page needs JavaScript, which formats you want and if the job runs now or later, and you pay a different price for each choice. With ScrapeUnblocker you send the URL and the service decides the route, at the same price for every call. You get less control, and in exchange the cost of a project is easier to calculate (I&#8217;ll leave it to you which side of this you prefer, it depends a lot on the project).</p><p>On dedicated scrapers, the two catalogs overlap less than I expected. Out of the 75 ScrapeUnblocker plugins and the 54 Oxylabs targets, only 12 websites are in both lists: Google, Bing, Amazon, Walmart, Target, Best Buy, Etsy, eBay, AliExpress, Alibaba, TikTok Shop and YouTube. Even there, the coverage isn&#8217;t the same. On Google both have the SERP, Shopping, Maps, Trends and Hotels, but only ScrapeUnblocker has Images, Jobs, News and Ads Transparency, and only Oxylabs has AI Mode and Lens. On Amazon both have products, search and best sellers, while reviews are only on ScrapeUnblocker and pricing and sellers only on Oxylabs.</p><p>Outside these 12, the catalogs go in different directions. The ScrapeUnblocker plugins cover a lot of Asian and European marketplaces (Shopee, Taobao and Tmall, 1688, Zalando, Vinted, Leboncoin) and travel (Booking, Airbnb, Skyscanner, Tripadvisor). They also cover real estate, jobs, review sites and SEO tools like Ahrefs, Semrush and Similarweb. The Oxylabs targets have a lot of US retail and grocery chains (Kroger and its regional stores, Costco, Instacart, Publix, Safeway) and Latin American marketplaces (Mercado Libre, Magazine Luiza, Falabella). Then there are Flipkart, Lazada, Allegro and the three AI assistants. Remember that both have a generic endpoint for any URL, so a site missing from a list can still be scraped. You just don&#8217;t get a parser maintained for you.</p><h2>Final considerations</h2><p>All three targets worked on every request I sent, 32 calls without a single error. On Shopee the search and product calls gave the same price for the same item, and Google results came from the country I asked for (Brazil included, even if the docs don&#8217;t list it). On Shein it&#8217;s less clean: on the product I checked in the HTML the plugin had the price shown on the screen and <code>parsed_data</code> didn&#8217;t, and on another product the price moved between calls. With these numbers, I can&#8217;t say more than this about reliability. What I can say is that the hard part of Google and Shopee, the part where a whole Lab article ended without a working scraper, is handled on their side.</p><p>What you still need to do is read the data correctly. On Shopee that means the new-customer prices, the price filter that lets some results through and the <code>originalPrice</code> that disappears in the product call. On Google it&#8217;s <code>totalResults</code>, which doesn&#8217;t match the real count. On Shein it&#8217;s the <code>parsed_data</code> price, which can be the price of one size instead of the one on the screen. Each of these takes a few lines of code to handle once you know about it, and each one would put wrong numbers in a dashboard if you used the JSON as it comes.</p><p>There are two next steps I&#8217;d like to try. The first is a bigger sample with some concurrency, to see how the success rate holds at the limit of a plan. The second is the same comparison I did on Shein, between the plugin and <code>/getPageSource</code>, on Shopee product URLs.</p><p>The code of the tests (<code>client.py</code>, the test scripts from <code>01_smoke_test.py</code> to <code>06_shein_page_prices.py</code>) is in <a href="https://github.com/TheWebScrapingClub/TheScrapingClubFree/tree/main/unblocker-tests">our GitHub repository</a>. Let me know in the comments if you&#8217;ve already used ScrapeUnblocker on these or other hard targets.</p>]]></content:encoded></item><item><title><![CDATA[Scraping JavaScript-Rendered Pages: Signs, Tests, and Tool Choice]]></title><description><![CDATA[A field guide to deciding when a headless browser can be the answer.]]></description><link>https://www.scraping.club/p/scraping-javascript-rendered-pages</link><guid isPermaLink="false">https://www.scraping.club/p/scraping-javascript-rendered-pages</guid><dc:creator><![CDATA[Federico Trotta]]></dc:creator><pubDate>Sun, 25 Oct 2026 04:01:10 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/b1bc542b-c360-4fbf-8189-f54841302ef7_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Understanding whether a page renders content using JavaScript should start with a simple question: &#8220;Is my scraper returning nothing because the page needs JavaScript, or because something else is wrong?&#8221;</p><p>That&#8217;s because you can consider any empty extraction as proof that the page is JavaScript-rendered and reach for a browser right away. But that only addresses part of the problem, and often it addresses the wrong one. This happens because an empty result can also come from a wrong selector, from a redirect your client followed without you noticing, or from a CAPTCHA page served in place of the content you actually wanted.</p><p>That&#8217;s where a diagnostic process comes in, and it makes the decision much faster and cheaper.</p><p>In this article, we&#8217;ll discuss how a headless browser differs from a plain HTTP request, the signs that suggest JavaScript rendering is involved, the quick tests to run before you commit to a browser, and how to choose between Puppeteer, Playwright, and Selenium based on the shape of your actual workload.</p><div><hr></div><div class="callout-block" data-callout="true"><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/B5JPeJ">Decodo&#8217;s Web Data API</a></strong> turns any site into clean, structured data your models can actually use.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/B5JPeJ" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png" width="716" height="286.4" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:280,&quot;width&quot;:700,&quot;resizeWidth&quot;:716,&quot;bytes&quot;:105634,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;text&quot;:&quot;Activate your free plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/B5JPeJ"><span>Activate your free plan</span></a></p></div><div><hr></div><h2>How a Headless Browser Differs From a Plain HTTP Request</h2><p>A headless browser is an actual browser engine that runs without a visible interface. It loads pages, <a href="http://substack.thewebscraping.club/p/scraping-with-llms-gpt-vision">builds the document object model (DOM)</a>, executes JavaScript, and handles interaction like a human visitor would do on the target website. In other words, you are programmatically controlling a real browser under the hood of your requests, managing clicks on an onscreen video.</p><p>That matters because a page&#8217;s initial HTML isn&#8217;t always the page you see after it finishes loading. In many web pages, what you don&#8217;t see is that JavaScript can request additional data, insert product listings, update prices, or reveal content after an interaction. This means that the website you&#8217;re visiting for scraping it displays the whole data after some interaction.</p><p>In those cases, headless browsers come in handy. That is because a headless browser can execute the needed interactions and let you inspect the resulting DOM. It can also click buttons, scroll, wait for specific elements to appear, and much more.</p><p>On the other hand, a plain HTTP client takes a narrower approach. It sends a request to a URL and downloads the server&#8217;s response. If that response contains HTML, you can parse it and extract whatever&#8217;s already there. However, the client doesn&#8217;t execute page scripts, build the live DOM, or automatically make the additional requests those scripts would trigger.</p><p>The image below schematically shows that difference in request mode:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CHAi!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CHAi!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png 424w, https://substackcdn.com/image/fetch/$s_!CHAi!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png 848w, https://substackcdn.com/image/fetch/$s_!CHAi!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png 1272w, https://substackcdn.com/image/fetch/$s_!CHAi!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!CHAi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png" width="981" height="509" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:509,&quot;width&quot;:981,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Plain HTTP requests vs using headless browsers by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Plain HTTP requests vs using headless browsers by Federico Trotta" title="Plain HTTP requests vs using headless browsers by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!CHAi!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png 424w, https://substackcdn.com/image/fetch/$s_!CHAi!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png 848w, https://substackcdn.com/image/fetch/$s_!CHAi!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png 1272w, https://substackcdn.com/image/fetch/$s_!CHAi!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7395d45a-8349-43c6-af63-77015c3dcc18_981x509.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Plain HTTP requests vs using headless browsers</figcaption></figure></div><div><hr></div><div class="callout-block" data-callout="true"><p><em>Playwright, Puppeteer, or Selenium? Choosing your browser is one part of the setup. <a href="https://anyip.io/?utm_source=thewebscrapingclub&amp;utm_medium=newsletter&amp;utm_campaign=gold_2026_10&amp;utm_content=banner_headless_browsers">anyIP</a> provides residential and mobile proxies worldwide to support your scraping workflows. From the very first request to the pages that need a full browser.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://anyip.io/?utm_source=thewebscrapingclub&amp;utm_medium=newsletter&amp;utm_campaign=gold_2026_10&amp;utm_content=banner_headless_browsers" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gpFu!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png 424w, https://substackcdn.com/image/fetch/$s_!gpFu!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png 848w, https://substackcdn.com/image/fetch/$s_!gpFu!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png 1272w, https://substackcdn.com/image/fetch/$s_!gpFu!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gpFu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png" width="1456" height="728" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:944781,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://anyip.io/?utm_source=thewebscrapingclub&amp;utm_medium=newsletter&amp;utm_campaign=gold_2026_10&amp;utm_content=banner_headless_browsers&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/219545817?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!gpFu!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png 424w, https://substackcdn.com/image/fetch/$s_!gpFu!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png 848w, https://substackcdn.com/image/fetch/$s_!gpFu!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png 1272w, https://substackcdn.com/image/fetch/$s_!gpFu!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa4fffdd2-539e-466b-b4b2-f48ebfe3a4ec_4585x2293.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://anyip.io/?utm_source=thewebscrapingclub&amp;utm_medium=newsletter&amp;utm_campaign=gold_2026_10&amp;utm_content=banner_headless_browsers&quot;,&quot;text&quot;:&quot;Explore anyIP Proxies &#8594;&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://anyip.io/?utm_source=thewebscrapingclub&amp;utm_medium=newsletter&amp;utm_campaign=gold_2026_10&amp;utm_content=banner_headless_browsers"><span>Explore anyIP Proxies &#8594;</span></a></p></div><div><hr></div><h2>Symptoms a Page Needs a Headless Browser</h2><p>Now, here&#8217;s a question for you: &#8220;How do you know if your requests actually need a headless browser before starting to scrape a target website?&#8221;</p><p>This question is important, but unfortunately there&#8217;s no quick answer. That&#8217;s because scraping websites is an engineering matter. And like any engineering task, you should proceed methodically and scientifically, as much as possible. Specifically speaking, an empty extraction result can make a headless browser seem like the obvious next step. But &#8220;my scraper found nothing&#8221; doesn&#8217;t necessarily mean &#8220;this page needs JavaScript&#8221; because:</p><ul><li><p>Your selector could be wrong.</p></li><li><p>Your request could&#8217;ve followed a redirect.</p></li><li><p>The server could&#8217;ve returned a CAPTCHA challenge instead of the content you expected.</p></li></ul><p>So, the signs presented in this paragraph suggest that fetching and parsing the page&#8217;s HTML won&#8217;t be enough. However, they don&#8217;t automatically prove you need a headless browser. For a complete engineering process, other than only the symptoms, consider also the tests you can perform in the next section before deciding whether you actually need a headless browser.</p><h3>Symptom #1: Empty or Skeletal HTML Source</h3><p>A skeletal response contains the structure of a web application, but little of the content you want to scrape. Below is an example of it:</p><pre><code><code>&lt;!DOCTYPE html&gt;
&lt;html lang="en"&gt;
  &lt;head&gt;
    &lt;meta charset="utf-8" /&gt;
    &lt;meta name="viewport" content="width=device-width, initial-scale=1" /&gt;
    &lt;title&gt;Acme Store&lt;/title&gt;
    &lt;meta name="description" content="" /&gt;
    &lt;link rel="icon" href="/static/favicon.ico" /&gt;
    &lt;link rel="preconnect" href="https://api.acme-store.com" crossorigin /&gt;
    &lt;link rel="stylesheet" href="/static/css/main.3f2a91c4.chunk.css" /&gt;
    &lt;link rel="preload" href="/static/js/runtime~main.8c1b2d0e.js" as="script" /&gt;
  &lt;/head&gt;
  &lt;body&gt;
    &lt;noscript&gt;You need to enable JavaScript to run this app.&lt;/noscript&gt;
    &lt;div id="root"&gt;&lt;/div&gt;

    &lt;script&gt;
      window.__APP_CONFIG__ = {
        apiBase: "https://api.acme-store.com/v2",
        locale: "en-US",
        buildId: "20261007.1"
      };
    &lt;/script&gt;
    &lt;script src="/static/js/runtime~main.8c1b2d0e.js"&gt;&lt;/script&gt;
    &lt;script src="/static/js/vendors~main.7b4e9a12.chunk.js"&gt;&lt;/script&gt;
    &lt;script src="/static/js/main.5d0c3f77.chunk.js"&gt;&lt;/script&gt;
  &lt;/body&gt;
&lt;/html&gt;</code></code></pre><p>In such cases, everything a scraper would want to get (like product names, prices, table rows, article text, etv) is absent and only exists once the system loads the three bundles. The bundling happens at the bottom when the script executes and fetches data from the <em>apiBase</em> endpoint declared in <em>window.<strong>APP_CONFIG</strong></em>. This means that, in such a case, a browser can execute the bundles, retrieve data, and build the missing elements. Only after that process is completed, your scraper can get the data you want from the target page.</p><p>Other similar responses look more complete, but still contain only placeholders. For example, you might find something like &#8220;Loading&#8230;&#8221; text, markup for a spinner, or an empty product grid. Often, even navigation and headings are present, yet product names, prices, and listings are absent. These symptoms suggest that JavaScript supplies the content after the initial document arrives.</p><p>However, missing elements don&#8217;t necessarily mean missing data. Some pages embed their records in script tags or serialized application state. If your target values are already there, you can potentially extract them without rendering. That is exactly what the upcoming example shows. Consider <a href="https://quotes.toscrape.com/js/">this target page that renders quotes dynamically</a>:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!HFmO!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!HFmO!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png 424w, https://substackcdn.com/image/fetch/$s_!HFmO!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png 848w, https://substackcdn.com/image/fetch/$s_!HFmO!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png 1272w, https://substackcdn.com/image/fetch/$s_!HFmO!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!HFmO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png" width="1456" height="613" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:613,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Quotes to scrape dynamically rendered by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Quotes to scrape dynamically rendered by Federico Trotta" title="Quotes to scrape dynamically rendered by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!HFmO!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png 424w, https://substackcdn.com/image/fetch/$s_!HFmO!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png 848w, https://substackcdn.com/image/fetch/$s_!HFmO!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png 1272w, https://substackcdn.com/image/fetch/$s_!HFmO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F036be454-14a7-4d6a-882c-449b770ccb37_1758x740.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Quotes to scrape dynamically rendered</figcaption></figure></div><p>In an activated virtual environment, install the needed libraries:</p><pre><code><code>pip install requests beautifulsoup4</code></code></pre><p>Use the following Python script:</p><pre><code><code>import requests

from bs4 import BeautifulSoup
response = requests.get("https://quotes.toscrape.com/js/", timeout=30)
soup = BeautifulSoup(response.text, "html.parser")

print(response.text)</code></code></pre><p>When running the snippet, the result you will receive is the following:</p><pre><code><code>&lt;!DOCTYPE html&gt;
&lt;html lang="en"&gt;
&lt;head&gt;
    &lt;meta charset="UTF-8"&gt;
    &lt;title&gt;Quotes to Scrape&lt;/title&gt;
    &lt;link rel="stylesheet" href="/static/bootstrap.min.css"&gt;
    &lt;link rel="stylesheet" href="/static/main.css"&gt;
&lt;/head&gt;
&lt;body&gt;
    &lt;div class="container"&gt;
        &lt;div class="row header-box"&gt;
            &lt;div class="col-md-8"&gt;
                &lt;h1&gt;
                    &lt;a href="/" style="text-decoration: none"&gt;Quotes to Scrape&lt;/a&gt;
                &lt;/h1&gt;
            &lt;/div&gt;
            &lt;div class="col-md-4"&gt;
                &lt;p&gt;
                    &lt;a href="/login"&gt;Login&lt;/a&gt;
                &lt;/p&gt;
            &lt;/div&gt;
        &lt;/div&gt;

        &lt;script src="/static/jquery.js"&gt;&lt;/script&gt;
        &lt;script&gt;
            var data = [
                {
                    "tags": [
                        "change",
                        "deep-thoughts",
                        "thinking",
                        "world"
                    ],
                    "author": {
                        "name": "Albert Einstein",
                        "goodreads_link": "/author/show/9810.Albert_Einstein",
                        "slug": "Albert-Einstein"
                    },
                    "text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d"
                },

                // {...OMITTED FOR BREVITY...}

            ];
            for (var i in data) {
                var d = data[i];
                var tags = $.map(d['tags'], function(t) {
                    return "&lt;a class='tag'&gt;" + t + "&lt;/a&gt;";
                }).join(" ");
                document.write("&lt;div class='quote'&gt;&lt;span class='text'&gt;" + d\['text'] + "&lt;/span&gt;&lt;span&gt;by &lt;small class='author'&gt;" + d['author'\]['name'] + "&lt;/small&gt;&lt;/span&gt;&lt;div class='tags'&gt;Tags: " + tags + "&lt;/div&gt;&lt;/div&gt;");
            }
        &lt;/script&gt;
        &lt;nav&gt;
            &lt;ul class="pager"&gt;
                &lt;li class="next"&gt;
                    &lt;a href="/js/page/2/"&gt;Next &lt;span aria-hidden="true"&gt;&amp;rarr;&lt;/span&gt;&lt;/a&gt;
                &lt;/li&gt;
            &lt;/ul&gt;
        &lt;/nav&gt;
    &lt;/div&gt;
    &lt;footer class="footer"&gt;
        &lt;div class="container"&gt;
            &lt;p class="text-muted"&gt;
                Quotes by: &lt;a href="https://www.goodreads.com/quotes"&gt;GoodReads.com&lt;/a&gt;
            &lt;/p&gt;
            &lt;p class="copyright"&gt;
                Made with &lt;span class='zyte'&gt;&#10084;&lt;/span&gt; by &lt;a class='zyte' href="https://www.zyte.com"&gt;Zyte&lt;/a&gt;
            &lt;/p&gt;
        &lt;/div&gt;
    &lt;/footer&gt;
&lt;/body&gt;
&lt;/html&gt;</code></code></pre><p>What matters here is focusing on &lt;<em>script src=&#8221;/static/jquery.js&#8221;&gt;&lt;/script&gt;</em>. This means that the content is loaded via a script. But since requests has no JavaScript engine and no rendering engine, BeautifulSoup parses the text into a tree, but it never executes anything inside a <em>script</em>. In other words, you get the whole nested content, not the &#8220;clean&#8221; one you&#8217;d like.</p><div><hr></div><div class="callout-block" data-callout="true"><p><em>&#8220;Data is the new oil.&#8221;<br>You&#8217;ve heard it a hundred times, but do you know who does the drilling?<br>With AI, it&#8217;s more true than ever. Models are trained on data scraped from the web. Agents need browser automation that doesn&#8217;t fall over at the first CAPTCHA. But who&#8217;s doing the heavy lifting? Proxies that actually work.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://luma.com/5c4v7zjk" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6re3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 424w, https://substackcdn.com/image/fetch/$s_!6re3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 848w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1272w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6re3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png" width="318" height="397.3582887700535" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1402,&quot;width&quot;:1122,&quot;resizeWidth&quot;:318,&quot;bytes&quot;:1471113,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://luma.com/5c4v7zjk&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/221688546?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!6re3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 424w, https://substackcdn.com/image/fetch/$s_!6re3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 848w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1272w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>That&#8217;s why we partnered with <strong><a href="https://www.linkedin.com/company/geonode/">Geonode</a></strong> for an aperitivo on 5 November in Milan.<br>No talks to sit through but just people who build scrapers, data pipelines and AI agents for a living, and a drink.<br><br>If you&#8217;re one of them, here&#8217;s the link to join: <strong><a href="https://www.linkedin.com/safety/go/?url=https%3A%2F%2Fluma%2Ecom%2F5c4v7zjk&amp;urlhash=TG6m&amp;mt=_MVhK0tQUX3JJQvNRPrQeq4u7WW0fNOSWRfZa99BOWYnJowsTzf1EnbBN_-HBrEpzjxD1u-cfdNwXHw28VhbwRGPBXuXqZ7OSb9qW7AhN0Eryqhpw3SR_IT3&amp;isSdui=true">https://luma.com/5c4v7zjk</a></strong></em></p></div><div><hr></div><p>Another way to see this is to inspect the page to intercept the selector that contains the quotes:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xIF0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xIF0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png 424w, https://substackcdn.com/image/fetch/$s_!xIF0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png 848w, https://substackcdn.com/image/fetch/$s_!xIF0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png 1272w, https://substackcdn.com/image/fetch/$s_!xIF0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xIF0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png" width="1456" height="592" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:592,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Selectors in quotes to scrape by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Selectors in quotes to scrape by Federico Trotta" title="Selectors in quotes to scrape by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!xIF0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png 424w, https://substackcdn.com/image/fetch/$s_!xIF0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png 848w, https://substackcdn.com/image/fetch/$s_!xIF0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png 1272w, https://substackcdn.com/image/fetch/$s_!xIF0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78144739-34cd-4a5f-9af2-4cf96ab4298b_1807x735.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Selectors in quotes to scrape</figcaption></figure></div><p>Now, if you run the following Python script:</p><pre><code><code>import requests

from bs4 import BeautifulSoup
response = requests.get("https://quotes.toscrape.com/js/", timeout=30)
soup = BeautifulSoup(response.text, "html.parser")

print(soup.select(".product-item"))</code></code></pre><p>You will obtain an empty list like so:</p><pre><code><code>[]</code></code></pre><p>That happens exactly because the code can&#8217;t intercept the selector, because the selector needs JavaScript rendering as a previous step.</p><h3>Symptom #2: Content Hidden Behind User Interaction</h3><p>Some content can&#8217;t be rendered until a visitor clicks, scrolls, hovers, or submits a form. <a href="https://www.scraping.club/p/handling-infinite-scrolling-python-js">Infinite-scroll listings</a>, &#8220;load more&#8221; buttons, and expandable menus are common examples of this behaviour: the initial response contains the first batch of records, while JavaScript requests additional batches as the user interacts.</p><p>Multi-step flows introduce further dependencies:</p><ul><li><p>A login can establish a session.</p></li><li><p>A date picker can set search parameters.</p></li><li><p>A filter can trigger a new request.</p></li><li><p>Later results may depend on cookies or values produced by earlier actions.</p></li></ul><p>So, merely opening the page in a headless browser won&#8217;t complete those steps automatically.</p><p>These rendering processes are important to understand, because successfully scraping the initial HTML doesn&#8217;t guarantee complete coverage in some cases.</p><h3>Symptom #3: Hitting Bot Defenses </h3><p>Sometimes your HTTP client receives a defense page rather than the requested content. In such cases, the underlying page might be fully rendered, but an access-verification layer prevents your client from reaching it.</p><p>Typical symptoms of this include an interstitial &#8220;checking your browser&#8221; page, a <em>403</em> response, or a <a href="https://www.scraping.club/p/bypassing-recaptchas-with-open-source">CAPTCHA that appears for your HTTP client</a> but not during an ordinary browser visit. However, none of these symptoms alone proves that JavaScript rendering is the solution. A <em>403</em> can indicate missing authorization, and CAPTCHA behavior can vary with session history, request volume, or network origin.</p><p>But note that these are not the only ones. For example, say you want to scrape the following <a href="https://www.loccitane.com/en-us/immortelle-divine-youth-oil-27DH030I22.html">target product from L&#8217;Occitane</a>:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!SNxk!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66553592-f747-416a-a975-f153be171ffd_997x888.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SNxk!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66553592-f747-416a-a975-f153be171ffd_997x888.png 424w, https://substackcdn.com/image/fetch/$s_!SNxk!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66553592-f747-416a-a975-f153be171ffd_997x888.png 848w, https://substackcdn.com/image/fetch/$s_!SNxk!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66553592-f747-416a-a975-f153be171ffd_997x888.png 1272w, https://substackcdn.com/image/fetch/$s_!SNxk!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66553592-f747-416a-a975-f153be171ffd_997x888.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SNxk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66553592-f747-416a-a975-f153be171ffd_997x888.png" width="997" height="888" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/66553592-f747-416a-a975-f153be171ffd_997x888.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:888,&quot;width&quot;:997,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Target product from L&#8217;Occitane by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Target product from L&#8217;Occitane by Federico Trotta" title="Target product from L&#8217;Occitane by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!SNxk!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66553592-f747-416a-a975-f153be171ffd_997x888.png 424w, https://substackcdn.com/image/fetch/$s_!SNxk!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66553592-f747-416a-a975-f153be171ffd_997x888.png 848w, https://substackcdn.com/image/fetch/$s_!SNxk!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66553592-f747-416a-a975-f153be171ffd_997x888.png 1272w, https://substackcdn.com/image/fetch/$s_!SNxk!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66553592-f747-416a-a975-f153be171ffd_997x888.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Target product from L&#8217;Occitane</figcaption></figure></div><p>Consider using the following script:</p><pre><code><code>import requests
from bs4 import BeautifulSoup

response = requests.get("https://www.loccitane.com/en-us/immortelle-divine-youth-oil-27DH030I22.html", timeout=30)
soup = BeautifulSoup(response.text, "html.parser")

print(f"HTTP status : {response.status_code}")
print(f"HTML length : {len(response.text):,} characters")</code></code></pre><p>When running the script, the result you will obtain is the following:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!8RJv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0f0659b-6819-4d06-b638-c3e8d9758a3a_220x36.png 424w, https://substackcdn.com/image/fetch/$s_!8RJv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0f0659b-6819-4d06-b638-c3e8d9758a3a_220x36.png 848w, https://substackcdn.com/image/fetch/$s_!8RJv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0f0659b-6819-4d06-b638-c3e8d9758a3a_220x36.png 1272w, https://substackcdn.com/image/fetch/$s_!8RJv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0f0659b-6819-4d06-b638-c3e8d9758a3a_220x36.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!8RJv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0f0659b-6819-4d06-b638-c3e8d9758a3a_220x36.png" width="220" height="36" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a0f0659b-6819-4d06-b638-c3e8d9758a3a_220x36.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:36,&quot;width&quot;:220,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The call result by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The call result by Federico Trotta" title="The call result by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!8RJv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0f0659b-6819-4d06-b638-c3e8d9758a3a_220x36.png 424w, https://substackcdn.com/image/fetch/$s_!8RJv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0f0659b-6819-4d06-b638-c3e8d9758a3a_220x36.png 848w, https://substackcdn.com/image/fetch/$s_!8RJv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0f0659b-6819-4d06-b638-c3e8d9758a3a_220x36.png 1272w, https://substackcdn.com/image/fetch/$s_!8RJv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa0f0659b-6819-4d06-b638-c3e8d9758a3a_220x36.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">The call result</figcaption></figure></div><p>This is where you could think that everything went well. After all, the status is <em>200</em> and content is rendered (807 characters). However, change the script to the following:</p><pre><code><code>import requests
from bs4 import BeautifulSoup

response = requests.get("https://www.loccitane.com/en-us/immortelle-divine-youth-oil-27DH030I22.html", timeout=30)
soup = BeautifulSoup(response.text, "html.parser")

print(response.text)</code></code></pre><p>When running it, you will see that your scraper has hit a CAPTCHA:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!N-rs!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00eaa110-4583-4bf9-8678-9f12ef7c12aa_1407x93.png 424w, https://substackcdn.com/image/fetch/$s_!N-rs!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00eaa110-4583-4bf9-8678-9f12ef7c12aa_1407x93.png 848w, https://substackcdn.com/image/fetch/$s_!N-rs!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00eaa110-4583-4bf9-8678-9f12ef7c12aa_1407x93.png 1272w, https://substackcdn.com/image/fetch/$s_!N-rs!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00eaa110-4583-4bf9-8678-9f12ef7c12aa_1407x93.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!N-rs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00eaa110-4583-4bf9-8678-9f12ef7c12aa_1407x93.png" width="1407" height="93" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/00eaa110-4583-4bf9-8678-9f12ef7c12aa_1407x93.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:93,&quot;width&quot;:1407,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The call result, hignlighting the CAPTCHA by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The call result, hignlighting the CAPTCHA by Federico Trotta" title="The call result, hignlighting the CAPTCHA by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!N-rs!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00eaa110-4583-4bf9-8678-9f12ef7c12aa_1407x93.png 424w, https://substackcdn.com/image/fetch/$s_!N-rs!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00eaa110-4583-4bf9-8678-9f12ef7c12aa_1407x93.png 848w, https://substackcdn.com/image/fetch/$s_!N-rs!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00eaa110-4583-4bf9-8678-9f12ef7c12aa_1407x93.png 1272w, https://substackcdn.com/image/fetch/$s_!N-rs!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00eaa110-4583-4bf9-8678-9f12ef7c12aa_1407x93.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">The call result, highlighting the CAPTCHA</figcaption></figure></div><p>Note that, in such cases, using a headless browser is not a guarantee to pass the CAPTCHA wall. Automated browsers can also be detected or blocked. However, you can use headless browsers for debugging purposes and, for example, take a screenshot when the content is rendered to visualize what&#8217;s happening:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!e0NS!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!e0NS!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg 424w, https://substackcdn.com/image/fetch/$s_!e0NS!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg 848w, https://substackcdn.com/image/fetch/$s_!e0NS!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!e0NS!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!e0NS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg" width="577" height="609" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:609,&quot;width&quot;:577,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Tha CAPTCHA from L&#8217;Occitaine by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Tha CAPTCHA from L&#8217;Occitaine by Federico Trotta" title="Tha CAPTCHA from L&#8217;Occitaine by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!e0NS!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg 424w, https://substackcdn.com/image/fetch/$s_!e0NS!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg 848w, https://substackcdn.com/image/fetch/$s_!e0NS!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!e0NS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffe8fbe83-2568-4ae4-8e71-353baa9cd395_577x609.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Tha CAPTCHA from L&#8217;Occitaine</figcaption></figure></div><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><h3>Quick Tests to Run Before Reaching for a Browser</h3><p>As you&#8217;ve seen until now, there&#8217;s a set of different situations in which you may need a headless browser to render content before you retrieve the data from a target website.</p><p>In this paragraph, I want to show you some quick tests you can perform on a target page to understand if you need JavaScript rendering.</p><h3>Inspect Elements</h3><p>The first test you should always do is inspect the page to see if there are some tags that suggest content is loaded dynamically. I previously showed that the website &#8220;Quotes to scrape&#8221; loads its content via &lt;<em>script src=&#8221;/static/jquery.js&#8221;&gt;&lt;/script&gt;</em>. This can be seen when inspecting the page:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!usIG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!usIG!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png 424w, https://substackcdn.com/image/fetch/$s_!usIG!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png 848w, https://substackcdn.com/image/fetch/$s_!usIG!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png 1272w, https://substackcdn.com/image/fetch/$s_!usIG!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!usIG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png" width="1456" height="727" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:727,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Quotes to scrape dynamically rendered content by a script by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Quotes to scrape dynamically rendered content by a script by Federico Trotta" title="Quotes to scrape dynamically rendered content by a script by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!usIG!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png 424w, https://substackcdn.com/image/fetch/$s_!usIG!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png 848w, https://substackcdn.com/image/fetch/$s_!usIG!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png 1272w, https://substackcdn.com/image/fetch/$s_!usIG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffbc369fc-708b-4a96-975e-b8a40885e470_1626x812.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Quotes to scrape dynamically rendered content by a script</figcaption></figure></div><h3>Disable JavaScript and Reload</h3><p>Sometimes, there is no visible script tag that tells you the content is dynamically rendered. For example, consider the <a href="https://www.scrapingcourse.com/javascript-rendering">following target page</a>:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!4hxz!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4hxz!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png 424w, https://substackcdn.com/image/fetch/$s_!4hxz!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png 848w, https://substackcdn.com/image/fetch/$s_!4hxz!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png 1272w, https://substackcdn.com/image/fetch/$s_!4hxz!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!4hxz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png" width="1446" height="844" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:844,&quot;width&quot;:1446,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Scraping course dynamycally rendered content by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Scraping course dynamycally rendered content by Federico Trotta" title="Scraping course dynamycally rendered content by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!4hxz!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png 424w, https://substackcdn.com/image/fetch/$s_!4hxz!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png 848w, https://substackcdn.com/image/fetch/$s_!4hxz!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png 1272w, https://substackcdn.com/image/fetch/$s_!4hxz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8782693c-b20e-48af-86ee-d8591c7d5cf3_1446x844.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Scraping dynamically rendered content</figcaption></figure></div><p>As you can see, the CSS selector is nested in a structure that seems ordinary. No signs of script tags. In such cases, before rushing into code, use your browser&#8217;s developer tools or site settings to disable JavaScript, then reload the page. The setting&#8217;s location varies, but the objective is the same: inspect what the server provides without page scripts executing. In Chrome, open the settings, type &#8220;JavaScript,&#8221; and click on <strong>Disable JavaScript</strong>:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!RGym!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RGym!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png 424w, https://substackcdn.com/image/fetch/$s_!RGym!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png 848w, https://substackcdn.com/image/fetch/$s_!RGym!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png 1272w, https://substackcdn.com/image/fetch/$s_!RGym!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!RGym!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png" width="1456" height="653" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:653,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Disabling JavaScript in Chrome by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Disabling JavaScript in Chrome by Federico Trotta" title="Disabling JavaScript in Chrome by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!RGym!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png 424w, https://substackcdn.com/image/fetch/$s_!RGym!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png 848w, https://substackcdn.com/image/fetch/$s_!RGym!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png 1272w, https://substackcdn.com/image/fetch/$s_!RGym!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe1c6028a-7bb6-4a07-9709-5c66266322b7_1700x763.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Disabling JavaScript in Chrome</figcaption></figure></div><p>After reloading the page, you&#8217;ll see that content needs to be rendered:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9eEo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9eEo!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png 424w, https://substackcdn.com/image/fetch/$s_!9eEo!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png 848w, https://substackcdn.com/image/fetch/$s_!9eEo!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png 1272w, https://substackcdn.com/image/fetch/$s_!9eEo!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9eEo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png" width="1421" height="628" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:628,&quot;width&quot;:1421,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Page content with JavaScript renderind disabled by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Page content with JavaScript renderind disabled by Federico Trotta" title="Page content with JavaScript renderind disabled by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!9eEo!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png 424w, https://substackcdn.com/image/fetch/$s_!9eEo!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png 848w, https://substackcdn.com/image/fetch/$s_!9eEo!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png 1272w, https://substackcdn.com/image/fetch/$s_!9eEo!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4fdb206a-6cef-4c2b-b1f4-c2925007a819_1421x628.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Page content with JavaScript rendering disabled</figcaption></figure></div><h3>Read the Network Tab</h3><p>There are pages in which the content is dynamically loaded via APIs, and there are two cases:</p><ul><li><p>APIs are external to the target website.</p></li><li><p>APIs are internal to the website (or are encrypted).</p></li></ul><p>This distinction is useful because, in the case of external APIs, you can try to intercept them and get the content from there - and in some cases that is the easier path. Consider the following example from a <a href="https://www.immoscout24.ch/">Swiss real estate website</a>:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sHTl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sHTl!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png 424w, https://substackcdn.com/image/fetch/$s_!sHTl!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png 848w, https://substackcdn.com/image/fetch/$s_!sHTl!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png 1272w, https://substackcdn.com/image/fetch/$s_!sHTl!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sHTl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png" width="1456" height="881" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:881,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Chrome DevTools showing the script tag with the initial state JSON on the target page by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Chrome DevTools showing the script tag with the initial state JSON on the target page by Federico Trotta" title="Chrome DevTools showing the script tag with the initial state JSON on the target page by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!sHTl!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png 424w, https://substackcdn.com/image/fetch/$s_!sHTl!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png 848w, https://substackcdn.com/image/fetch/$s_!sHTl!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png 1272w, https://substackcdn.com/image/fetch/$s_!sHTl!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b776e9-0995-4320-b23e-0f63aeedf770_1500x908.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Chrome DevTools showing the script tag with the initial state JSON on the target page</figcaption></figure></div><p>As you can see, the content is rendered via a script. Now, if you go to the <strong>Network</strong> tab, you can see the API that is called under the hood to render the content:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5qkY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5qkY!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png 424w, https://substackcdn.com/image/fetch/$s_!5qkY!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png 848w, https://substackcdn.com/image/fetch/$s_!5qkY!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png 1272w, https://substackcdn.com/image/fetch/$s_!5qkY!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5qkY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png" width="1456" height="769" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:769,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Chrome DevTools Network tab filtered to Fetch/XHR showing background API calls on the target website by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Chrome DevTools Network tab filtered to Fetch/XHR showing background API calls on the target website by Federico Trotta" title="Chrome DevTools Network tab filtered to Fetch/XHR showing background API calls on the target website by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!5qkY!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png 424w, https://substackcdn.com/image/fetch/$s_!5qkY!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png 848w, https://substackcdn.com/image/fetch/$s_!5qkY!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png 1272w, https://substackcdn.com/image/fetch/$s_!5qkY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1d4129a1-b79c-47ca-b4f8-348ea5175099_1500x792.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Chrome DevTools Network tab filtered to Fetch/XHR showing background API calls on the target website</figcaption></figure></div><p>In this case, the rendering APIs are internal, so you can&#8217;t reverse-engineer them to extract the content. However, the inspection process stays the same, and it&#8217;s valid.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://substack.thewebscraping.club/p/consulting&quot;,&quot;text&quot;:&quot;Need help with your scraping project?&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://substack.thewebscraping.club/p/consulting"><span>Need help with your scraping project?</span></a></p><div><hr></div><h2>Choosing the Headless Browser That Fits Your Needs</h2><p>Once the diagnostic process above confirms that your target renders its content client-side, the next decision is which automation tool drives that rendering. The three mainly established options are Puppeteer, Playwright, and Selenium.</p><p>They overlap in what they can do, but they differ in protocol, language support, browser coverage, and how much resilience they build in for you. One thing holds for all three, though, and it echoes the caveat from the bot-defense section: running a real browser engine does not automatically defeat fingerprinting or a CAPTCHA. All three set <em>navigator.webdriver</em> to <em>true</em> by default, and headless builds can leak engine-specific tells such as a <em>HeadlessChrome</em> user-agent token, depending on the engine and the version you pin.</p><p>Tool choice affects your maintainability and coverage in two ways: what the framework does for you, and how well you can keep it disguised. Both matter, and the second one is covered separately.</p><h3>Puppeteer</h3><p><a href="https://pptr.dev/">Puppeteer</a> is a Node.js library maintained by Google that controls <a href="https://www.scraping.club/p/webdriver-vs-cdp-vs-bidi">Chrome through the Chrome DevTools Protocol (CDP), the same low-level channel DevTools itself uses</a>. Because it speaks the browser&#8217;s native protocol rather than a generic abstraction, it feels less like an external driver and more like a scripted extension of the browser.</p><p>Since version 23, Puppeteer also drives <a href="https://pptr.dev/webdriver-bidi">Firefox stably, over WebDriver BiDi</a>, the protocol the browser vendors and the Selenium project designed together. BiDi is now what Puppeteer uses by default when you launch Firefox, while Chrome still defaults to CDP. Treat Chromium as the primary path regardless: the full feature surface still lives in CDP, and CDP-only capabilities such as coverage, tracing, accessibility emulation, screencasts, most network and CPU emulation, and service-worker bypass throw <em>UnsupportedOperation</em> when you are on the BiDi path.</p><p><strong>Main features</strong>:</p><ul><li><p>Automatic download of a known-compatible browser build, so you rarely fight version mismatches, and a reproducible Chrome for Testing version across machines and CI.</p></li><li><p>A concise high-level API covering navigation, clicking, typing, and <em>waitForSelector</em>-style waits.</p></li><li><p>A <a href="https://pptr.dev/api/puppeteer.locator">Locator API</a> that checks actionability before acting. Locators retry the entire operation until the element is inside the viewport, visible, enabled, and its bounding box is stable across two consecutive animation frames. The documentation recommends locators as the standard way to select and interact with elements.</p></li><li><p>Native network interception over CDP, letting you block, modify, or mock requests and responses in flight.</p></li><li><p>Access to tracing, coverage, and performance metrics for profiling.</p></li><li><p>Incognito browser contexts through <em>browser.createBrowserContext()</em>, for isolated sessions inside a single browser process.</p></li></ul><p><strong>Pros</strong>:</p><ul><li><p>Speed and depth of control on Chrome, with a raw CDP escape hatch for anything the high-level API does not cover.</p></li><li><p>Locators remove most timing flake without you hand-rolling waits, because actionability is checked for you on every retry.</p></li><li><p>The API is compact enough to learn in an afternoon.</p></li><li><p>It is the de facto standard for Chrome-focused scraping and rendering pipelines, so documentation and community solutions are abundant.</p></li></ul><p><strong>Cons</strong>:</p><ul><li><p>Its auto-waiting stops short of Playwright&#8217;s. Locators do not perform a hit-target check, so an element hidden behind an overlay still receives the click, and there is no strict-mode resolution when a selector matches several elements. The legacy <em>page.click(selector)</em> path also waits only for existence, for backward compatibility. On modern single-page applications you will still write explicit waits in places.</p></li><li><p>Chrome and Firefox only: there is no WebKit engine, so nothing Safari-shaped.</p></li><li><p>Browser coverage is real but lopsided, since the Firefox path exposes a smaller API than the Chromium one.</p></li><li><p>Node.js-first orientation. Alternatives in other languages are unofficial ports.</p></li><li><p>It has shifted across major versions in ways that occasionally break existing scripts, including the headless-mode rename and the protocol default for Firefox.</p></li></ul><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://github.com/TheWebScrapingClub/scraping-wiki&quot;,&quot;text&quot;:&quot;Find more in the wiki&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://github.com/TheWebScrapingClub/scraping-wiki"><span>Find more in the wiki</span></a></p><div><hr></div><h3>Playwright</h3><p><a href="https://playwright.dev/">Playwright</a> is a Microsoft-developed automation framework built by much of the team behind Puppeteer, designed from the start to drive Chromium, Firefox, and WebKit through a single unified API. It ships official bindings for Node.js, Python, Java, and .NET, which makes it the most accessible of the three if your stack isn&#8217;t JavaScript.</p><p>One scope detail worth stating plainly: Playwright ships its own patched builds of Firefox and WebKit. WebKit behaves like Safari but is not Safari, and Firefox is not the Firefox you have installed. If you need the vendor browser, Playwright lets you switch to it with <em>channel</em> (<em>chrome</em>, <em>msedge</em>, <em>chrome-beta</em>, <em>chrome-dev</em>, <em>chrome-canary</em>) or <em>executablePath</em>, but that applies to the Chromium family only. It cannot drive a system Firefox or a genuine Safari installation.</p><p><strong>Main features:</strong></p><ul><li><p>Automatic actionability waiting. Clicks and assertions wait until the element is actionable, including a hit-target check that confirms the element would actually receive the event rather than being covered by something else.</p></li><li><p>Strict locators. A selector that resolves to more than one element throws instead of silently acting on the first match, which surfaces bad selectors early rather than scraping the wrong row.</p></li><li><p>Network interception and response mocking through <em>page.route</em>, plus a plain HTTP request context for endpoints you do not need to render.</p></li><li><p>HAR recording, both from the command line with <em>&#8211;save-har</em> and through the API, and the ability to route a run from a previously saved HAR. This gives you every request a page made, not just the clicks you performed.</p></li><li><p>Browser contexts for isolated sessions, each with its own cookies, storage, and proxy settings, all inside one browser process.</p></li><li><p>A built-in codegen recorder, a trace viewer for replaying failed runs, video recording, and mobile device emulation.</p></li><li><p>Remote attachment through <em>connect()</em> and <em>connectOverCDP()</em>, so you can drive browsers running elsewhere.</p></li></ul><p><strong>Pros</strong>:</p><ul><li><p>Cross-browser coverage from one API.</p></li><li><p>Dramatically fewer flaky runs thanks to actionability waiting and strictness.</p></li><li><p>Cheap parallelism through contexts, since many sessions share one browser process instead of each paying for a full instance.</p></li><li><p>A debugging experience that is currently the best in the category.</p></li><li><p>Very active development.</p></li><li><p>CDP remains reachable on Chromium through <em>context.newCDPSession(page)</em>, so you are not locked out of Chrome-specific tooling by choosing the abstraction.</p></li></ul><p><strong>Cons</strong>:</p><ul><li><p>A larger installation footprint, since it typically downloads three browser binaries.</p></li><li><p>A steeper learning curve than Puppeteer&#8217;s straightforward page object, and the real costs sit at library level rather than in the test runner: locator strictness, frame locators, <em>storageState</em> and context lifecycle, and in Python the choice between the synchronous and asynchronous APIs. That last one is a concurrency decision, because the sync API cannot run inside a running asyncio loop.</p></li><li><p>Newer capabilities sometimes land in the Node binding before reaching the other languages.</p></li><li><p>Browser builds are Playwright&#8217;s choice, not yours, so version pinning follows its release cadence.</p></li><li><p>Like the others, it remains recognizable as an automated browser without additional masking.</p></li></ul><h3>Selenium</h3><p><a href="https://www.selenium.dev/">Selenium</a> is the oldest and most broadly deployed of the three, and it drives browsers through the standardized W3C WebDriver protocol rather than a vendor-specific one. Bindings exist for Java, Python, C#, Ruby, and JavaScript, and Selenium Grid lets you distribute sessions across machines or remote device farms. Since 4.6, Selenium Manager resolves and caches the correct driver automatically, which removes the driver-version-mismatch work people still associate with the tool.</p><p>It is also no longer a purely classic WebDriver tool. The Selenium project co-created WebDriver BiDi with the browser vendors and is moving its implementation from Classic to BiDi, and the result is visible in the current releases: <a href="https://www.selenium.dev/documentation/webdriver/bidi/">network interception is a first-class BiDi API</a> across all five bindings, with no proxy in the path.</p><p><strong>Main features</strong>:</p><ul><li><p>Standards-based WebDriver classic communication, which means the browser vendor, not a third-party project, keeps the driver in sync.</p></li><li><p>WebDriver BiDi network control: <em>addIntercept</em> with <em>beforeRequestSent</em> and <em>responseStarted</em> phases, then <em>continueRequest</em>, <em>continueResponse</em>, <em>provideResponse</em>, and <em>failRequest</em> to modify, mock, or block traffic, plus <em>authRequired</em> handling for basic authentication. Subscriptions such as <em>beforeRequestSent</em> and <em>responseCompleted</em> let you log everything a page fetched. These are enabled by opting into a <em>webSocketUrl</em>.</p></li><li><p>Grid for parallel and remote execution, and the Remote WebDriver URL for attaching to browsers running elsewhere.</p></li><li><p>Relative locators for expressing element positions in human terms.</p></li><li><p>Partial CDP support for Chromium-specific operations such as network conditions.</p></li><li><p>Selenium IDE, a browser extension for recording and playing back simple flows.</p></li></ul><p><strong>Pros</strong>:</p><ul><li><p>Ecosystem breadth and longevity.</p></li><li><p>Language-agnostic, with official bindings for five languages.</p></li><li><p>Supported by virtually every commercial testing platform and cloud device farm.</p></li><li><p>The largest body of existing answers and tutorials.</p></li><li><p>Standards-based drivers mean less risk of vendor abandonment, and the protocol itself is moving toward BiDi rather than staying static.</p></li><li><p>It can drive the browser you actually have installed, vendor build included, which means it is the only one of the three that can drive real Safari.</p></li></ul><p><strong>Cons</strong>:</p><ul><li><p>No built-in auto-wait. Implicit waits exist but are discouraged, so you must implement explicit waits to avoid timing failures.</p></li><li><p>More verbose than its peers for equivalent tasks.</p></li><li><p>Classic WebDriver round-trips cost more than CDP or BiDi ones. For scraping, this is the smallest of the three levers, because throughput is dominated by page render time, browser memory, and the target&#8217;s own rate limiting rather than by command latency.</p></li><li><p>BiDi is opt-in and still covers roughly 70% of the CDP surface, with no heap profiling or performance traces. The documentation itself describes the migration as a work in progress, and maturity varies from binding to binding, so this needs testing on your stack before you rely on it.</p></li><li><p>Each session is heavier than a Playwright context, which raises the cost of large-scale parallel scraping.</p></li></ul><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>Disguise Is a Separate Layer</h2><p>The caveat about invisibility is worth expanding, because in scraping it is where most of the ongoing maintenance actually goes. None of the three frameworks ships usable anti-detection on its own, and each has its own ecosystem of masking layers, maintained by different projects on different schedules: <em>puppeteer-extra-plugin-stealth</em> and rebrowser-patches on the Puppeteer side, patchright and rebrowser-playwright on the Playwright side, and Camoufox, a hardened Firefox build driven through the Playwright Python API, alongside <em>undetected-chromedriver</em> and nodriver in the Python Chrome-driven world.</p><p>This means the practical question is not only &#8220;which framework fits my workload&#8221; but &#8220;which masking layer for that framework can I keep current against this target&#8217;s fingerprinting checks&#8221;. It is a legitimate selection factor; it interacts with your language binding, and it is easy to underestimate until the first wave of <em>403</em>s. Pair it with proxy management: Playwright supports per-context proxies, which is what makes proxy pooling feasible alongside parallelism, whereas the others generally require one proxy per browser instance or per-process configuration.</p><h3>Which Tool to Choose: A Scenario-by-Scenario Comparison</h3><p>The tools above can all render the pages described in this article. What actually decides the choice is the shape of your scraping workload. In other words:</p><ul><li><p>Whether you stay inside one browser engine.</p></li><li><p>How many sessions run at once.</p></li><li><p>Which language your pipeline is written in.</p></li><li><p>How much interaction the target demands.</p></li><li><p>Whether you are integrating with infrastructure that already exists.</p></li><li><p>How aggressively the target fingerprints automated browsers.</p></li></ul><p>The table below maps those situations to a recommendation. Where a scenario is genuinely tool-neutral, it is marked as such, because paying for cross-browser coverage you will never use is a cost you should not take on. Where a scenario is really a testing requirement rather than a scraping one, that is stated too, since the two goals point at different tools:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!2E9n!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!2E9n!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png 424w, https://substackcdn.com/image/fetch/$s_!2E9n!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png 848w, https://substackcdn.com/image/fetch/$s_!2E9n!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png 1272w, https://substackcdn.com/image/fetch/$s_!2E9n!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!2E9n!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png" width="1148" height="790" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:790,&quot;width&quot;:1148,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:136791,&quot;alt&quot;:&quot;The comparison table by federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/219545817?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The comparison table by federico Trotta" title="The comparison table by federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!2E9n!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png 424w, https://substackcdn.com/image/fetch/$s_!2E9n!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png 848w, https://substackcdn.com/image/fetch/$s_!2E9n!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png 1272w, https://substackcdn.com/image/fetch/$s_!2E9n!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F059c5463-1cf2-4c04-af12-3ec9a835c5e7_1148x790.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">The comparison table</figcaption></figure></div><h2>Conclusion</h2><p>In this article, you learned that deciding whether a page needs a headless browser is a diagnostic matter. You saw how a browser differs from a plain HTTP request, which symptoms suggest JavaScript rendering is involved, how to confirm those symptoms with a few quick tests, and how to pick the right tool once rendering turns out to be unavoidable.</p><p>The value is in the order you do those things. An empty extraction result doesn&#8217;t tell you much if you can&#8217;t tell whether the data was never sent, was already embedded in a script tag, or was sitting behind an interaction or an access check. But a process that takes you from symptom to confirmed cause before you pay for rendering? That&#8217;s where the real value is! And the same reasoning carries into the tool choice, because a real browser engine buys you coverage and maintainability, never invisibility. So the decisions that actually matter are which framework fits the shape of your workload, and which masking layer for it you can keep current against the target&#8217;s checks.</p><p>So, let us know: What&#8217;s your process to decide whether a page needs JavaScript rendering? And which one of Puppeteer, Playwright, and Selenium ended up in your stack, and what made you pick it? Let&#8217;s discuss in the comments!</p><div><hr></div><p><em>Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/p/scraping-javascript-rendered-pages?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.scraping.club/p/scraping-javascript-rendered-pages?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[The LAB #117: Scraping club data and results from Fussball.de]]></title><description><![CDATA[23,000 Club Websites and the Font That Hides the Scores]]></description><link>https://www.scraping.club/p/scraping-club-data-fussball-de</link><guid isPermaLink="false">https://www.scraping.club/p/scraping-club-data-fussball-de</guid><dc:creator><![CDATA[Pierluigi Vinciguerra]]></dc:creator><pubDate>Thu, 22 Oct 2026 21:41:40 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/0c14ac2b-906f-4912-be2c-2c97a99e8ffe_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Having a look at The Scraping Club's Google Search Console, I&#8217;ve seen hundreds of searches for how to scrape club data from fussball.de. Of course, given the name of this newsletter, it was ranking pretty high in the results, even if not particularly relatable, but this number of requests made me curious, and I want to dig deeper into this website.</p><div><hr></div><div class="callout-block" data-callout="true"><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/B5JPeJ">Decodo&#8217;s Web Data API</a></strong> turns any site into clean, structured data your models can actually use.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/B5JPeJ" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png" width="716" height="286.4" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:280,&quot;width&quot;:700,&quot;resizeWidth&quot;:716,&quot;bytes&quot;:105634,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;text&quot;:&quot;Activate your free plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/B5JPeJ"><span>Activate your free plan</span></a></p></div><div><hr></div><p>To be honest, when I started, I expected a hard target. If everyone in Germany is looking for a solution to scrape it, it should be difficult to do. </p><p>Well, it wasn&#8217;t, at least for the club directory: there&#8217;s no anti-bot at all, and the data is available in plain HTML from endpoints anyone can call. But the interesting part is somewhere else, on match dates and scores. They are hidden behind a custom web font, and the HTML you download contains characters that don&#8217;t have any meaning until the browser draws them. </p><p>So the article has two parts: in the first one, we&#8217;ll scrape the club directory, which is what people search for. In the second one, we&#8217;ll see how to get the results, where we decode the font, first the easy way and then in a way that keeps working if the site removes the shortcut we use today.</p><p>All the code is in <a href="https://github.com/TheWebScrapingClub/thelab">The Lab repository</a>, in the <code>117.fussball_de_scraper</code> directory.</p><h2>How fussball.de is built</h2><p>If you want data about German amateur football, there&#8217;s basically one place where you can find it: fussball.de, the portal of the German Football Association (DFB). Every registered club, from the Bundesliga down to the last Kreisliga team, has a profile there, and every league has its fixtures and results there.</p><p>So, before writing any code, I mapped the layers between my client and the data.</p><p>The first layer is the delivery. Every response carries <code>x-varnish</code> and <code>x-azure-ref</code> headers, so pages come from a Varnish cache behind Azure Front Door. I looked for the usual signs of an anti-bot (challenge scripts in the HTML, a 403 on the first request, cookies set by the server) and found none. The homepage, a club profile and the search endpoint don&#8217;t send a single <code>Set-Cookie</code> header. Even Wappalizer said there was nothing installed so I tried a basic test: I fetched a club page with <code>python-requests</code> and its default user agent (<code>python-requests/2.33.0</code>, no custom headers): 200, full content. </p><p>The scraper we&#8217;ll create later uses a Chrome user agent and a German <code>Accept-Language</code> header (see <code>http_client.py</code>), and with those it ran at 48 requests per second without a block. </p><div><hr></div><div class="callout-block" data-callout="true"><p><em>Tired of getting blocked while scraping the web?<br><strong><a href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club">ScrapingBee</a></strong> handles proxies, browsers, anti-bot systems, and retries so you can focus on your data. Get clean Markdown, JSON, or HTML from the web with up to a 99.9% success rate. ScrapingBee is SOC 2 Type II and GDPR compliant, and trusted by 4,000+ developers.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 424w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 848w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png" width="1456" height="437" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:437,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:128210,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/216891176?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 424w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 848w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club&quot;,&quot;text&quot;:&quot;Discover ScrapingBee&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club"><span>Discover ScrapingBee</span></a></p></div><div><hr></div><p>The second layer is how the data is delivered. The front end is an AngularJS application (all those <code>data-ng-*</code> attributes), but almost everything is rendered on the server. League tables, club profiles and matchday pages arrive as complete HTML. A few blocks are loaded later from endpoints with an <code>ajax.</code> prefix (<code>ajax.club.prev.games</code>, <code>ajax.actual.table</code>, <code>ajax.search.club.loadmore</code>), and they return HTML fragments, sometimes wrapped in JSON.</p><p>The third layer is the obfuscation, and it&#8217;s the only real obstacle; it covers dates, kick-off times, scores, and referee names in fixtures. </p><h2>Part 1: the club directory</h2><h3>Finding the search endpoint</h3><p>There&#8217;s no public list of clubs, but there&#8217;s a club search at <code>/suche.verein</code>, with two fields: club name and postal code (PLZ) or city. My first try was the obvious one, <code>/suche.verein?plz=84428</code>, and it returned &#8220;Keine Ergebnisse gefunden&#8221; (no results). The form has a <code>data-rest-form</code> attribute, so I opened the site&#8217;s JavaScript bundle (<code>script.js</code>, 2.3 MB) and found the directive behind it:</p><pre><code><code>function onFormSubmit(event){
    // Don't submit the form
    event.preventDefault();
    // Get prepared URL with REST data
    var url = AjaxDataService.getDataUrl(attributes.action, collectFormData(), true);
    // Do the actual request
    requestGET(url);
}</code></code></pre><p>The form fields become path segments, the same <code>/-/key/value</code> style that every fussball.de URL uses. So the URL the site really sends is <code>/suche.verein/-/plz/84428</code>, and it returned two clubs. </p><p>Side note: later in the day the query string version returned the same two clubs too, and I can&#8217;t say why it didn&#8217;t work the first time. I stick with the path form, because it&#8217;s what the site&#8217;s own form sends.</p><p>With a city it&#8217;s more interesting: <code>/suche.verein/-/plz/M&#252;nchen</code> says &#8220;167 Treffer&#8221; (167 results) but shows only 20, with a &#8220;Mehr laden&#8221; (load more) button. The button is a form, and its attributes contain the real endpoint:</p><pre><code><code>&lt;form data-ajax-type="json" data-ajax-target="#clublist ul" data-ajax-defaults="{'max':20,'offset':20}"
      data-ajax-resource="https://www.fussball.de/ajax.search.club.loadmore/-/mime-type/JSON/or/false/plz/M%C3%BCnchen" ...&gt;</code></code></pre><p>I called it directly. It returns JSON with an <code>html</code> field (the list items), a <code>lastIndex</code> and a <code>final</code> flag. The page size is a parameter, and the server accepts bigger values than 20: with <code>max/500/offset/0</code> I got all 167 clubs of M&#252;nchen in one call, with <code>final: true</code>.</p><div><hr></div><div class="callout-block" data-callout="true"><p><em><strong><a href="https://dataimpulse.com/">DataImpulse</a></strong> is trusted by teams running ad verification, web scraping, SERP tracking, and market research. Ethically sourced proxies, globally accessible, and fairly priced.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://dataimpulse.com/" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9eCD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!9eCD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!9eCD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!9eCD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9eCD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png" width="642" height="361.125" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:576,&quot;width&quot;:1024,&quot;resizeWidth&quot;:642,&quot;bytes&quot;:534517,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://dataimpulse.com/&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/221688546?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!9eCD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png 424w, https://substackcdn.com/image/fetch/$s_!9eCD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png 848w, https://substackcdn.com/image/fetch/$s_!9eCD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png 1272w, https://substackcdn.com/image/fetch/$s_!9eCD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff8aecd85-b842-405b-ba53-912a678c44e9_1024x576.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://dataimpulse.com/&quot;,&quot;text&quot;:&quot;Get Started With DataImpulse&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://dataimpulse.com/"><span>Get Started With DataImpulse</span></a></p></div><div><hr></div><h3>Covering all of Germany</h3><p>The next question is how to list every club, since there&#8217;s no &#8220;all clubs&#8221; query. I tested what the <code>plz</code> parameter accepts:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!PJfE!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!PJfE!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png 424w, https://substackcdn.com/image/fetch/$s_!PJfE!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png 848w, https://substackcdn.com/image/fetch/$s_!PJfE!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png 1272w, https://substackcdn.com/image/fetch/$s_!PJfE!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!PJfE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png" width="808" height="246" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:246,&quot;width&quot;:808,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:16966,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/221688546?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!PJfE!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png 424w, https://substackcdn.com/image/fetch/$s_!PJfE!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png 848w, https://substackcdn.com/image/fetch/$s_!PJfE!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png 1272w, https://substackcdn.com/image/fetch/$s_!PJfE!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F16af5ddd-f786-4100-ba92-2b5076894a02_808x246.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>So one or two digits return nothing, and three digits work. Then I looked at the 103 clubs for <code>844</code>, and they weren&#8217;t all in the 844xx area. There was 1.FC Wolfsburg (38440), ASG Vorw&#228;rts Dessau (06844) and Brinkumer SV in Weyhe (28844). T</p><p>he match is on any part of the postal code, not only on the start. And this is great news for full coverage: every five-digit code contains its own first three digits, so querying every combination from 000 to 999 reaches every club. The cost is that each club comes back a few times, and you deduplicate by club ID.</p><p>This is the core of <code>clubs.py</code>:</p><pre><code><code>SEARCH = BASE + "/ajax.search.club.loadmore/-/mime-type/JSON/or/false/plz/{plz}/max/5000/offset/0"
CLUB_URL = re.compile(r"https://www\.fussball\.de/verein/(?P&lt;slug&gt;[^/]+)/-/id/(?P&lt;id&gt;[\w-]+)")


def search_prefix(prefix):
    r, _ = get(SEARCH.format(plz=prefix))
    data = r.json()
    sel = Selector(data["html"])
    clubs = []
    for li in sel.css("li"):
        href = li.css("a::attr(href)").get("")
        m = CLUB_URL.match(href)
        if m:
            name = " ".join(t.strip() for t in li.css("p.name::text").getall()).strip()
            clubs.append({"club_id": m["id"], "slug": m["slug"], "name": name, "url": href})
    return prefix, clubs, data.get("final")</code></code></pre><p>And this is the output of <code>clubs.py directory</code> with 4 workers:</p><pre><code><code>prefixes queried: 1000 in 20.1s (4 workers)
prefixes with results: 1000, largest: 552 with 208 clubs
prefixes where the endpoint said final=false: []
search hits: 69503, unique clubs: 23220
HTTP status counts: {200: 1000}</code></code></pre><p>So 1,000 requests and 20 seconds are enough to list 23,220 unique clubs. The largest prefix returned 208 clubs, well under the <code>max/5000</code> I send, and no response had <code>final: false</code>, so no list was cut.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://github.com/TheWebScrapingClub/scraping-wiki&quot;,&quot;text&quot;:&quot;Find more in the wiki&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://github.com/TheWebScrapingClub/scraping-wiki"><span>Find more in the wiki</span></a></p><div><hr></div><h3>Reading the club profiles</h3><p>Each club has a profile at <code>/verein/{slug}/-/id/{id}</code>. The data we want is in a block with id <code>clubProfile</code>, as label and value pairs:</p><pre><code><code>&lt;div class="column-left"&gt;
    &lt;div class="value"&gt;
        &lt;a href="http://www.tsv-buchbach.de/fussball" target="_blank"&gt;www.tsv-buchbach.de/fussball&lt;/a&gt;
    &lt;/div&gt;
    &lt;div class="label"&gt;Website&lt;/div&gt;
&lt;/div&gt;</code></code></pre><p>The labels are <code>Gr&#252;ndungsjahr</code> (founding year), <code>Vereinsfarben</code> (club colors), <code>Adresse</code> and <code>Website</code>. Empty fields contain only a <code>space</code>. I read each block by its label, and I take the link <code>href</code> when there is one, so the website comes out as a full URL:</p><pre><code><code>FIELDS = {
    "Gr&#252;ndungsjahr": "founded",
    "Vereinsfarben": "colours",
    "Adresse": "address",
    "Website": "website",
}


def parse_profile(html_text):
    sel = Selector(html_text)
    row = {v: "" for v in FIELDS.values()}
    for block in sel.css("#clubProfile .factfile-data .row &gt; div"):
        label = block.css(".label::text").get("").strip()
        if label not in FIELDS:
            continue
        href = block.css(".value a::attr(href)").get()
        text = " ".join(t.strip() for t in block.css(".value ::text").getall() if t.strip())
        row[FIELDS[label]] = (href or text).strip()
    row["name"] = sel.css("#clubProfile .factfile-headline .headline::text").get("").strip()
    row["association"] = sel.css("#clubProfile .factfile-headline .subline a::text").get("").strip()
    return row</code></code></pre><p>Before the full run, I tested 2,000 random clubs with 8 workers. I wanted to see where the rate limit is:</p><pre><code><code>profiles fetched: 2000 in 41.7s (8 workers, 48.0 req/s)
HTTP status counts (including retries): {200: 2000}
latency median 0.138s, p95 0.220s, max 0.515s
  founded   1871/2000 (94%)
  colours   1552/2000 (78%)
  address   1616/2000 (81%)
  website   1497/2000 (75%)</code></code></pre><p>At 48 requests per second I didn&#8217;t find it: 2,000 responses, all 200, and the latency didn&#8217;t grow. Then I ran all 23,220 clubs with 4 workers (I don&#8217;t need more speed than that, and there&#8217;s no reason to load a public site more than necessary):</p><pre><code><code>profiles fetched: 23220 in 797.8s (4 workers, 29.1 req/s)
HTTP status counts (including retries): {200: 23220}
latency median 0.121s, p95 0.196s, max 0.643s
  founded  21732/23220 (94%)
  colours  18041/23220 (78%)
  address  18831/23220 (81%)
  website  17422/23220 (75%)</code></code></pre><p>It took about 13 minutes to read every club registered on fussball.de, without a single error or retry. </p><div><hr></div><div class="callout-block" data-callout="true"><p><em>&#8220;Data is the new oil.&#8221;<br>You&#8217;ve heard it a hundred times, but do you know who does the drilling?<br>With AI, it&#8217;s more true than ever. Models are trained on data scraped from the web. Agents need browser automation that doesn&#8217;t fall over at the first CAPTCHA. But who&#8217;s doing the heavy lifting? Proxies that actually work.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://luma.com/5c4v7zjk" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!6re3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 424w, https://substackcdn.com/image/fetch/$s_!6re3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 848w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1272w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!6re3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png" width="318" height="397.3582887700535" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1402,&quot;width&quot;:1122,&quot;resizeWidth&quot;:318,&quot;bytes&quot;:1471113,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://luma.com/5c4v7zjk&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/221688546?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!6re3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 424w, https://substackcdn.com/image/fetch/$s_!6re3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 848w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1272w, https://substackcdn.com/image/fetch/$s_!6re3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe6933c15-9bbc-42eb-9024-2d69afd87847_1122x1402.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><em>That&#8217;s why we partnered with <strong><a href="https://www.linkedin.com/company/geonode/">Geonode</a></strong> for an aperitivo on 5 November in Milan.<br>No talks to sit through but just people who build scrapers, data pipelines and AI agents for a living, and a drink.<br><br>If you&#8217;re one of them, here&#8217;s the link to join: <strong><a href="https://www.linkedin.com/safety/go/?url=https%3A%2F%2Fluma%2Ecom%2F5c4v7zjk&amp;urlhash=TG6m&amp;mt=_MVhK0tQUX3JJQvNRPrQeq4u7WW0fNOSWRfZa99BOWYnJowsTzf1EnbBN_-HBrEpzjxD1u-cfdNwXHw28VhbwRGPBXuXqZ7OSb9qW7AhN0Eryqhpw3SR_IT3&amp;isSdui=true">https://luma.com/5c4v7zjk</a></strong></em></p></div><div><hr></div><p>So the answer to the search that started this article is 17,422 club websites, in a CSV with the club name, the regional association, the address and the founding year. </p><p>I also checked that the empty fields are really empty. 23,218 of 23,220 pages have the <code>clubProfile</code> block with its name and association, so a missing website there means the club didn&#8217;t enter one (the value is just a <code>space</code>). The other two pages answered 200 without the block. When I opened them again later, SG Fulde had its profile and FC Wesuwe still didn&#8217;t, so a retry on &#8220;200 but no profile&#8221; is worth adding if you need every last row.</p><p>One detail if you plan to use this data: the same club can be registered in two regional associations, and then it has two profiles with two IDs. For example, Brinkumer SV appears as <code>brinkumer-sv-bremen</code> and <code>brinkumer-sv-niedersachsen</code>, with the same address. If you need one row per club, deduplicate on website or address after the scrape.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><h2>Part 2: the results</h2><h3>What the HTML contains</h3><p>The results are where fussball.de puts its only protection. This is the score cell of FV Illertissen vs TSV Buchbach, as it arrives from the server:</p>
      <p>
          <a href="https://www.scraping.club/p/scraping-club-data-fussball-de">
              Read more
          </a>
      </p>
   ]]></content:encoded></item><item><title><![CDATA[Building a price comparison agent with the Oxylabs Web API]]></title><description><![CDATA[Let&#8217;s discover the new Oxylabs Web API and its features]]></description><link>https://www.scraping.club/p/building-a-price-comparison-oxylabs-web-api</link><guid isPermaLink="false">https://www.scraping.club/p/building-a-price-comparison-oxylabs-web-api</guid><dc:creator><![CDATA[Pierluigi Vinciguerra]]></dc:creator><pubDate>Tue, 20 Oct 2026 09:07:59 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/01bbe3bd-1522-414c-ad72-91003c75dd7d_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Every one of us who has tried to build an AI agent that fetches live data from the web has seen the limitations of the standard fetch tool included in the harness. You get blocked by anti-bot, the page is so big that your context is getting bloated by useless HTML, and so on.  </p><p>Today, for this issue of &#8220;Tool of the Week&#8221;, we&#8217;re testing the <a href="https://oxylabs.go2cloud.org/aff_c?offer_id=7&amp;aff_id=665&amp;url_id=222">Oxylabs Web API</a>, a new product from Oxylabs that helps in building reliable agents, and we&#8217;ll see it live by building together a live price comparison agent in Claude.</p><h2>What the Oxylabs Web API is</h2><p>The Web API is composed of two endpoints, each with a different task, as you can see <a href="https://developers.oxylabs.io/get-started/quick-start-web-api?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">from the quick start guide</a>:</p><ul><li><p><code>POST /v1/search</code> takes a query and returns ranked results (title, URL, a few truncated snippets called <code>outline</code>, plus related searches).</p></li><li><p><code>POST /v1/scrape</code> takes a URL and returns the page as HTML, Markdown, JSON or a screenshot.</p></li></ul><p>On top of the generic scraper there&#8217;s a long list of dedicated scrapers, under <code>/v1/scrape/{target}/{collection}</code>, for sites like Amazon, Google and Best Buy, where Oxylabs maintains the parser for you. </p><p>We also have <a href="https://developers.oxylabs.io/ai-workflows/mcp">an MCP integration</a> and the relative <a href="https://developers.oxylabs.io/ai-workflows/agent-skills">SKILL for the agent</a> so that it knows what to call and how when you need live web data.</p><p>Additionally, there&#8217;s a page in the docs written for coding agents, <a href="https://developers.oxylabs.io/products/web-api/for-agents.md?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">For Agents</a>, that contains all the instructions in one Markdown file: both endpoint schemas, a working Python client, the error rules, and a checklist. </p><p>In my opinion, this is the most interesting part of the documentation because it shows who Oxylabs believes the customer is. Every docs page also has a <code>.md</code> version, and you can append <code>?ask=&lt;question&gt;&amp;goal=&lt;goal&gt;</code> to get an answer generated from the docs. </p><p>But let&#8217;s see the main features together.</p><h3>Markdown instead of HTML</h3><p>As every API thought for agents, also Oxylabs Web API can return both Markdown and plain HTML files. Markdowns are, of course, the preferred option if you&#8217;re working with LLMs. On the Best Buy product page of the Sony WH-1000XM5 (the headphones I used for all the tests), the HTML returned by <code>/v1/scrape</code> is 1,234,158 characters. The Markdown of the same page is 10,318 characters, about 120 times smaller. Handling such a large HTML page will cost tokens and bloat all the context you&#8217;re working on.</p><h3>JavaScript rendering with run_js</h3><p>Setting <code>run_js: true</code> renders the page in a real browser before the results. On the same Best Buy page, the Markdown grew from 10,318 to 91,368 characters and contained the $228 price, which was not available in the plain HTML page.  The cost is time: 2.3 seconds without rendering, 40.7 seconds with it. </p><h3>Geolocation</h3><p>Both endpoints accept a <code>location</code> parameter with a two-letter country code. On <code>/search</code> it changes the results in the way you&#8217;d expect for a shopping query. Searching &#8220;sony wh-1000xm5 price&#8221; with <code>US</code> returned Amazon, camelcamelcamel and sony.com, while with <code>DE</code> it returned idealo.de, billiger.de and geizhals.de. Using<code>IT</code> , we got idealo.it, sony.it and trovaprezzi.it.</p><h3>AI parsing</h3><p>AI parsing extracts JSON from any page in the same request that fetches it. You set <code>output: ["json"]</code> and pass either a natural language <code>prompt</code> or an OpenAPI <code>schema</code>. The docs say the schema mode &#8220;guarantees the response matches your defined types, nested structures, and required fields&#8221;, and recommend it for production.</p><p>I wanted to see how stable it is, so I ran both modes three times on the Best Buy page, with and without JavaScript. This is the schema I used:</p><pre><code><code>{
  "type": "object",
  "properties": {
    "product_name": {"type": "string"},
    "price": {"type": "number", "description": "Current price the buyer pays, as a number"},
    "currency": {"type": "string", "description": "ISO 4217 code"},
    "availability": {"type": "string", "enum": ["in_stock", "out_of_stock", "preorder", "unknown"]},
    "seller": {"type": "string", "nullable": true}
  },
  "required": ["product_name", "price", "currency", "availability"]
}</code></code></pre><p>With <code>run_js</code>, the schema mode returned the complete object in 3 runs out of 3, including <code>seller: "Best Buy"</code> and <code>availability: "in_stock"</code>. </p><p>Without JavaScript, it returned <code>price: null</code> in 2 runs out of 3, even if <code>price</code> is marked as required. So the schema guarantees the shape of the output, not that every field is filled, and your code still needs to check for nulls. </p><p>AI parsing is also the most expensive feature: $3 per 1,000 parsed pages, on top of the scrape.</p><h3>Dedicated scrapers and endpoint discovery</h3><p>One of the most interesting features for agents is having the list of managed APIs with a simple command. <code>GET /v1/scrapers</code> returns the list of every scrape endpoint, and <code>OPTIONS</code> on one of them returns its parameters, with type, constraints and a description. When I called it, the list had 351 endpoints, 160 of them synchronous, from Amazon and Google to grocery chains like Kroger and Publix, plus YouTube, ChatGPT, Perplexity and Gemini.</p><pre><code><code>from client import describe_scraper, list_scrapers

scrapers = list_scrapers()["scrapers"]
print(len(scrapers))  # 351
print(describe_scraper("scrape/amazon/product")["query"])
# [string] [required] [min_len: 10] [max_len: 18]. ASIN of the product, e.g. `B0CX23V2ZK`.</code></code></pre><p>An agent can read this at runtime instead of using a fixed list of websites and options.  For <code>amazon/product</code>, for example, <code>OPTIONS</code> tells you that <code>location</code> is a postal code and not a country, and that you can force the <code>currency</code>.</p><p>The output of a dedicated scraper is already parsed. <code>amazon/product</code> with an ASIN returned 50 fields in about 3 seconds (price, buybox, stock, rating, reviews count and so on). <code>bestbuy/product</code> with the SKU returned title, price, <code>is_sold_out</code> and rating in 2.2 seconds, and <code>walmart/product</code> returned the price together with the marketplace seller that offers it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Dr02!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Dr02!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png 424w, https://substackcdn.com/image/fetch/$s_!Dr02!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png 848w, https://substackcdn.com/image/fetch/$s_!Dr02!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png 1272w, https://substackcdn.com/image/fetch/$s_!Dr02!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Dr02!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png" width="861" height="1037" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1037,&quot;width&quot;:861,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:176635,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/221137498?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Dr02!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png 424w, https://substackcdn.com/image/fetch/$s_!Dr02!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png 848w, https://substackcdn.com/image/fetch/$s_!Dr02!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png 1272w, https://substackcdn.com/image/fetch/$s_!Dr02!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3ed51414-6287-42c7-9abd-cba33fc07ef6_861x1037.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Example of dedicated scrapers</figcaption></figure></div><h3>MCP server and agent skills</h3><p>If you don&#8217;t want to write code, Oxylabs gives you two ready integrations. The <a href="https://developers.oxylabs.io/ai-workflows/mcp?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">MCP server</a> (<code>uv tool install git+https://github.com/oxylabs/web-api-mcp</code>) exposes seven tools: <code>search</code>, <code>scrape</code>, <code>extract</code>, <code>check_scrape</code>, <code>read_scraped</code> (for large pages that are saved to disk and read in chunks), <code>list_scrapers</code> and <code>scrape_target</code>. The <a href="https://developers.oxylabs.io/ai-workflows/agent-skills?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">agent skills</a> (<code>npx skills add oxylabs/web-api-skills</code>) give a coding agent the instructions on how to use the API, without a running server. For this article, I preferred to write my own tools because I wanted to control which endpoint the agent can call and see every request it makes.</p><h2>Building the price comparison agent</h2><p>The documentation says that &#8220;Installing beats building&#8221;: if the web access is for your coding agent, install the skills or the MCP server and you&#8217;re done. The same page says to build a client &#8220;when the project&#8217;s code has to make the calls itself&#8221;. </p><p>Our agent is a program that runs on its own, so we&#8217;re in the second case. But I could still have connected it to the official MCP server instead of writing five tools, but I preferred writing some light wrappers.</p><p>With the Claude API, the model can&#8217;t make HTTP calls on its own. It can only ask to use a tool that you declared, and your code runs it. So my tools are just thin wrappers, a few lines each, around a single Web API call, created for mainly these four reasons.</p><p>The first is visibility. Every call of my tools goes through <code>client.py</code>, which logs status, latency, request id and size. All the numbers in this article come from that log, and with the MCP server the calls happen inside another process. The second is control. As you&#8217;ll see in the UK run, two details changed the result: a single retry on a failing page and where to cut a long page before passing it to the model. With my own code, I was able to change both in one line. The third is the tool descriptions. They&#8217;re what the model reads to choose a tool, and in mine I wrote the rules we found in the tests above, like &#8220;if the price is missing, call it again with run_js=true&#8221;. The fourth is cost. The agent can call only five endpoints, so it can&#8217;t choose the more expensive ones (Google with JavaScript, ChatGPT, or Perplexity scrapers) for a price check.</p><p>Now the tutorial. The goal is an agent that receives a product name and a country, finds the product in three online shops of that country and returns a JSON array with price, currency, availability and the tool it used for each shop. It stops at three results, because the point here is to see how the agent uses the tools, not to build the biggest price list.</p><p>I used Claude Opus 5.5 through the Anthropic Python SDK, with a plain tool use loop. There are higher-level options, like the SDK&#8217;s tool runner or connecting Claude to the Oxylabs MCP server, but with a manual loop you can print every tool call and see why the agent does what it does. The only dependencies are <code>anthropic</code>, <code>requests</code> and <code>python-dotenv</code>, which were already installed in our environment. You need two keys in a <code>.env</code> file: <code>OXYLABS_WEB_API_KEY</code> and <code>ANTHROPIC_API_KEY</code>.</p><h3>The tools</h3><p>The agent gets five tools, and each one maps to a single Web API call:</p><ul><li><p><code>web_search</code> calls <code>/v1/search</code></p></li><li><p><code>scrape_page</code> calls the generic <code>/v1/scrape</code> on any URL and returns Markdown, with optional <code>run_js</code></p></li><li><p><code>extract_product</code> calls the generic <code>/v1/scrape</code> with AI parsing and the schema above</p></li><li><p><code>amazon_product</code> and <code>bestbuy_product</code> call the dedicated scrapers by ASIN and SKU</p></li></ul><p>The descriptions are where you put the knowledge from the tests above, because they&#8217;re what the model reads when it chooses a tool. This is the definition of <code>scrape_page</code>:</p><pre><code><code>{
    "name": "scrape_page",
    "description": "Read any URL as Markdown (Oxylabs generic /v1/scrape). Fast without JavaScript. "
                   "If the price or the main content is missing, call it again with run_js=true "
                   "(real browser render, 20-40 seconds).",
    "input_schema": {
        "type": "object",
        "properties": {
            "url": {"type": "string"},
            "run_js": {"type": "boolean", "default": False},
        },
        "required": ["url"],
    },
}</code></code></pre><p>And this is the code that runs the tools. Pages longer than 60,000 characters are cut, starting from the first H1 heading, and the agent is told about the cut. I&#8217;ll explain in the UK run why the cut starts from the H1 and not from the top of the page.</p><pre><code><code>def run_tool(name: str, args: dict) -&gt; dict:
    if name == "web_search":
        out = search(args["query"], max_results=10, location=args.get("location"))
        return {"results": [{"title": r["title"], "url": r["url"], "outline": r.get("outline", [])[:1]}
                            for r in out["results"]]}
    if name == "scrape_page":
        # Oxylabs already retries a failed scrape server-side before answering 500,
        # so the agent gets the error once and decides what to do (usually run_js=true).
        out = scrape(args["url"], output=["markdown"], run_js=bool(args.get("run_js")), attempts=1)
        md = out["results"][0].get("markdown") or ""
        result = {"chars": len(md), "markdown": md[:MAX_PAGE_CHARS]}
        if len(md) &gt; MAX_PAGE_CHARS:
            # Rendered pages can start with 100K+ chars of menus: start from the first H1 (the product title).
            h1 = md.find("\n# ")
            start = h1 if h1 &gt; 0 else 0
            result["markdown"] = md[start:start + MAX_PAGE_CHARS]
            result["note"] = f"{len(md)} chars, showing {start}-{start + MAX_PAGE_CHARS} (from the first H1)"
        return result
    if name == "extract_product":
        out = scrape(args["url"], output=["json"], json={"schema": PRICE_SCHEMA}, run_js=bool(args.get("run_js")))
        return out["results"][0].get("json") or {}
    if name == "amazon_product":
        out = scrape_target("amazon/product", {"query": args["asin"], "domain": args.get("domain", "com"),
                                               "output": ["json"]})
        j = out["results"][0].get("json") or {}
        keep = ("title", "price", "price_buybox", "currency", "stock", "rating", "reviews_count", "url")
        return {k: j.get(k) for k in keep}
    if name == "bestbuy_product":
        out = scrape_target("bestbuy/product", {"query": args["sku"], "output": ["json"]})
        return out["results"][0].get("json") or {}
    raise ValueError(f"unknown tool {name}")</code></code></pre><p>For <code>scrape_page</code> I turned off the client retries (<code>attempts=1</code>). In my first version the client retried a failing page three times, and since Oxylabs had already retried on its side, the agent waited more than two minutes before seeing the error. Now it gets the error at the first failure and can switch to <code>run_js</code> immediately.</p><h3>The system prompt</h3><p>The system prompt contains the rules from the For Agents page, adapted to prices:</p><pre><code><code>You are a price research agent. The user gives you a product and a country.
Find the current price of that exact product in 3 different online shops of that country, then stop.

Rules:
- Start with one web_search. Choose shop product pages, not reviews, forums or price trackers.
- Read prices only from pages you scraped. Never use search snippets or your memory for a price.
- For Amazon use amazon_product, for Best Buy use bestbuy_product. For other shops use scrape_page first.
- If scrape_page fails or returns a page without the product price, call it again with run_js=true.
- Use extract_product only when the Markdown is ambiguous.
- If a page fails twice, report it as failed and pick another shop.
- Stop as soon as you have 3 prices.
- If the user lists shop URLs, skip the search and read exactly those pages.

Final answer: a JSON array with one object per shop:
{"shop", "url", "price", "currency", "availability", "tool_used", "run_js"}
followed by one short line on anything that failed.</code></code></pre><p>The rule about search snippets isn&#8217;t there by chance, and we&#8217;ll see why in a moment.</p><h3>The loop</h3><p>The loop sends the conversation to Claude, runs every tool call in the response, sends back the results and repeats until the model stops asking for tools. A tool error goes back as a <code>tool_result</code> with <code>is_error: true</code>, so the model can react to it like any other result.</p><pre><code><code>for turn in range(20):
    response = client.messages.create(
        model=MODEL, max_tokens=16000, system=SYSTEM, tools=TOOLS, messages=messages,
    )
    messages.append({"role": "assistant", "content": response.content})

    if response.stop_reason != "tool_use":
        final = "".join(b.text for b in response.content if b.type == "text")
        break

    results = []
    for block in response.content:
        if block.type != "tool_use":
            continue
        try:
            output, is_error = run_tool(block.name, block.input), False
        except (WebApiError, KeyError, IndexError, ValueError) as exc:
            output, is_error = {"error": str(exc)}, True
        results.append({"type": "tool_result", "tool_use_id": block.id,
                        "content": json.dumps(output, ensure_ascii=False), "is_error": is_error})
    messages.append({"role": "user", "content": results})</code></code></pre><p>The full script, with the logging I removed here, is <code>agent.py</code> in the <code>oxylabs_web_api_agent</code> folder of <a href="https://github.com/TheWebScrapingClub/TheScrapingClubFree/tree/main/Oxylabs%20Web%20API">our GitHub repository</a>. You run it with <code>python agent.py "Sony WH-1000XM5" US</code>, or you add up to three shop URLs after the country to skip the search.</p><h2>Running the agent</h2><h3>United States</h3><p>The first run, on the US market, took 22 seconds and four tool calls. The agent searched once, then called the three shops in parallel in the same turn:</p><pre><code><code>[turn 0] web_search({"query": "Sony WH-1000XM5 headphones buy", "location": "US"}) -&gt; 1.7s
[turn 1] amazon_product({"asin": "B09XS7JWHH"}) -&gt; 3.1s, 199.99 USD, In Stock
[turn 1] bestbuy_product({"sku": "6505727"}) -&gt; 2.4s, 228 USD, is_sold_out: false
[turn 1] scrape_page({"url": "https://electronics.sony.com/audio/headphones/headband/p/wh1000xm5-b"}) -&gt; 5.3s, 23,023 chars</code></code></pre><p>Amazon at $199.99 and Best Buy at $228 came from the dedicated scrapers. The Sony store came from the generic scraper, with $299.99 (marked down from $399.99). The agent then added a note on its own. The search snippet for the Sony page said &#8220;Sale Price $228.00 Save $171.99&#8221;, so maybe the Markdown without JavaScript was stale and the page should be checked fully loaded.</p><p>I checked it manually and the Sony page shows $299.99 with and without <code>run_js</code> (4.1 seconds against 23.6), so the price in the Markdown was right, and the snippet was wrong. That&#8217;s exactly why the docs tell you to never answer from<code>outline</code>, and why that rule is in the system prompt.</p><h3>Italy</h3><p>On the Italian market (<code>location: IT</code>) the agent searched &#8220;Sony WH-1000XM5 prezzo&#8221;, used the Amazon dedicated scraper with <code>domain: it</code> and read Unieuro and the Sony Store Italia with the generic scraper. Everything was in the static Markdown: Amazon.it at &#8364;216.49, Unieuro at &#8364;199.00 and the Sony Store at &#8364;259.00. Total time was 25 seconds with four tool calls, and the agent also noted that Unieuro adds an extra 5% discount in the cart, which isn&#8217;t in the &#8364;199 price.</p><h3>United Kingdom, where JavaScript is needed</h3><p>In the first two markets, every page the agent read with the generic scraper had the price in the static Markdown. To see the JavaScript fallback in action, I needed a shop without a dedicated scraper that doesn&#8217;t work without rendering. To find one, I ran the same search in seven countries, scraped every shop page without JavaScript and checked which ones had no price.</p><p>Argos UK was the clearest case. The scrape without JavaScript failed every time I tried it (four times, between 22 and 43 seconds each), always with the 500 <code>REQUEST_FAILED_AFTER_TOO_MANY_RETRIES</code> that isn&#8217;t charged. With <code>run_js</code> the page came back in about 22 seconds, with the price. So for this run I passed the agent three UK URLs: Argos, Currys and Amazon UK.</p><pre><code><code>[turn 0] scrape_page({"url": "https://www.argos.co.uk/product/2847319"}) -&gt; 21.9s
         error: 500 REQUEST_FAILED_AFTER_TOO_MANY_RETRIES
[turn 0] scrape_page({"url": "https://www.currys.co.uk/products/sony-wh1000xm5-...-10238756.html"}) -&gt; 4.4s, 41,021 chars
[turn 0] amazon_product({"asin": "B09Y2MYL5C", "domain": "co.uk"}) -&gt; 2.8s, 199 GBP, In stock
[turn 1] scrape_page({"url": "https://www.argos.co.uk/product/2847319", "run_js": true}) -&gt; 22.6s
         172,012 chars, showing 137143-197143 (from the first H1)</code></code></pre><p>The agent read the error, followed the rule in the system prompt and asked for the same page with <code>run_js: true</code>. The final answer had Currys at &#163;199.00 from the static Markdown, Argos at &#163;229.99 from the rendered Markdown and Amazon UK at &#163;199.00 from the dedicated scraper. Total time was 62 seconds with four tool calls. For Argos the agent also wrote that it couldn&#8217;t confirm availability, because the page shows an &#8220;Add to trolley&#8221; button but the stock depends on the postcode.</p><p>This run didn&#8217;t work at the first attempt, and the reason is worth telling. The rendered Argos page is 172,012 characters of Markdown, and the product title starts at character 137,143. Everything before it is menus, mega menus and navigation links. In my first version the tool cut the page at the first 60,000 characters, so the agent received only the menu. It then called <code>extract_product</code> with <code>run_js</code> (another 42 seconds) to get the price, and it couldn&#8217;t read the availability. After I changed the cut to start from the first H1, the same run went from 153 to 62 seconds and from 130,000 to 62,000 input tokens. With rendered pages, the Markdown conversion doesn&#8217;t remove the navigation, so it&#8217;s up to you to decide which part of the page the model reads.</p><p>The same first run also failed on Amazon UK with a 500 <code>PARSE_NOT_SUPPORTED</code>. That one was my error: I had passed the amazon.com ASIN of the headphones, and on amazon.co.uk the product has a different one (B09Y2MYL5C, the same as on amazon.it).</p><h2>What it costs</h2><p><a href="https://oxylabs.io/pricing?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">Oxylabs prices</a> start with a Starter plan at $30 per month. At that tier, <code>/search</code> costs $1 per 1,000 searches, the generic <code>/scrape</code> starts from $0.50 per 1,000 results, custom domains with JavaScript cost $1.50, Amazon without JavaScript costs $0.50 and AI parsing $3 per 1,000 parsed pages. Volume discounts bring the generic scrape down to $0.25 per 1,000 above 10 million results.</p><p>So one run of our agent, with a search and three scrapes, costs a fraction of a cent on the Oxylabs side, even when one of the pages needs JavaScript. The bigger cost is the LLM. The US run used 20,472 input tokens and 848 output tokens, which at the Claude Opus 5.5 price ($4 per million input tokens and $20 per million output tokens) is about 10 cents. The UK run, with the rendered Argos page, used 61,759 input tokens, about 26 cents. Most of those tokens are page Markdown, so the 120 times reduction from HTML to Markdown is what makes this agent affordable at all, and what you cut from the Markdown before passing it to the model is the next place to save money.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!JvxX!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JvxX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png 424w, https://substackcdn.com/image/fetch/$s_!JvxX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png 848w, https://substackcdn.com/image/fetch/$s_!JvxX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png 1272w, https://substackcdn.com/image/fetch/$s_!JvxX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!JvxX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png" width="469" height="733" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:733,&quot;width&quot;:469,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:33481,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/221137498?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!JvxX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png 424w, https://substackcdn.com/image/fetch/$s_!JvxX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png 848w, https://substackcdn.com/image/fetch/$s_!JvxX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png 1272w, https://substackcdn.com/image/fetch/$s_!JvxX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6f74ea-fc66-47c9-b4c4-a5f6d2ff48c2_469x733.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>Final considerations</h2><p>The <a href="https://oxylabs.go2cloud.org/aff_c?offer_id=7&amp;aff_id=665&amp;url_id=222">Oxylabs Web API</a> does what the quick start promises: a search endpoint and a scrape endpoint that an agent can use without knowing anything about proxies or rendering. In my tests, the searches answered in under 2 seconds, the dedicated scrapers in 2 to 4 seconds, and the generic scraper handled every shop we sent it to, using <code>run_js</code> on the one that needed it.</p><p>What I liked most is that the product is clearly designed for agents. You see it in the documentation, in the Markdown output, in the endpoint discovery that a model can read at runtime, and in the faulted scrapes that you don&#8217;t pay for. </p><p>The part that still depends on you is the agent itself: which tools you expose, what you write in their descriptions, and how much of each page you pass to the model. In our case, the difference between a slow, expensive run and a fast, cheap one was a single line that chose where to cut the Markdown. The next step for this agent would be a discovery tool so it can automatically pick the dedicated scraper for every shop that has one, instead of the two I gave it.</p><p>Let me know in the comments if you&#8217;ve already tried the Web API, and which agent you&#8217;d build with it.</p>]]></content:encoded></item><item><title><![CDATA[SERP APIs vs. Web Search APIs: Understanding the Real Differences]]></title><description><![CDATA[Are SERP APIs and web search APIs different names for the same tool? Let&#8217;s find out!]]></description><link>https://www.scraping.club/p/serp-apis-vs-web-search-apis</link><guid isPermaLink="false">https://www.scraping.club/p/serp-apis-vs-web-search-apis</guid><dc:creator><![CDATA[Antonello Zanini]]></dc:creator><pubDate>Sun, 18 Oct 2026 19:19:53 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/cc2f5b68-00b3-411d-a948-db752c6d2820_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>The web data retrieval landscape has evolved dramatically over the past few years, particularly when it comes to accessing web search results. Today, you can choose from specialized APIs tailored not only to traditional SEO-based search workflows but also to AI agents and modern LLM-based applications.</p><p>In this post, I&#8217;ll explore the world of SERP APIs and web search APIs, explain how  and where they differ. I&#8217;ll also compare their use cases and how they work under the hood.</p><h2>Why the Confusion Between SERP APIs and Web Search APIs?</h2><p>Talk to people in the web scraping industry, and you&#8217;ll notice that many use &#8220;SERP APIs&#8221; and &#8220;web search APIs&#8221; interchangeably. That confusion didn&#8217;t emerge by accident... It reflects how the market has evolved alongside the new demands of AI-powered applications.</p><p>Let me cover how we got here!</p><h3>The Traditional SERP API Era</h3><p>For years, SERP APIs were the standard solution for accessing search engine results. These <a href="https://www.scraping.club/p/web-scraping-for-seo">solutions are built around SEO professionals</a>, competitive analysts, and market researchers who need reliable access to search data.</p><p>SERP APIs provide a straightforward workflow:</p><ol><li><p>Send a search query to a given search engine through an API endpoint.</p></li><li><p>Receive the resulting <a href="https://www.semrush.com/blog/serp/">search engine results page (SERP)</a> as raw HTML or parsed JSON.</p></li></ol><p>That&#8217;s where the name &#8220;<em>SERP API&#8221;</em> comes from. The model is simple and effective for its intended audience.</p><p>However, the emergence of AI systems introduced new requirements. <a href="https://github.com/HaoooWang/llm-knowledge-cutoff-dates">Traditional LLMs have knowledge cutoff dates</a> and can&#8217;t access real-time information on their own. to retrieve and ground their responses in current information, they need external tools.</p><p>As users increasingly expected AI systems to answer questions about recent events, breaking news, and other up-to-date topics, responses such as &#8220;Sorry, I don&#8217;t have information about this event because of my knowledge cutoff&#8221; became less acceptable.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_gyH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_gyH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png 424w, https://substackcdn.com/image/fetch/$s_!_gyH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png 848w, https://substackcdn.com/image/fetch/$s_!_gyH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png 1272w, https://substackcdn.com/image/fetch/$s_!_gyH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_gyH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png" width="1456" height="770" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:770,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!_gyH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png 424w, https://substackcdn.com/image/fetch/$s_!_gyH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png 848w, https://substackcdn.com/image/fetch/$s_!_gyH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png 1272w, https://substackcdn.com/image/fetch/$s_!_gyH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc7fafaf0-2759-4899-9856-440294bd2872_3072x1624.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Asking ChatGPT a question about a recent event with &#8220;Web search&#8221; tool disabled</figcaption></figure></div><p>This created an opportunity for a new category of tools: <em>web search APIs</em>, built to help AI agents and LLM-powered applications retrieve relevant information from the web.</p><div><hr></div><blockquote><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/B5JPeJ">Decodo's Web Data API</a></strong> turns any site into clean, structured data your models can actually use.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/B5JPeJ" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png" width="716" height="286.4" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:280,&quot;width&quot;:700,&quot;resizeWidth&quot;:716,&quot;bytes&quot;:105634,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!YKjX!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 424w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 848w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1272w, https://substackcdn.com/image/fetch/$s_!YKjX!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc01dd577-256f-4488-8aa6-695010486270_700x280.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/B5JPeJ&quot;,&quot;text&quot;:&quot;Activate your free plan&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://visit.decodo.com/B5JPeJ"><span>Activate your free plan</span></a></p><div><hr></div></blockquote><h2>The Rebranding Wave</h2><p>As demand for AI-powered search capabilities grew, something interesting happened in the market. In detail, existing SERP API providers recognized the opportunity and began adapting their products to meet the needs of AI applications.</p><p>Some providers transformed their existing solutions into AI-ready products by adding:</p><ul><li><p>Options to return results in Markdown (the <a href="https://www.scraping.club/p/why-scraping-return-markdown-llm-ai">ideal data format for LLMs ingestion</a>).</p></li><li><p>Integrations to simplify adoption, such as MCP servers, agent skills, OpenAPI specifications, and plugins for frameworks such as LangChain, LlamaIndex, OpenClaw, and more.</p></li></ul><p>Some of those companies even rebranded their SERP API products entirely as &#8220;web search APIs.&#8221; At the same time, new providers launched purpose-built tools designed specifically for AI applications, including Exa, Tavily, and others.</p><p>This combination of approaches created the confusion we see today. <em>Are SERP APIs and web search APIs the same thing? </em>Not exactly. <em>Are they completely different?</em> Not quite.</p><p>The reality is more nuanced&#8230;</p><h2>What Are SERP APIs and Web Search APIs?</h2><p>Time to take a closer look at SERP APIs and web search APIs, and understand whether you can use the former to implement the latter.</p><h3>SERP API: Definition</h3><p>A SERP API, short for <em>Search Engine Results Page API</em>, automates the extraction of data directly from search engine results pages, such as Google, Bing, Yandex, Brave, DuckDuckGo, and others.</p><p>The provider handles the technical complexity of accessing the search engine, including proxy management, CAPTCHA solving, and, optionally, parsing the response into a structured format.</p><p><strong>Note</strong>: By extension, some providers also classify APIs that return scraped search results from websites such as Amazon and YouTube as &#8220;SERP APIs.&#8221; In these cases, the scraping API isn&#8217;t providing traditional web search data. Instead, it returns product search results, video search results, or other types of platform-specific search data.</p><div><hr></div><blockquote><p>Building your own search data pipeline? From collecting search results to extracting the pages behind them, <strong><a href="https://anyip.io/?utm_source=thewebscrapingclub&amp;utm_medium=newsletter&amp;utm_campaign=gold_2026_9&amp;utm_content=banner_search_data">anyIP provides residential and mobile proxies worldwide</a></strong>. You control the collection logic. We power the connections.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://anyip.io/?utm_source=thewebscrapingclub&amp;utm_medium=newsletter&amp;utm_campaign=gold_2026_9&amp;utm_content=banner_search_data" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!0anC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b8a015-041d-4496-a523-460a84abb832_2160x1080.png 424w, https://substackcdn.com/image/fetch/$s_!0anC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b8a015-041d-4496-a523-460a84abb832_2160x1080.png 848w, https://substackcdn.com/image/fetch/$s_!0anC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b8a015-041d-4496-a523-460a84abb832_2160x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!0anC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b8a015-041d-4496-a523-460a84abb832_2160x1080.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!0anC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b8a015-041d-4496-a523-460a84abb832_2160x1080.png" width="606" height="303" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/54b8a015-041d-4496-a523-460a84abb832_2160x1080.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:728,&quot;width&quot;:1456,&quot;resizeWidth&quot;:606,&quot;bytes&quot;:302147,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://anyip.io/?utm_source=thewebscrapingclub&amp;utm_medium=newsletter&amp;utm_campaign=gold_2026_9&amp;utm_content=banner_search_data&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b8a015-041d-4496-a523-460a84abb832_2160x1080.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!0anC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b8a015-041d-4496-a523-460a84abb832_2160x1080.png 424w, https://substackcdn.com/image/fetch/$s_!0anC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b8a015-041d-4496-a523-460a84abb832_2160x1080.png 848w, https://substackcdn.com/image/fetch/$s_!0anC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b8a015-041d-4496-a523-460a84abb832_2160x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!0anC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F54b8a015-041d-4496-a523-460a84abb832_2160x1080.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://anyip.io/?utm_source=thewebscrapingclub&amp;utm_medium=newsletter&amp;utm_campaign=gold_2026_9&amp;utm_content=banner_search_data&quot;,&quot;text&quot;:&quot;Explore anyIP Proxies &#8594;&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://anyip.io/?utm_source=thewebscrapingclub&amp;utm_medium=newsletter&amp;utm_campaign=gold_2026_9&amp;utm_content=banner_search_data"><span>Explore anyIP Proxies &#8594;</span></a></p></blockquote><div><hr></div><h3>Web Search API: Definition</h3><p>A web search API is a more general-purpose tool that provides web search functionality without necessarily exposing the underlying source results. These solutions are mostly engineered with AI applications in mind from day one, serving as tools that LLMs can call directly.</p><p>They may be exposed through MCP or similar integrations and return data in formats optimized for LLM consumption, such as relevant text, summaries, and URLs. Before returning the results, these APIs often handle tasks such as ranking, relevance scoring, and content extraction.</p><h3>Can a Web Search API Use a SERP API Underneath?</h3><p>Absolutely. In fact, many modern web search APIs employ SERP APIs as one of their underlying data sources. The distinction isn&#8217;t about where the data comes from, but about what the provider does with that data before returning it to you.</p><p>A web search API might retrieve results through a SERP API and then apply additional processing, such as ranking, relevance filtering, content extraction from landing pages, semantic analysis, or AI-powered summarization.</p><p>For example, <a href="https://techcrunch.com/2025/03/21/anthropic-appears-to-be-using-brave-to-power-web-searches-for-its-claude-chatbot/">Anthropic appears to use the Brave Search API to provide web search capabilities in Claude</a>. In this type of setup, the underlying search provider is only one part of the overall system.</p><p>This additional processing layer is what can turn a traditional search-results interface into a broader AI-ready web search tool.</p><div><hr></div><blockquote><p><em><strong><a href="https://serpapi.com/webscrapingclub">SerpApi now goes beyond the search results page</a></strong>: with the new Extract API, the same platform you use for Google results turns any public URL into JSON, Markdown, HTML or plain text, and can map or crawl a whole domain in a single request.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://serpapi.com/webscrapingclub" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Xp9J!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png 424w, https://substackcdn.com/image/fetch/$s_!Xp9J!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png 848w, https://substackcdn.com/image/fetch/$s_!Xp9J!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!Xp9J!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Xp9J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png" width="522" height="293.625" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:522,&quot;bytes&quot;:46370,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://serpapi.com/webscrapingclub&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Xp9J!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png 424w, https://substackcdn.com/image/fetch/$s_!Xp9J!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png 848w, https://substackcdn.com/image/fetch/$s_!Xp9J!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png 1272w, https://substackcdn.com/image/fetch/$s_!Xp9J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a57c87b-43d4-48c6-97bf-b9d7f48e0d77_1920x1080.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://serpapi.com/&quot;,&quot;text&quot;:&quot;Try Extract API&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://serpapi.com/"><span>Try Extract API</span></a></p></blockquote><div><hr></div><h3>How SERP APIs and Web Search APIs Work</h3><p>Let me explain the process that SERP APIs and web search APIs follow to go from input to output.</p><h3>SERP API Operations</h3><p>The process works as follows:</p><ol><li><p><strong>You define the search parameters</strong>: You provide the query and specify the search engine and other parameters, such as location, language, and search type (e.g., organic results, news, maps, or shopping).</p></li><li><p><strong>The provider handles the technical complexity</strong>: Its infrastructure takes care of tasks such as proxy rotation, CAPTCHA solving, JavaScript rendering, and geo-targeting.</p></li><li><p><strong>The provider retrieves and returns the search results</strong>:<strong> </strong>The API accesses the target search engine and <a href="https://www.scraping.club/p/scraping-serp-google-search">scrapes the results from it</a>. Depending on the output options, you may receive raw HTML, parsed JSON, LLM-friendly Markdown, or other formats.</p></li></ol><p>One important thing to keep in mind is that a SERP API returns data from a search engine results page. You&#8217;ll receive URLs, titles, and snippets, but not the complete text of the underlying pages.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!zO5J!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!zO5J!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png 424w, https://substackcdn.com/image/fetch/$s_!zO5J!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png 848w, https://substackcdn.com/image/fetch/$s_!zO5J!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png 1272w, https://substackcdn.com/image/fetch/$s_!zO5J!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!zO5J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png" width="1456" height="928" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:928,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Example output for the &#8220;web scraping club&#8221; search query using the Bright Data SERP API&quot;,&quot;title&quot;:&quot;Example output for the &#8220;web scraping club&#8221; search query using the Bright Data SERP API&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Example output for the &#8220;web scraping club&#8221; search query using the Bright Data SERP API" title="Example output for the &#8220;web scraping club&#8221; search query using the Bright Data SERP API" srcset="https://substackcdn.com/image/fetch/$s_!zO5J!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png 424w, https://substackcdn.com/image/fetch/$s_!zO5J!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png 848w, https://substackcdn.com/image/fetch/$s_!zO5J!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png 1272w, https://substackcdn.com/image/fetch/$s_!zO5J!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F04e8790c-5082-4382-9e90-7546a512544c_2720x1734.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Example output for the &#8220;web scraping club&#8221; search query using the Bright Data SERP API</figcaption></figure></div><p>This distinction matters in AI workflows. An LLM can use the search results to identify the most relevant URLs and then pass those URLs to another tool that extracts the actual page content, such as a <a href="https://www.scraping.club/p/cloudflare-web-unblocker-benchmark">web unblocker</a>. This creates the <em><a href="https://hackernoon.com/search-and-extract-why-this-ai-pattern-matters-tutorial-and-example">search-and-extract</a></em><a href="https://hackernoon.com/search-and-extract-why-this-ai-pattern-matters-tutorial-and-example"> pattern</a> commonly employed by AI agents for web grounding.</p><p>A big advantage of this approach to web search data retrieval is reproducibility. Because you control the search query and parameters, you can reproduce the request by running the same search directly on the target search engine.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_Dna!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_Dna!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png 424w, https://substackcdn.com/image/fetch/$s_!_Dna!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png 848w, https://substackcdn.com/image/fetch/$s_!_Dna!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png 1272w, https://substackcdn.com/image/fetch/$s_!_Dna!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_Dna!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png" width="1456" height="778" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:778,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Performing the &#8220;web scraping club&#8221; search directly on Google. The results match those returned by the SERP API&quot;,&quot;title&quot;:&quot;Performing the &#8220;web scraping club&#8221; search directly on Google. The results match those returned by the SERP API&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Performing the &#8220;web scraping club&#8221; search directly on Google. The results match those returned by the SERP API" title="Performing the &#8220;web scraping club&#8221; search directly on Google. The results match those returned by the SERP API" srcset="https://substackcdn.com/image/fetch/$s_!_Dna!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png 424w, https://substackcdn.com/image/fetch/$s_!_Dna!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png 848w, https://substackcdn.com/image/fetch/$s_!_Dna!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png 1272w, https://substackcdn.com/image/fetch/$s_!_Dna!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F13d15c69-16cd-435f-b9c7-bf0f3beb3d4b_3072x1642.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Performing the &#8220;web scraping club&#8221; search directly on Google. The results match those returned by the SERP API</figcaption></figure></div><p>That&#8217;s useful for debugging, verification, and compliance. If your pipeline retrieves information through a SERP API, you can also record the exact query, parameters, and timestamp to document what data was retrieved and when.</p><h3>Web Search API Operations</h3><p>Web Search APIs usually operate more like a black box. You receive the final output, but the path from your query to those results may not be fully transparent.</p><p>The workflow follows these steps:</p><ol><li><p><strong>You submit a search request</strong>: You provide a search query, which is typically a natural-language sentence or prompt rather than a traditional, SEO-oriented search query. Depending on the API, you could also specify parameters such as location, language, result type, or latency preferences.</p></li><li><p><strong>The provider searches one or more sources</strong>: The API does its magic and fetches the information you&#8217;re looking for.</p></li><li><p><strong>The provider processes and returns the results</strong>: It might apply ranking, relevance filtering, content extraction, semantic analysis, or other processing to determine which information is most useful. The final response is returned as structured JSON, Markdown, plain text, summaries, URLs, or a combination of these formats.</p></li></ol><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gAyr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gAyr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png 424w, https://substackcdn.com/image/fetch/$s_!gAyr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png 848w, https://substackcdn.com/image/fetch/$s_!gAyr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png 1272w, https://substackcdn.com/image/fetch/$s_!gAyr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gAyr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png" width="1456" height="723" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:723,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Note the textual response from the Exa Search API to a natural-language search prompt&quot;,&quot;title&quot;:&quot;Note the textual response from the Exa Search API to a natural-language search prompt&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Note the textual response from the Exa Search API to a natural-language search prompt" title="Note the textual response from the Exa Search API to a natural-language search prompt" srcset="https://substackcdn.com/image/fetch/$s_!gAyr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png 424w, https://substackcdn.com/image/fetch/$s_!gAyr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png 848w, https://substackcdn.com/image/fetch/$s_!gAyr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png 1272w, https://substackcdn.com/image/fetch/$s_!gAyr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7010fb1-9b91-4aef-977b-8b975fa8fe9a_3032x1506.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Note the textual response from the Exa Search API to a natural-language search prompt</figcaption></figure></div><p>This approach has both advantages and disadvantages. On the positive side, providers can optimize their algorithms and infrastructure without exposing their implementation details. They can also combine multiple sources, apply machine learning to rank results, and tailor the output to AI-powered applications.</p><p>The downside is reduced transparency and reproducibility. A web search API can return different results for the same query over time, adding further uncertainty to the already probabilistic nature of LLMs.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><h3>Data Sources: Where Do These APIs Get Their Information?</h3><p>Discover where SERP APIs and web search APIs get their data from.</p><h3>SERP API Data Sources</h3><p>SERP APIs retrieve data from identifiable search engines. Common options include Google, Bing, Yandex, Baidu, DuckDuckGo, Naver, Seznam, and Yahoo. As mentioned earlier, some SERP APIs also cover specialized search platforms and verticals, such as product search, news, and social platforms.</p><p>The data you retrieve depends on the parameters you provide. Location can greatly affect the results, just like language and device (e.g., a mobile search may return different content from a desktop search). The search type matters as well: organic results, paid ads, featured snippets, knowledge panels, and maps may require different handling.</p><h3>Web Search API Data Sources</h3><p>Web search APIs can draw data from multiple sources. Some rely entirely on SERP APIs from different search engines. Others maintain proprietary web indexes built through continuous crawling. More sophisticated solutions tend to combine several approaches, selecting different sources depending on the query.</p><p>This flexibility can give web search APIs broader coverage than any single SERP API. They&#8217;re not limited by the coverage or limitations of one search engine. Yet, this comes with a trade-off: you generally have less visibility into exactly where each result came from and how it was generated.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://substack.thewebscraping.club/p/consulting&quot;,&quot;text&quot;:&quot;Need help with your scraping project?&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://substack.thewebscraping.club/p/consulting"><span>Need help with your scraping project?</span></a></p><div><hr></div><h3>Leading SERP and Web Search API Providers</h3><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XBJr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XBJr!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg 424w, https://substackcdn.com/image/fetch/$s_!XBJr!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg 848w, https://substackcdn.com/image/fetch/$s_!XBJr!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!XBJr!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!XBJr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg" width="1456" height="959" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:959,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:271919,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215677959?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!XBJr!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg 424w, https://substackcdn.com/image/fetch/$s_!XBJr!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg 848w, https://substackcdn.com/image/fetch/$s_!XBJr!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg 1272w, https://substackcdn.com/image/fetch/$s_!XBJr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe372f154-4db4-4fb7-b7ab-5634b04d874e_1456x959.jpeg 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2>Choosing Between SERP APIs and Web Search APIs</h2><p>Choosing between a SERP API and a web search API ultimately depends on your use case, not on pricing or technical skills.</p><p><strong>Prefer a web search API when</strong>:</p><ul><li><p>You need real-time information for AI agents and want search results returned in a format they can consume directly.</p></li><li><p>You require multi-source search without caring which search engine provides each result.</p></li><li><p>You want direct textual responses rather than raw SERP data, saving you from building your own content extraction pipeline or logic.</p></li></ul><p><strong>Opt for SERP APIs when</strong>:</p><ul><li><p>You want SEO or rank-tracking data to monitor keyword positions, competitor visibility, and search engine rankings.</p></li><li><p>You need reproducible, auditable results and want precise control over the search engine, query, location, language, and other parameters, while being able to document exactly how each result was produced.</p></li><li><p>You require specific SERP data such as organic results, ads, featured snippets, People Also Ask, knowledge panels, or maps.</p></li></ul><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>SERP APIs vs. Web Search APIs: Comparison Table</h2><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!7Ht0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7Ht0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png 424w, https://substackcdn.com/image/fetch/$s_!7Ht0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png 848w, https://substackcdn.com/image/fetch/$s_!7Ht0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png 1272w, https://substackcdn.com/image/fetch/$s_!7Ht0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!7Ht0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png" width="1456" height="1344" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1344,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!7Ht0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png 424w, https://substackcdn.com/image/fetch/$s_!7Ht0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png 848w, https://substackcdn.com/image/fetch/$s_!7Ht0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png 1272w, https://substackcdn.com/image/fetch/$s_!7Ht0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F90cba6c1-39db-46cb-8ace-40c3689ae59c_1766x1630.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">SERP APIs vs. web search APIs: comparison table</figcaption></figure></div><h2>Conclusion</h2><p>Here, I covered what SERP APIs and web search APIs are, how they work, and where they source their data. In particular, you learned that these two types of APIs serve overlapping but distinct purposes.</p><p>SERP APIs provide direct and reproducible access to search engine results, while web search APIs abstract away these details and provide search capabilities optimized for AI systems.</p><p>Complex applications can end upt using both types of services. SERP APIs can handle specific search engine needs, while web search APIs can power AI-powered search and agent workflows. This hybrid strategy provides the flexibility needed to support diverse, real-world use cases.</p><p>I hope you found this post useful. If you have any questions or comments, feel free to leave them below. Thanks for reading, and see you in the next one!</p><div><hr></div><p><em>Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/p/serp-apis-vs-web-search-apis?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.scraping.club/p/serp-apis-vs-web-search-apis?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Crawl of Fame - October 2026]]></title><description><![CDATA[Social media scraping between closing backdoors, new datasets and lawsuits.]]></description><link>https://www.scraping.club/p/reddit-linkedin-tiktok-scraping</link><guid isPermaLink="false">https://www.scraping.club/p/reddit-linkedin-tiktok-scraping</guid><dc:creator><![CDATA[Pierluigi Vinciguerra]]></dc:creator><pubDate>Tue, 13 Oct 2026 03:00:58 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/fd497bbc-7d5b-4a1d-bb71-f91a9e01d16b_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Welcome back to the Crawl of Fame; in this October edition, we&#8217;ll talk about social networks but from different angles. Let&#8217;s start from Reddit, which announced the end of its RSS feed and its public API, making it more difficult to scrape.</p><p>A few days later, a small company published the metadata of 5.6 billion TikTok videos on Hugging Face, which is remarkable since TikTok is known for being a hard target to scrape.<br><br>Then, from the legal perspective, LinkedIn closed another lawsuit against a scraping company, and we&#8217;ll dive a bit into it.</p><div><hr></div><blockquote><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/rE0PQB">Decodo&#8217;s Web Scraping API</a></strong> turns any site into clean, structured data your models can actually use.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/rE0PQB" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aRfI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png" width="724" height="362" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:512,&quot;width&quot;:1024,&quot;resizeWidth&quot;:724,&quot;bytes&quot;:716985,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/rE0PQB&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://substack.thewebscraping.club/i/204450518?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!aRfI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/rE0PQB&quot;,&quot;text&quot;:&quot;Activate starter plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/rE0PQB"><span>Activate starter plan</span></a></p></blockquote><div><hr></div><h2>Reddit closes RSS on November 13 and the public API in March 2027</h2><p>On September 30, Reddit published two posts, one in <a href="https://www.reddit.com/r/modnews/comments/1wubgvt/continuing_our_infrastructure_updates_whats/">r/modnews</a> for moderators and one in <a href="https://www.reddit.com/r/redditdev/comments/1wubcvf/moving_data_api_apps_to_the_developer_platform/">r/redditdev</a> for developers. Starting from October 31, Reddit won&#8217;t accept new requests for public API access, and from November 13, RSS feeds will stop working. Then, starting January 12, 2027, Reddit begins removing API access for apps and users that haven&#8217;t registered, and in March 2027, public access closes to everyone.</p><p>The reason is obvious and clearly stated by the moderators: &#8220;Because RSS is now a common surface for large-scale scraping and automated abuse, we&#8217;ll stop supporting RSS feeds on November 13, 2026&#8221;. This means that apps and bots should move to the Developer Platform (Devvit) managed by Reddit itself, rather than being hosted on third-party servers. And there will be no replacement for RSS feeds.  </p><p>In addition, in the past months, Reddit already required logging in to the &#8220;old&#8221; Reddit endpoint, which was used as a backdoor by scrapers.</p><h3>What&#8217;s left for people who collect Reddit data</h3><p>After March 2027, the official paths are three: a Devvit app, a data licensing deal, or nothing. Devvit is made to build apps that run inside Reddit communities; it uses TypeScript, and Reddit told&nbsp;<a href="https://ppc.land/reddit-to-halt-public-data-api-access-in-march-2027-as-14-000-apps-register/">PPC Land</a>&nbsp;that &#8220;adding additional language support has not been a scalable thing for us,&#8221; which is bad news for the many bots written with PRAW. </p><p>The licensing deals, of course, are where Reddit is pushing. A look at their balance sheet shows that the &#8220;other revenue&#8221; line, which includes data licensing, grew 24% year over year to $43 million in Q2 2026,&nbsp;<a href="https://techcrunch.com/2026/09/30/reddit-is-killing-rss-feeds-ending-public-api-access-because-of-ai-bots/">according to TechCrunch</a>. This is the way Google and OpenAI gather data, and they&#8217;re already paying for it.</p><p>For everyone else, the web version is what remains, and today it&#8217;s already behind a JavaScript check. In my opinion, from next spring Reddit scraping becomes a browser job with good residential proxies, and probably with logged-in accounts for anything that goes beyond the public listing. That&#8217;s way more expensive per page than an RSS feed, so I expect many small social listening tools and research projects to pay for a vendor or to stop collecting.</p><h2>5.6 billion TikTok videos on Hugging Face</h2><p>On October 3, a user called datasocial published <a href="https://huggingface.co/datasets/datasocial/tiktok-5.6B-videos">tiktok-5.6B-videos</a> on Hugging Face. Yes, it&#8217;s a dataset about 5.6 billion videos of TikTok, but there are just the metadata, not the videos themselves. </p><p>This means we have ID, publish date, country, language, caption, hashtags, mentions, on-screen text, sound ID, ad and TikTok Shop flags, the AI-generated flag when TikTok provides it, and the counts of views, likes, comments, shares, saves and downloads, with the date when they were read. The files are 148 parquet files, one per month from July 2014 to October 2026, for 460.5 GB in total.</p><h3>How it was collected</h3><p>The dataset card links to <a href="https://datasocial.ai/writing/scraping-tiktoks-mobile-api">a long technical guide</a> on datasocial.ai, and for us this is the most interesting part. Instead of scraping the website, they used the private API of TikTok&#8217;s Android app, with 24 endpoints. To make it work, they register fake devices built from a catalog of about 250 real Android profiles, they sign every request with the same headers the app uses (X-Argus, X-Ladon and the others), and they copy the TLS fingerprint of Android with uTLS. There&#8217;s no login anywhere, so everything behind an account is out of reach.</p><p>Two details from the guide are useful for anyone who works on mobile APIs. The first is that a failure is often a clean HTTP 200 with an empty body, and the guide counts four unrelated causes that produce it. The second is the proxy: since the rate limit is per exit IP, they use a rotating datacenter proxy at about $150 per month, with success rates between 88.2% and 99.3% depending on how you rotate and retry. With this setup, they write they collected 3.23 billion creator profiles, 5.94 billion videos and 2.8 billion comments in three weeks. I didn&#8217;t verify these numbers beyond the 5.6 billion rows of the dataset, but if they&#8217;re correct, it&#8217;s an impressive result for the cost of a datacenter proxy plan.</p><p>The guide is definitely worth reading, it could be helpful for other apps too.</p><h2>LinkedIn wins against ProAPIs, and the scraper must delete everything</h2><p>On September 16, Judge P. Casey Pitts of the Northern District of California signed a <a href="https://storage.courtlistener.com/recap/gov.uscourts.cand.457309/gov.uscourts.cand.457309.36.0.pdf">final judgment on consent</a> in the case LinkedIn Corporation v. ProAPIs Inc. (I found it through <a href="https://therecord.media/linkedin-wins-court-order-blocking-mass-scraping">The Record</a>, then I read the documents on <a href="https://www.courtlistener.com/docket/71527603/linkedin-corporation-v-proapis-inc/">CourtListener</a>). The defendants are ProAPIs, a Delaware company, Netswift, a Pakistani company founded in 2022, and their CEO Rehmat Alam. </p><p>LinkedIn filed <a href="https://storage.courtlistener.com/recap/gov.uscourts.cand.457309/gov.uscourts.cand.457309.1.0.pdf">the complaint</a> on October 2, 2025, so the case lasted less than a year and never went to trial. The defendants never filed an answer and still deny liability, but they agreed to the judgment as part of a settlement.</p><p>The complaint is the interesting part for us, because it describes the business in detail. ProAPIs sold the iScraper API, which promised to &#8220;scrape LinkedIn data efficiently in real-time and at scale&#8221; at up to &#8220;150 requests / second&#8221;, with a plan at $15,000 per month for 5,000,000 API calls. According to LinkedIn, the data came from fake accounts: they created email addresses with automated tools, registered LinkedIn accounts with false names and stock photos, and used them to scrape while logged in. LinkedIn writes that it detects and restricts these accounts &#8220;within a few hours&#8221;, but in that time each account could scrape hundreds of profiles, and ProAPIs registered &#8220;hundreds if not thousands of new accounts per day&#8221;. In total, LinkedIn says they used more than a million fake accounts. They also scraped the profiles that members set as visible only to logged-in users, which contradicts the claim of ProAPIs that they only collected &#8220;data that&#8217;s public on personal profiles&#8221;.</p><p>It&#8217;s a clear case of massive scraping behind the login, something that is dangerous to do from a legal perspective, and this case confirms it.</p><p>The defendants are permanently barred from accessing LinkedIn with fake accounts, scrapers or bots, from circumventing any technical measure, and from selling or giving away LinkedIn data or any information derived from it (the text says &#8220;including for free&#8221;). They also can&#8217;t use any third-party service that provides LinkedIn data or accesses LinkedIn in an automated way, so they can&#8217;t simply become customers of another vendor. They certified, under penalty of perjury, that they deleted all the data, the reports made with it, the scraping code and the SDKs and APIs used to deliver it, and that they removed from their website every reference to LinkedIn scraping, &#8220;regardless of whether LinkedIn is referred to by name&#8221;. </p><p>For me, this case follows a pattern we have already seen. In the hiQ case, the court found a breach of the User Agreement for the same reason: the fake accounts used to scrape behind the login (I wrote about it in <a href="https://www.scraping.club/p/scraping-linkedin-public-data">my article on LinkedIn&#8217;s hidden APIs</a>). Proxycurl was sued by LinkedIn in January 2025 and <a href="https://nubela.co/blog/goodbye-proxycurl/">shut down on July 4, 2025</a>, and its founder explained that even winning wouldn&#8217;t pay the legal fees, while &#8220;LinkedIn, owned by Microsoft, has more or less an unlimited war chest&#8221;. ProAPIs settled in less than a year. In all three cases, the weak point is the same: when you create an account, you accept the User Agreement, and from that moment scraping becomes a breach of contract, before any discussion about public data. So if you buy LinkedIn data from a vendor today, ask them where it comes from, because a judgment like this one can close that source in a few months.</p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><p>For this month, that&#8217;s it; see you in November with the next edition of the Crawl of Fame.</p>]]></content:encoded></item><item><title><![CDATA[Can You Scrape Real-Time Market Data for Profitable ML Trading?]]></title><description><![CDATA[Making the case on the challenges associated on scraping financial real-time data, feeding ML models with it, and understanding if you can make profitable bids on financial markets.]]></description><link>https://www.scraping.club/p/scrape-real-time-market-data-ml-trading</link><guid isPermaLink="false">https://www.scraping.club/p/scrape-real-time-market-data-ml-trading</guid><dc:creator><![CDATA[Federico Trotta]]></dc:creator><pubDate>Sun, 11 Oct 2026 18:57:06 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/6c9b094d-c71f-484b-9e55-4d7561041ad5_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>So, you&#8217;re thinking about scraping real-time financial data, using it to train a machine-learning model, and making profitable trades based on scientific predictions. Sounds like a great plan, right?!</p><p>Well, unfortunately, it&#8217;s not that easy&#8230;Yeah, I know: I&#8217;m already delivering bad news, and we haven&#8217;t even started yet!</p><p>The truth is that collecting real-time market data involves much more than repeatedly retrieving a stock price from a website. And even when you manage to solve that, you still have to deal with the machine-learning part, which is not a walk in the park.</p><p>So, yes: scraping real-time financial data to train ML models is possible. But turning that process into reliable and profitable trading decisions is a completely different story.</p><p>In this article, I&#8217;ll guide you through the main challenges involved in the whole process. Let&#8217;s get started!</p><div><hr></div><blockquote><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/rE0PQB">Decodo&#8217;s Web Scraping API</a></strong> turns any site into clean, structured data your models can actually use. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/rE0PQB" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aRfI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png" width="724" height="362" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:512,&quot;width&quot;:1024,&quot;resizeWidth&quot;:724,&quot;bytes&quot;:716985,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/rE0PQB&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://substack.thewebscraping.club/i/204450518?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!aRfI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/rE0PQB&quot;,&quot;text&quot;:&quot;Activate starter plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/rE0PQB"><span>Activate starter plan</span></a></p><div><hr></div></blockquote><h2>Defining Real-Time Market Data</h2><p>Before building a stock market scraper, you need a precise definition of what &#8220;real-time&#8221; means. The term sounds straightforward, but the reality is that &#8220;real-time&#8221; means different things in different scenarios (and also in different industries, to be precise).</p><p>In finance, data can pass through several systems before reaching your application. This means that a price displayed on a website may already be seconds - or even minutes - behind the event that occurred at an exchange.</p><h3>What &#8220;Real-Time&#8221; Actually Means</h3><p>Let&#8217;s specifically focus only on financial data. In this context, real-time data is updated live in a time interval that is as close as possible to the moment an exchange processes a financial event. A typical example is the live value of a stock price.</p><p>On the other hand, delayed data is intentionally published after a fixed interval. Depending on the source and market, the delay may be several minutes or longer. Delayed data is useful for educational tools, historical analysis, and applications that don&#8217;t support immediate decisions. And note that the same financial exchange reports both the live value of a financial index and its historical value.</p><p>A common example is the USD/EUR daily rate. The currency exchange values change over time, just like stock prices do:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!RgnC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!RgnC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png 424w, https://substackcdn.com/image/fetch/$s_!RgnC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png 848w, https://substackcdn.com/image/fetch/$s_!RgnC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png 1272w, https://substackcdn.com/image/fetch/$s_!RgnC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!RgnC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png" width="1428" height="832" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:832,&quot;width&quot;:1428,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;USD/EUR exchange value live by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="USD/EUR exchange value live by Federico Trotta" title="USD/EUR exchange value live by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!RgnC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png 424w, https://substackcdn.com/image/fetch/$s_!RgnC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png 848w, https://substackcdn.com/image/fetch/$s_!RgnC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png 1272w, https://substackcdn.com/image/fetch/$s_!RgnC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2cfff9a4-15cc-4d11-96a7-3dd374c832e9_1428x832.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">USD/EUR exchange value live.</figcaption></figure></div><p>However, if you are a freelancer and need to send an invoice in USD to your client, but your government wants it in EUR because you&#8217;re in Europe, then you&#8217;ll need to refer to the daily rate. Every day, the European Central Bank defines the daily USD/EUR exchange rate, and publishes it on its website at a fixed and predefined hour (and so do Yahoo Finance and the other providers):</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Vevj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Vevj!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png 424w, https://substackcdn.com/image/fetch/$s_!Vevj!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png 848w, https://substackcdn.com/image/fetch/$s_!Vevj!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png 1272w, https://substackcdn.com/image/fetch/$s_!Vevj!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Vevj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png" width="1030" height="837" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:837,&quot;width&quot;:1030,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;EUR vs reference exchanges' daily rates by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="EUR vs reference exchanges' daily rates by Federico Trotta" title="EUR vs reference exchanges' daily rates by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!Vevj!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png 424w, https://substackcdn.com/image/fetch/$s_!Vevj!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png 848w, https://substackcdn.com/image/fetch/$s_!Vevj!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png 1272w, https://substackcdn.com/image/fetch/$s_!Vevj!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92b1b173-3d77-4e37-a2ac-10b9bddb7d0c_1030x837.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">EUR vs. reference exchanges' daily rates.</figcaption></figure></div><p>So that is the distinction between real-time and historical data. But there&#8217;s another category that falls between those two: near-real-time. This kind of data arrives quickly enough for dashboards, monitoring, and many analytical workloads, yet it doesn&#8217;t guarantee immediate delivery. For example, a scraper polling a financial website every ten seconds can only capture changes after the website publishes them and the next request runs.</p><div><hr></div><blockquote><p><em>Tired of getting blocked while scraping the web?<br><strong><a href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club">ScrapingBee</a></strong> handles proxies, browsers, anti-bot systems, and retries so you can focus on your data. Get clean Markdown, JSON, or HTML from the web with up to a 99.9% success rate. ScrapingBee is SOC 2 Type II and GDPR compliant, and trusted by 4,000+ developers.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 424w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 848w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png" width="1456" height="437" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:437,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:128210,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/216891176?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 424w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 848w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club&quot;,&quot;text&quot;:&quot;Discover ScrapingBee&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club"><span>Discover ScrapingBee</span></a></p></blockquote><div><hr></div><p>So, the line of differentiation between real-time and near-real-time is very fine. Also, note that this distinction can be affected simply by total latency. In other words, an exchange&#8217;s publication delay, vendor processing, network transmission, your scraping interval, parsing, queueing, and database writes affect the total time it takes for data to &#8220;travel&#8221; from source to storage or visualization. Which means two things:</p><ul><li><p>Increasing the scraping frequency won&#8217;t eliminate latency that already exists upstream.</p></li><li><p>The line between actual real-time data and near-real-time often depends on things you can&#8217;t manage.</p></li></ul><p>As a final consideration, note that you&#8217;ll encounter inconsistent terminology. For instance, exchanges may define real-time according to feed-distribution rules, while brokers may focus on what customers see in their trading interfaces. Financial websites sometimes label a page &#8220;live&#8221; even when updates are delayed, cached, or refreshed periodically.</p><p>Overall, as you can understand, the concept of &#8220;real-time&#8221;, &#8220;near real-time&#8221; and &#8220;delayed time&#8221; is tied to your specific use case, but it also depends on conditions you often can&#8217;t manage.</p><h3>Common Data Types and Use Cases</h3><p>Regarding the &#8220;real-time&#8221; thing, you also have to consider that financial markets include more than just the latest traded price:</p><ul><li><p><strong>Trade records</strong> describe completed transactions.</p></li><li><p><strong>Quotes</strong> contain available bid and ask prices.</p></li><li><p><strong>Order books</strong> show buy and sell interest across multiple price levels.</p></li></ul><p>Other useful inputs include trading volume, market indices, corporate actions such as dividends and stock splits, and time-sensitive financial news.</p><p>This means that different applications require different levels of freshness. For example:</p><ul><li><p>A public dashboard may tolerate updates every few seconds or minutes.</p></li><li><p>A price alert needs sufficiently frequent collection to detect a threshold crossing before the information becomes irrelevant.</p></li><li><p>A machine-learning prediction system may require synchronized, high-frequency observations from several sources.</p></li></ul><p>In other words, your scraping architecture should follow the actual use case you need. So, don&#8217;t begin by scraping as quickly as possible. First define the acceptable latency, required history, and reliability target; then design the collection pipeline around them.</p><h2>Evaluating Sources, Access Methods, and Constraints</h2><p>Choosing a market-data source isn&#8217;t simply a matter of finding a page that displays the right number. If you want to scrape financial data, you have to compare public websites with official APIs, broker endpoints, commercial data vendors, and direct exchange feeds to be as precise as possible.</p><p>This is because each option provides a different balance of freshness, reliability, complexity, cost, and permitted use. And you need to account for those before deciding the path you want to pursue when scraping their data.</p><div><hr></div><blockquote><p><em>Reliable access is the foundation of any scraping project. <strong><a href="https://iproyal.com/pricing/residential-proxies/?r=380132">IPRoyal</a></strong> offers 64M+ residential IPs to help you reach any website without blocks or bans.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://iproyal.com/pricing/residential-proxies/?r=380132" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uAvv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uAvv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png" width="1024" height="512" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:512,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:506839,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://iproyal.com/pricing/residential-proxies/?r=380132&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215976609?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!uAvv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://iproyal.com/pricing/residential-proxies/?r=380132&quot;,&quot;text&quot;:&quot;Visit IPRoyal&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://iproyal.com/pricing/residential-proxies/?r=380132"><span>Visit IPRoyal</span></a></p><div><hr></div></blockquote><h3>Choosing Between Directly Scraping Websites or Using Their APIs</h3><p>Public financial websites are a common starting point for scraping real-time financial data. Indeed, their pages expose prices through rendered HTML, embedded JSON, background HTTP requests, or streaming connections. The trade-off is the same you already know if you&#8217;ve been scraping for a while: page structures can change without warning, fields may disappear, and anti-automation controls can interrupt retrieval.</p><p>Now, if you&#8217;ve been scraping for a while, I bet that your first thought is to start by inspecting the target page. If that&#8217;s so, I have bad news for you. This (common) procedure is probably the hardest thing to do when it comes to scraping real-time data. This is because often real-time data is rendered via internal requests that you can not intercept. Here is an example from Yahoo Finance:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!dCE9!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dCE9!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png 424w, https://substackcdn.com/image/fetch/$s_!dCE9!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png 848w, https://substackcdn.com/image/fetch/$s_!dCE9!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png 1272w, https://substackcdn.com/image/fetch/$s_!dCE9!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!dCE9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png" width="1456" height="751" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:751,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!dCE9!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png 424w, https://substackcdn.com/image/fetch/$s_!dCE9!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png 848w, https://substackcdn.com/image/fetch/$s_!dCE9!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png 1272w, https://substackcdn.com/image/fetch/$s_!dCE9!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2259b9e1-c5df-4a78-b1eb-2a1274cb8190_1687x870.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Graphs are displayed from internal requests on Yahoo Finance.</figcaption></figure></div><p>On the other hand, historical data are always presented in tables, so they&#8217;re easier to retrieve:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sZDf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sZDf!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png 424w, https://substackcdn.com/image/fetch/$s_!sZDf!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png 848w, https://substackcdn.com/image/fetch/$s_!sZDf!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png 1272w, https://substackcdn.com/image/fetch/$s_!sZDf!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sZDf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png" width="1432" height="761" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:761,&quot;width&quot;:1432,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Historical data on Yahoo Finance by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Historical data on Yahoo Finance by Federico Trotta" title="Historical data on Yahoo Finance by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!sZDf!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png 424w, https://substackcdn.com/image/fetch/$s_!sZDf!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png 848w, https://substackcdn.com/image/fetch/$s_!sZDf!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png 1272w, https://substackcdn.com/image/fetch/$s_!sZDf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F61c0adad-feb0-43ff-a090-0509d39fad5f_1432x761.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Historical data on Yahoo Finance.</figcaption></figure></div><p>For this reason, another scraping option you have is websites&#8217; official APIs (hopefully, well-documented&#8230;). This makes them generally more stable - when they exist - than scraping the rendered HTML. The downside is that APIs may impose request quotas, restrict historical access, provide delayed data under lower subscription tiers, and ask you to pay for a monthly subscription or per request.</p><p>So, when comparing data sources, you should assess at least the following six characteristics, instead of rushing for &#8220;the best scraping method&#8221;:</p><ul><li><p><strong>Latency:</strong> How much time passes between a market event and its availability from the source?</p></li><li><p><strong>Coverage:</strong> Which exchanges, securities, fields, trading sessions, and historical periods are included?</p></li><li><p><strong>Reliability:</strong> Does the source provide consistent availability, ordering, timestamps, and recovery after interruptions?</p></li><li><p><strong>Documentation:</strong> Are schemas, status codes, limits, and update policies explained clearly?</p></li><li><p><strong>Authentication:</strong> Does access require API keys, account credentials, tokens, or market-data entitlements?</p></li><li><p><strong>Long-term stability:</strong> Is the interface versioned, supported, and likely to remain available?</p></li></ul><p>And what&#8217;s important to bear in mind is that visible real-time prices on a website may differ from the prices you get via the APIs. The technical scenarios under the hood are mainly the following two:</p><ul><li><p>A website dashboard might update through a dedicated streaming channel, while its public API returns cached or delayed values.</p></li><li><p>Conversely, an API may provide fresher structured records while the browser displays a throttled view.</p></li></ul><p>This means that, unfortunately, you can&#8217;t always trust what you see, when you are making a comparison between what a website displays on its dashboard and what its API returns. So, what you should do is validate each interface independently, and choose one depending on the characteristics listed above. This is because being consistent (using the same source) is more important than understanding which of the two sources reports data that is &#8220;more real-time&#8221; than the other (when the temporal difference matters, of course).</p><p>As an example, the following image shows a table from Yahoo Finance reporting the timing they display the data in their UI&#8217;s dashboards:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!GRVc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!GRVc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png 424w, https://substackcdn.com/image/fetch/$s_!GRVc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png 848w, https://substackcdn.com/image/fetch/$s_!GRVc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png 1272w, https://substackcdn.com/image/fetch/$s_!GRVc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!GRVc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png" width="1170" height="842" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:842,&quot;width&quot;:1170,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Yahoo Finance&#8217;s declared data delay by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Yahoo Finance&#8217;s declared data delay by Federico Trotta" title="Yahoo Finance&#8217;s declared data delay by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!GRVc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png 424w, https://substackcdn.com/image/fetch/$s_!GRVc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png 848w, https://substackcdn.com/image/fetch/$s_!GRVc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png 1272w, https://substackcdn.com/image/fetch/$s_!GRVc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3d89a5a-6876-465c-b5a4-5c7aca96055d_1170x842.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Yahoo Finance&#8217;s declared data delay.</figcaption></figure></div><p>So, just because you see indexes displayed on a website, you can&#8217;t assume they&#8217;re always in real-time.</p><h3>Legal, Licensing, and Operational Boundaries</h3><p>Another thing to do before scraping is reviewing the source&#8217;s terms of service, exchange licensing conditions, attribution requirements, and restrictions on storage and redistribution. This is because market data can carry contractual and licensing obligations even when it appears on a publicly accessible page. Some providers permit personal viewing but restrict automated collection, commercial use, model training, or republishing. For example, below is <a href="https://help.yahoo.com/kb/finance-for-web/SLN2310.html?locale=en_US">Yahoo Finance&#8217;s data disclaimer</a>:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!H8TP!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!H8TP!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png 424w, https://substackcdn.com/image/fetch/$s_!H8TP!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png 848w, https://substackcdn.com/image/fetch/$s_!H8TP!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png 1272w, https://substackcdn.com/image/fetch/$s_!H8TP!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!H8TP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png" width="1168" height="703" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:703,&quot;width&quot;:1168,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Yahoo Finance&#8217;s data disclaimer by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Yahoo Finance&#8217;s data disclaimer by Federico Trotta" title="Yahoo Finance&#8217;s data disclaimer by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!H8TP!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png 424w, https://substackcdn.com/image/fetch/$s_!H8TP!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png 848w, https://substackcdn.com/image/fetch/$s_!H8TP!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png 1272w, https://substackcdn.com/image/fetch/$s_!H8TP!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F49b0abc3-1dbf-4a14-a920-1a4c51ac4150_1168x703.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Yahoo Finance&#8217;s data disclaimer.</figcaption></figure></div><p>You should also inspect the site&#8217;s <span>robots.txt</span> file. As largely discussed in our article &#8220;<a href="https://www.scraping.club/p/understanding-robotstxt-and-its-implications">Understanding robots.txt and its Implications</a>&#8221;, while it isn&#8217;t a statement of legal permission, you should strive to respect its directives if you want to be at least compliant. You know, just in case you need to respond to a judge&#8230;</p><p>Rate limits require similar attention too, because you don&#8217;t want to overload the servers and get banned by anti-bots. For the sake of scraping data in real-time, it&#8217;s easy to get carried away out of fear of losing data at any moment. Generally speaking, published limits appear in API documentation or website terms of service. But if no limit is publicly stated, that doesn&#8217;t mean you can send unlimited requests. The general idea is to always <a href="https://www.scraping.club/p/how-to-scrape-open-source-datasets">scrape ethically</a>, and to do so, use conservative request frequencies, <a href="https://www.scraping.club/p/python-async-for-faster-scraping">apply async scraping best practices</a>, cache responses where appropriate, <a href="https://www.scraping.club/p/rate-limit-scraping-exponential-backoff">apply backoff after errors</a>, and avoid bypassing authentication.</p><p>Finally, verify permission before building your scraper, because accessibility and public availability aren&#8217;t the same as authorization. In other words, data being visible on a browser doesn&#8217;t automatically make unrestricted scraping, storage, redistribution, or commercial reuse acceptable.</p><h2>Challenges in Storing Real-Time and Near-Real-Time Data</h2><p>When scraping financial data in real-time, collecting stock prices frequently is only one of the challenges. Another one is that your system must also store incoming records without losing events, creating uncontrolled duplication, or slowing down as the dataset grows.</p><p>This means that there are several engineering considerations to make and decisions you have to take before starting to scrape regarding your infrastructure. And as with any technical decision, the right one depends heavily on the scenario that suits your case. Specifically speaking for financial data, the main decision revolves around whether you need to preserve every market event or maintain only the latest known state.</p><h3>Storage Challenges and Data Modeling Challenges</h3><p>If you&#8217;ve never dealt with financial data, you&#8217;ll soon discover that frequent market updates generate a surprisingly high write volume in databases. That&#8217;s easy to see if you think of the underlying processes: if a scraper records several observations per second, the database must continuously process inserts while supporting queries, validation, and downstream analysis.</p><p>And of course, adding more data sources compounds the problem, because each source may publish its own version of the same event. In such scenarios, storing duplicate observations is very common. However, note that removing duplicates based only on price is unsafe because multiple legitimate trades can occur at the same price. The practical implication of this is that reliable deduplication usually requires at least a source-provided event identifier. In a nutshell, only accounting for deduplication is a hard engineering job.</p><p>But that&#8217;s not everything&#8230;</p><p>For genuinely real-time ingestion, a streaming platform is generally a better foundation than writing every message directly into a relational database. This is because <a href="https://dzone.com/articles/an-introduction-to-stream-processing">streaming systems are designed to accept continuous event flows</a>, preserve ordering within defined partitions, buffer traffic spikes, and allow multiple consumers to process the same data independently. This way, if a database temporarily slows down, the stream can retain events in the correct temporal order.</p><p>Of course, that doesn&#8217;t mean streaming platforms replace relational databases. A stream system is the ingestion and transport layer, while relational systems provide the queryable storage. The point here is that if you need to scrape real-time data, chances are that you&#8217;ll have to use a streaming platform as the ingestion layer. And if you&#8217;ve never dealt with them, believe me: the technicalities under the hood are quite a lot (but there are specialized people on the market that can help you, right?!).</p><h3>Retention and Performance Challenges</h3><p>Data storage is important, but storing every raw tick forever is rarely practical, especially when focusing on real-time data. In such cases, a tiered retention policy lets you preserve the necessary detail while controlling cost. The main applicable ideas you can take into account as guidance are the following:</p><ul><li><p>Recent raw ticks might remain in fast storage for debugging and immediate analysis.</p></li><li><p>Daily summaries may remain available indefinitely.</p></li></ul><p>This means that, to preserve storage (and cost), historical archives should preserve the source data, schema version, transformation logic, and correction history needed to rebuild derived datasets. Nothing more.</p><p>Since you&#8217;re focusing on real-time data, there are two specific ingestion situations you have to account for when managing storage:</p><ul><li><p><strong>Late-arriving</strong>: These are events that reach your pipeline later than the expected delivery window. This may happen because of network latency, retries, buffering, source outages, or slow processing. For example, a trade executed at 10:00:05 might not reach your system until 10:01:10, after the corresponding one-minute aggregate has already been calculated.</p></li><li><p><strong>Out-of-order events</strong>: These are events that arrive in a different sequence from the one in which they occurred. For example, an event from 10:00:08 may reach your pipeline before an event from 10:00:06. This can happen when records travel through different network paths, are processed by parallel workers, or are retried after a failure.</p></li></ul><p>These two kinds of occurrences create another design decision because network delays, retries, buffering, and source corrections can cause an older event to arrive after newer ones. Event-time processing can place it in the correct historical position, but previously calculated minute bars may already be published. In such cases, you have to define a logic that allows your pipeline to decide whether to correct those aggregates, issue a new version, or preserve the original result with a quality warning.</p><p>Note that near-real-time architectures simplify many of these problems by accepting a small, controlled delay. In this model, records can be buffered briefly, validated together, reordered by event time, deduplicated, and written in batches. Recovery is also easier because failed batches can be retried without coordinating every individual event.</p><p>So, the practical thing to bear in mind is that lower latency always has a cost. If an application can tolerate updates every few seconds or minutes, near-real-time processing may deliver better reliability, simpler operations, and lower storage overhead than a fully streaming design. In other words, you must define the required freshness first (what actually &#8220;real-time data&#8221; means for your case), then choose the least complex architecture that consistently meets it.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><h2>Challenges in Training Machine-Learning Models on Real-Time Financial Data</h2><p>All right, let&#8217;s say you finalized everything. You decided what &#8220;real-time&#8221; means for you, defined the data model and the data storage, and you were also able to retrieve the financial data you want. And you&#8217;re also pretty happy with the process!</p><p>At this point, you may have a reasonable idea: if you can <a href="https://www.scraping.club/p/predictive-analytics-web-scraped-data">scrape the data in real time, you can use it to train a machine-learning model to make predictions</a> on future prices, and potentially make profitable trading decisions. That sounds interesting, right?!</p><p>Well, unfortunately, the relationship between real-time data and profitable predictions isn&#8217;t so straightforward&#8230;Yeah, other bad news, sorry about that!</p><p>Having access to a continuous stream of market data only solves the data acquisition problem. You still need to create a reliable training dataset, validate the machine learning model, and reproduce the same process in production. And even if the model predicts prices accurately, its signals may not remain profitable after accounting for the practical costs of executing trades.</p><p>Let me guide you through the challenges of this part of the process that may follow the data acquisition one.</p><h3>Building a Reliable Training Dataset</h3><p>The first problem is the real-time pipeline itself. Basically, what we discussed so far in this article is the foundation for creating your dataset of historical data. But don&#8217;t let the confusion get you here: you need to work on real-time data while scraping to create an historical dataset to train an ML model that&#8217;s able to predict future values:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Eq79!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Eq79!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png 424w, https://substackcdn.com/image/fetch/$s_!Eq79!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png 848w, https://substackcdn.com/image/fetch/$s_!Eq79!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png 1272w, https://substackcdn.com/image/fetch/$s_!Eq79!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Eq79!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png" width="430" height="423" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:423,&quot;width&quot;:430,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Historical datasets are made of real-time data by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Historical datasets are made of real-time data by Federico Trotta" title="Historical datasets are made of real-time data by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!Eq79!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png 424w, https://substackcdn.com/image/fetch/$s_!Eq79!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png 848w, https://substackcdn.com/image/fetch/$s_!Eq79!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png 1272w, https://substackcdn.com/image/fetch/$s_!Eq79!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe9d44d5e-62e5-43c6-bda3-a66bebd665d7_430x423.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">Historical datasets are made of real-time data.</figcaption></figure></div><p>In this case, you create a dataset out of real-time data. So, the real-time data becomes historical because it is stored. But for the model, they represent the value in each recorded time interval. In other words, the ML model learns to predict real-time values from historical data. And there&#8217;s no way you can do it differently.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://substack.thewebscraping.club/p/consulting&quot;,&quot;text&quot;:&quot;Need help with your scraping project?&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://substack.thewebscraping.club/p/consulting"><span>Need help with your scraping project?</span></a></p><div><hr></div><h3>Dealing With Time Series</h3><p>Creating a historical dataset with occurrences that change over time means dealing with time-series analytics:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!qpuf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!qpuf!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png 424w, https://substackcdn.com/image/fetch/$s_!qpuf!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png 848w, https://substackcdn.com/image/fetch/$s_!qpuf!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png 1272w, https://substackcdn.com/image/fetch/$s_!qpuf!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!qpuf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png" width="686" height="433" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:433,&quot;width&quot;:686,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A representation of data occurrences that change over time by Federico Trotta&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A representation of data occurrences that change over time by Federico Trotta" title="A representation of data occurrences that change over time by Federico Trotta" srcset="https://substackcdn.com/image/fetch/$s_!qpuf!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png 424w, https://substackcdn.com/image/fetch/$s_!qpuf!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png 848w, https://substackcdn.com/image/fetch/$s_!qpuf!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png 1272w, https://substackcdn.com/image/fetch/$s_!qpuf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F39e7bca3-9cdd-4082-aed3-a9689f984d54_686x433.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">A representation of data occurrences that change over time.</figcaption></figure></div><p>But time-series modeling introduces another set of difficulties. Most standard machine learning workflows assume that observations are independent and that the relationship between input features (aka, what happens at a company, etc&#8230;) and targets (aka, companies&#8217; stock prices, and any data associated with it) remains reasonably stable. However, financial time series rarely satisfy those assumptions. This is because prices, returns, volatility, and trading volume are temporally dependent, while their statistical properties can change as market conditions evolve.</p><p>Mathematically speaking, this means that a pattern learned during a low-volatility period may disappear when volatility increases. So, in that case, the ML model learned an exception instead of something general, and this will likely make it mistake future values.</p><p>So, apart from all the technical difficulties you&#8217;ve learned so far, there is one that is intrinsic to the ML industry applied to financial data - and that&#8217;s a big one.</p><h3>Defining What the Model Should Predict</h3><p>The prediction target also requires careful definition. &#8220;Predict the price&#8221; isn&#8217;t precise enough for a machine learning model. You need to specify which price you want to predict, at what future horizon, and relative to which reference value. For example, predicting the next recorded trade is different from predicting the midpoint between the best bid and ask five minutes into the future.</p><p>This choice directly affects the dataset and the usefulness of the result. An ML model can correctly predict that the price will increase while failing to estimate whether the movement will be large enough to cover the bid-ask spread. So, note that predictive accuracy and trading profitability aren&#8217;t equivalent measures.</p><h3>Evaluating the Model Without Future Information</h3><p>Model evaluation is equally delicate, because financial models should be evaluated chronologically. A typical procedure trains the model on an initial period, tests it on the next period, and then moves the training and testing windows forward.</p><p>This approach is closer to how the model would operate in production. However, even this method must account for overlapping prediction horizons and features whose calculations include rolling windows, as those can still introduce leakage near the boundaries between datasets.</p><h3>Running the Model in Production</h3><p>Finally, after training is successful, you have to deploy the ML model to production so that it can make predictions based on real-time data, the moment they&#8217;re ingested.</p><p>Technically speaking, this means that incoming records must be transformed into features using exactly the same definitions applied during training. Also, the production system must monitor feature freshness, missing inputs, inference latency, prediction distributions, and realized performance.</p><p>Without going deep into details, if you&#8217;ve ever dealt with pushing software to production, you understand the challenges under the hood. And when it comes to pushing ML models to production, the challenges increase even more.</p><p>Finally, as market conditions change and time goes by, every ML model requires recalibration or retraining. In other words, from time to time, you need to restart the whole process from the beginning&#8230;so, let me wish you good luck!</p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>Conclusion</h2><p>Overall, in this article I wanted to share the difficulties associated with scraping real-time financial data, especially if your goal is to train ML models to predict where the market is going - and eventually get some profits.</p><p>Don&#8217;t get me wrong: I&#8217;m not saying that you should not try. I wanted the message of this article to be this one: It&#8217;s very hard to put the pipeline in place, and it&#8217;s very unlikely that you&#8217;ll get good results in terms of financial satisfaction.</p><p>So, let&#8217;s discuss in the comments: Have you ever thought of scraping real-time financial data to train ML models? Have you ever tried it, other than just thinking of it?</p><div><hr></div><p><em>Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/p/scrape-real-time-market-data-ml-trading?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.scraping.club/p/scrape-real-time-market-data-ml-trading?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[Building an online grocery agent with Browserless]]></title><description><![CDATA[Let&#8217;s test antibot and CAPTCHA solving features, together with session persistence.]]></description><link>https://www.scraping.club/p/building-an-online-grocery-agent-with-browserless</link><guid isPermaLink="false">https://www.scraping.club/p/building-an-online-grocery-agent-with-browserless</guid><dc:creator><![CDATA[Pierluigi Vinciguerra]]></dc:creator><pubDate>Tue, 06 Oct 2026 18:18:27 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/022d701d-9def-4264-83a8-04ea2076b93b_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>When building an AI agent to automate an online task, we recognize the value of our local browser in terms of ease of use. We&#8217;re already logged in to most of our preferred websites; the browser&#8217;s sessions store the useful part of our most recent navigation, and we&#8217;re so used to this that we take it for granted. Our IP score is usually golden, and our hardware is legit, so we don&#8217;t get blocked on any website.</p><p>But when we build an agent, all this disappears: we need a browser-as-a-service to work for us, usually running in a Linux VM inside a data center and using its IP address, which screams red flags from the first request.</p><p>Today, for this issue of &#8220;Tool of the Week&#8221;, we&#8217;re using Browserless to build an AI agent that logs in to a grocery website in Italy and adds something to the cart, simulating an online shopping experience.</p><p>Last week Joel Griffith, founder of <a href="https://www.browserless.io/?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">Browserless</a>, wrote a <a href="https://www.scraping.club/p/authenticated-scraper-sessions">guest post here on what breaks in long-running authenticated sessions</a>. He described two Browserless features for this problem: Authenticated Profiles (a saved copy of cookies, localStorage, and IndexedDB) and <a href="https://docs.browserless.io/baas/session-management/persisting-state?utm_source=thewebscrapingclub&amp;utm_medium=tool-review&amp;utm_content=session-api">the Session API</a> (a browser identity that persists on disk for days). </p><p>We&#8217;ll also test this feature by closing and reopening the browser and will check if, by using the same profile, we&#8217;ll be able to avoid logging in for the following runs.</p><h2>The setup</h2><p><a href="https://www.browserless.io/?utm_source=thewebscrapingclub&amp;utm_medium=tool-review">Browserless</a> is a browser-as-a-service: you connect Playwright or Puppeteer to a remote Chromium over a WebSocket, and you configure it with query parameters in the URL. For these tests I used:</p><ul><li><p>the <code>/chromium/stealth</code> route (and <code>/chromium</code> as a comparison on the first target)</p></li><li><p>the built-in residential proxy with <code>proxy=residential&amp;proxyCountry=it</code>, so every session exits from an Italian IP</p></li><li><p><code>solveCaptchas=true</code>, <a href="https://docs.browserless.io/baas/bot-detection/captchas?utm_source=thewebscrapingclub&amp;utm_medium=tool-review&amp;utm_content=captchas">the native CAPTCHA solver</a>, which detects a CAPTCHA, solves it and injects the token into the page</p></li><li><p><a href="https://docs.browserless.io/baas/features/authenticated-profiles?utm_source=thewebscrapingclub&amp;utm_medium=tool-review&amp;utm_content=authenticated-profiles">Authenticated Profiles</a> and the Session API for the persistence part</p></li></ul><p>All the code is Python with Playwright, connected over CDP (<code>connect_over_cdp</code>), always reusing <code>browser.contexts[0]</code>. I do this because the Browserless docs say that profiles are applied only to the default context, and the CAPTCHA events are visible only if you reuse the existing page.</p><p>I started on the free plan and moved to the Prototyping plan during testing, since the free tier was a bit limited for me.</p><h2>First target: Esselunga</h2><p>The starting idea for this article was to log in to the Esselunga website (one of the most famous grocery chains in Italy) using my credentials.</p><p>So I requested the <code>account.esselunga.it/area-utenti/</code> page both with the Chromium endpoint and the stealth one. While the page was served on both cases, i got the TS* cookie, a signal of the presence <a href="https://www.f5.com/products/big-ip">of the F5 BIG-IP suite</a>, only with the stealth endpoint, so I completed the test with it.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bEv5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bEv5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png 424w, https://substackcdn.com/image/fetch/$s_!bEv5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png 848w, https://substackcdn.com/image/fetch/$s_!bEv5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png 1272w, https://substackcdn.com/image/fetch/$s_!bEv5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!bEv5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png" width="817" height="256" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/bd529d76-a152-491d-8660-7d8c94ed0893_817x256.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:256,&quot;width&quot;:817,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:26669,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/218985994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!bEv5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png 424w, https://substackcdn.com/image/fetch/$s_!bEv5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png 848w, https://substackcdn.com/image/fetch/$s_!bEv5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png 1272w, https://substackcdn.com/image/fetch/$s_!bEv5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbd529d76-a152-491d-8660-7d8c94ed0893_817x256.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The script types the email and password one character at a time, with a random pause of 70 to 220 ms, and waits for the <code>Browserless.captchaAutoSolved</code> event before clicking the button:</p><pre><code><code>endpoint = ws_url("chromium/stealth", timeout=120000, proxy="residential",
                  proxyCountry="it", proxySticky="true", solveCaptchas="true")
browser = await p.chromium.connect_over_cdp(endpoint)
context = browser.contexts[0]
page = context.pages[0]
cdp = await context.new_cdp_session(page)
cdp.on("Browserless.captchaAutoSolved",
       lambda e: solved.set_result(e) if not solved.done() else None)</code></code></pre><p>The native solver worked in 5 runs out of 7, with solve times between 10 and 35 seconds. In the other 2 runs it stayed in the <code>solving</code> state for 75 seconds without producing a token, and the checkbox in the screenshot was still empty. When the token arrived, the login POST went through F5 without problems (<code>POST /area-utenti/loginExt</code>, then a 302).</p><p>So, everything worked?</p><p>Well, not really in this case. The website asked for a 2FA on my phone, probably because I didn&#8217;t set any OS to emulate in the first place. My hypothesis is that the Linux identity triggered the second factor, even if I can&#8217;t prove it from the outside. But after the first 2FA request, Esselunga kept asking for it every time, even when I set the Browserless session to emulate macOS (with the <code>emulationOs=macos</code> parameter) and even from my normal browser. It seems that my account, once flagged, stays in the &#8220;ask for 2FA&#8221; state for a while, regardless of the device you use.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!BkEq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BkEq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png 424w, https://substackcdn.com/image/fetch/$s_!BkEq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png 848w, https://substackcdn.com/image/fetch/$s_!BkEq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png 1272w, https://substackcdn.com/image/fetch/$s_!BkEq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!BkEq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png" width="1295" height="1354" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1354,&quot;width&quot;:1295,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:190229,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/218985994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!BkEq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png 424w, https://substackcdn.com/image/fetch/$s_!BkEq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png 848w, https://substackcdn.com/image/fetch/$s_!BkEq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png 1272w, https://substackcdn.com/image/fetch/$s_!BkEq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb9472ee-57e2-4e70-a5cb-f257ad71fe27_1295x1354.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>For this, I decided to move on to the second target, Tigros.</p><h2>Second target: Tigros</h2><p>Tigros is another Italian grocery chain, with an online shop at <code>tigros.it/tigros-spesa-online</code>. The stack is completely different from Esselunga:</p><ul><li><p>Cloudflare is in front of the site (<code>server: cloudflare</code>, <code>cf-ray</code>)</p></li><li><p>the shop is a Vue single-page app that talks to an API under <code>/ebsn/api/</code></p></li><li><p>the CAPTCHA is reCAPTCHA Enterprise, loaded with <code>render=&lt;sitekey&gt;</code>, which means the invisible, score-based version: no checkbox at all</p></li></ul><p>To be fair, Cloudflare didn&#8217;t challenge the stealth session. It didn&#8217;t issue any <code>cf_clearance</code> cookie and the <code>/ebsn/api/*</code> calls returned 200 from the first load, while Browserless detected the CAPTCHA as <code>recaptcha-v3-enterprise</code>.</p><p>The login is a dropdown panel with email, password and a &#8220;Ricordami su questo dispositivo&#8221; (remember me) checkbox, checked by default. I used the same typing approach as before, and the first result was surprising.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!xZo2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!xZo2!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png 424w, https://substackcdn.com/image/fetch/$s_!xZo2!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png 848w, https://substackcdn.com/image/fetch/$s_!xZo2!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png 1272w, https://substackcdn.com/image/fetch/$s_!xZo2!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!xZo2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png" width="1440" height="900" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:900,&quot;width&quot;:1440,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:741335,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/218985994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!xZo2!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png 424w, https://substackcdn.com/image/fetch/$s_!xZo2!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png 848w, https://substackcdn.com/image/fetch/$s_!xZo2!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png 1272w, https://substackcdn.com/image/fetch/$s_!xZo2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd3feb43-de06-40b9-a7c5-0f3f2cf6f439_1440x900.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Since there was a CAPTCHA involved, I used the option <code>solveCaptchas=true</code> , but this led to failure on each try I made (as a side note, this is an issue the Browserless team is already solving).</p><p>Then I ran the same stealth session without<code>solveCaptchas</code>, letting the page generate its own token with <code>grecaptcha.enterprise</code>. The login returned 200 with a welcome message, 2 runs out of 2. Good!<br>Since on reCAPTCHA Enterprise your session gets a score on the server side, and you don&#8217;t know how good it was or why, your request eventually fails. We cannot say why the solveCaptchas option didn&#8217;t work. But we can surely say that a stealth browser session was good enough to get a valid token without using the solver.</p><p>Anyway, we were able to log in! Now let&#8217;s see if we&#8217;re able to access the website again without doing it.</p><h3>Saving the login as a profile</h3><p>After the successful login, I saved the browser state as an Authenticated Profile. It&#8217;s one CDP command, and it worked from a normal stealth connection, without creating a profile session first:</p><pre><code><code>saved = await cdp.send("Browserless.saveProfile", {"name": "tigros-stealth"})
# {'ok': True, 'name': 'tigros-stealth', 'cookieCount': 25, 'originCount': 1, ...}</code></code></pre><p>To check the profile, I open a new browser with <code>profile=tigros-stealth</code> in the connection URL and ask the application, not the cookie jar, if I&#8217;m logged in. Griffith suggests the same thing in his post. On Tigros, the signal I use is this: a logged-in session calls <code>/ebsn/api/cart/view</code> after load, and the header doesn&#8217;t show &#8220;Accedi/Registrati&#8221; anymore.</p><p>The profile worked. A new browser, with no login and a different exit IP (the login came from Ancona and the check from Milan), opened the shop as the logged-in user.</p><h2>Does the session survive over time?</h2><p>Browserless gives you two models here, and Griffith described the difference well. A profile is a copy by value of the state at the moment you saved it. A persisted session is one browser identity whose data on disk keeps changing. I tested both side by side. For the session, I created it with the Session API, starting from the same profile:</p><pre><code><code>body = {"ttl": 7 * 24 * 3600 * 1000, "stealth": True, "profile": "tigros-stealth",
        "proxy": {"type": "residential", "country": "it", "sticky": True}}
# POST https://production-ams.browserless.io/session?token=...
# returns connect (wss URL) and stop (DELETE URL)</code></code></pre><p>A <code>ttl</code> of 7 days is the maximum on the Prototyping plan (1 day on free). One detail from the docs: with Playwright you get the state on disk but not the live browser process between connections, because <code>processKeepAlive</code> needs <code>browser.disconnect()</code>, which Playwright doesn&#8217;t have. So every reconnection starts a new process that reloads cookies and localStorage from disk.</p><p>I checked both models from new connections, each one going through the Italian residential proxy. (I verified that the session also exits from an Italian residential IP: in my last check it was Rome, on Telecom Italia.)</p><p>Within this window the site never rotated the token, so the two models stayed identical and I can&#8217;t show a difference between them. The only expiry I know is the one written in the JWT (15 days), and I didn&#8217;t observe it. </p><p>In this particular case, the IP address change didn&#8217;t matter on Tigros, but it&#8217;s not a golden rule. Sometimes this could lead to a break in the pipeline, so usually it&#8217;s better to use a static IP.</p><h2>Doing the shopping</h2><p>The last step is acting as the user, which is the ultimate goal for our agent. Once the profile is loaded into a new browser, the script types &#8220;latte&#8221; in the header search box and clicks the cart button on the first product.</p><p>Before adding to the cart, I needed to confirm my delivery address and select a random available delivery slot. <br>Everything was smooth, and then I could finally click the add to cart button for my latte, which triggered this internal API call:</p><pre><code><code>POST /ebsn/api/cart/add
{"items":[{"productId":7728410,"quantity":1,"infos":{"accept_alternatives":false}, ...}]}</code></code></pre><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!CheI!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!CheI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png 424w, https://substackcdn.com/image/fetch/$s_!CheI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png 848w, https://substackcdn.com/image/fetch/$s_!CheI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png 1272w, https://substackcdn.com/image/fetch/$s_!CheI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!CheI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png" width="1440" height="900" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:900,&quot;width&quot;:1440,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:331190,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/218985994?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!CheI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png 424w, https://substackcdn.com/image/fetch/$s_!CheI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png 848w, https://substackcdn.com/image/fetch/$s_!CheI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png 1272w, https://substackcdn.com/image/fetch/$s_!CheI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff410ba83-e156-4d4b-8ed1-db9ad96fb938_1440x900.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>The <code>cart/view</code> endpoint confirmed one item, 1.25 euros of UHT milk. Then removed it, and we can definitely say that our agent works! I didn&#8217;t try following the purchase process but I think the actual process was enough to show the potential of Browserless for the agentic web.</p><h2>What it costs</h2><p><a href="https://www.browserless.io/pricing?utm_source=thewebscrapingclub&amp;utm_medium=tool-review&amp;utm_content=pricing">Browserless bills in units</a>: 1 unit per 30 seconds of browser time, 6 units per MB of residential proxy traffic, and 10 units per successful CAPTCHA solve (failed attempts are free). A login-state check on Tigros cost about 8 units. The cart runs were much heavier, because a full grocery page with images, when using a residential proxy, it means burning a lot of MB: all the tests after the plan upgrade, mostly cart exploration, used 914 units. If you plan to run an agent like this every day, blocking images on the pages where you don&#8217;t need them is the first thing I would do.</p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>Final considerations</h2><p>This small test confirms the progress made by the browser-as-a-service industry in building reliable tools for the agentic web era. We stopped a couple of steps from creating a fully working  grocery shopping assistant, just because we didn&#8217;t need it at the moment. The targets were not the most protected websites on the web, but this is true of most websites. I hope you enjoyed this small test, and let me know in the comments section which agent you built recently.</p>]]></content:encoded></item><item><title><![CDATA[Can invisible_playwright Beat Bot Detection? A Hands-On Test and Review]]></title><description><![CDATA[Have we finally found a good replacement for playwright-stealth, or even a strong alternative to Camoufox or Patchright?]]></description><link>https://www.scraping.club/p/invisible-playwright-beat-bot-detection-scraping</link><guid isPermaLink="false">https://www.scraping.club/p/invisible-playwright-beat-bot-detection-scraping</guid><dc:creator><![CDATA[Antonello Zanini]]></dc:creator><pubDate>Sun, 04 Oct 2026 13:14:11 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/7ebafb85-ee43-4b49-8b07-d1573adc4d8e_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>invisible_playwright is an open-source anti-detect browser engine for Playwright. It lets you write automation scripts that control Firefox while reducing the risk of bot detection.</p><p>In this post, I&#8217;ll explain what the tool is, how it works, and which features it offers. I&#8217;ll also show you how to use it, evaluate its performance against anti-bot detection systems, and compare it with other similar libraries.</p><h2>What Is invisible_playwright?</h2><p><a href="https://github.com/feder-cr/invisible_playwright">invisible_playwright</a> is a Python library created by <a href="https://it.linkedin.com/in/federico-elia-5199951b6">Federico Elia</a>. It patches Firefox at the C++ source level to help bypass bot detection systems.</p><p>The project is open source and free to use, and is designed to help avoid anti-bot systems and CAPTCHA challenges such as reCAPTCHA v3, Cloudflare Turnstile, DataDome, Kasada, Akamai Bot Manager, PerimeterX, hCaptcha, and Imperva.</p><p>The library addresses two questions that anti-bot systems typically ask:</p><ol><li><p><em>Is this a real browser?</em> Yes, as the tool employs Firefox patched at the native level.</p></li><li><p><em>Is a real person using it?</em> It simulates human-like interactions to make automated activity look more natural.</p></li></ol><p>At its core, invisible_playwright works as a drop-in replacement for Playwright. The core difference is that it combines a modified browser engine, configurable fingerprints, and built-in human-like interactions. As of this writing, the project boasts more than 2k GitHub stars!</p><p>Further reading:</p><ul><li><p><em><a href="https://github.com/feder-cr/invisible_playwright/tree/main/docs">Official documentation</a></em></p></li><li><p><em><a href="https://github.com/feder-cr/invisible_playwright/wiki">Official wiki</a></em></p></li><li><p><em><a href="https://github.com/feder-cr/invisible_playwright/blob/main/docs/guides.md">Official guides</a></em></p></li><li><p><em><a href="https://publish.obsidian.md/twsc-public/Web+Scraping/Wiki/entities/invisible-playwright">invisible_playwright TWSC wiki page</a></em></p></li></ul><div><hr></div><blockquote><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/rE0PQB">Decodo&#8217;s Web Scraping API</a></strong> turns any site into clean, structured data your models can actually use. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/rE0PQB" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aRfI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png" width="724" height="362" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:512,&quot;width&quot;:1024,&quot;resizeWidth&quot;:724,&quot;bytes&quot;:716985,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/rE0PQB&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://substack.thewebscraping.club/i/204450518?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!aRfI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/rE0PQB&quot;,&quot;text&quot;:&quot;Activate starter plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/rE0PQB"><span>Activate starter plan</span></a></p><div><hr></div></blockquote><h2>Main Features</h2><p>Now that you know what invisible_playwright is, let me introduce what it brings to the table.</p><h3>Drop-In Playwright Replacement</h3><p>invisible_playwright is 100% compatible with Playwright. That means you don&#8217;t need to rewrite your code. Both sync and async work, and every method remains the same. So, there&#8217;s no learning curve.</p><p>You can switch from standard Playwright like this:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!QLb5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea433a74-c230-44f6-9c9d-db3a321bc909_1676x262.png 424w, https://substackcdn.com/image/fetch/$s_!QLb5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea433a74-c230-44f6-9c9d-db3a321bc909_1676x262.png 848w, https://substackcdn.com/image/fetch/$s_!QLb5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea433a74-c230-44f6-9c9d-db3a321bc909_1676x262.png 1272w, https://substackcdn.com/image/fetch/$s_!QLb5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea433a74-c230-44f6-9c9d-db3a321bc909_1676x262.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!QLb5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea433a74-c230-44f6-9c9d-db3a321bc909_1676x262.png" width="1456" height="228" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ea433a74-c230-44f6-9c9d-db3a321bc909_1676x262.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:228,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Old code vs new code&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Old code vs new code" title="Old code vs new code" srcset="https://substackcdn.com/image/fetch/$s_!QLb5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea433a74-c230-44f6-9c9d-db3a321bc909_1676x262.png 424w, https://substackcdn.com/image/fetch/$s_!QLb5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea433a74-c230-44f6-9c9d-db3a321bc909_1676x262.png 848w, https://substackcdn.com/image/fetch/$s_!QLb5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea433a74-c230-44f6-9c9d-db3a321bc909_1676x262.png 1272w, https://substackcdn.com/image/fetch/$s_!QLb5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fea433a74-c230-44f6-9c9d-db3a321bc909_1676x262.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Old code vs new code</figcaption></figure></div><p>That&#8217;s the only change needed. Now, each Playwright browser automation session will automatically get a distinct fingerprint. Note that the <a href="https://www.scraping.club/p/understanding-browser-fingerprint">browser fingerprint</a> generated includes GPU, audio, fonts, screen, and roughly 200 other fields.</p><h3>Human-like movement and interaction</h3><p>invisible_playwright simulates more natural mouse interactions. As opposed to instantly moving the pointer to a target, the cursor travels toward the target along a human-like path. This behavior is controlled through the <code>humanize</code> option, which applies to actions such as clicking, hovering, and dragging.</p><p>Enabled by default (<code>humanize=True</code>), you can disable it by setting it to <code>False</code>:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">with InvisiblePlaywright(humanize=True) as browser:
    page = browser.new_page()
    # Browser automation logic....</code></pre></div><h3>Fingerprint Seeds for Reproducibility</h3><p>To make runs reproducible, you can pin a specific fingerprint. This helps with debugging, as if a scrape fails with a specific seed, you can replay it exactly.</p><p>You can use this feature like this:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python"># Initialize the InvisiblePlaywright engine
sf = InvisiblePlaywright()

# Create a new Playwright-like browser instance
with sf as browser:
    # Access the randomly assigned fingerprint for each run.
    print("seed =", sf.seed)

    page = browser.new_page()
    # Browser automation logic....</code></pre></div><p>This will produce an output like this:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sjLe!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7982e1db-490b-4eee-9804-b17cc5764d0c_2150x38.png 424w, https://substackcdn.com/image/fetch/$s_!sjLe!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7982e1db-490b-4eee-9804-b17cc5764d0c_2150x38.png 848w, https://substackcdn.com/image/fetch/$s_!sjLe!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7982e1db-490b-4eee-9804-b17cc5764d0c_2150x38.png 1272w, https://substackcdn.com/image/fetch/$s_!sjLe!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7982e1db-490b-4eee-9804-b17cc5764d0c_2150x38.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sjLe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7982e1db-490b-4eee-9804-b17cc5764d0c_2150x38.png" width="1456" height="26" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7982e1db-490b-4eee-9804-b17cc5764d0c_2150x38.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:26,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Note the seed number&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Note the seed number" title="Note the seed number" srcset="https://substackcdn.com/image/fetch/$s_!sjLe!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7982e1db-490b-4eee-9804-b17cc5764d0c_2150x38.png 424w, https://substackcdn.com/image/fetch/$s_!sjLe!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7982e1db-490b-4eee-9804-b17cc5764d0c_2150x38.png 848w, https://substackcdn.com/image/fetch/$s_!sjLe!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7982e1db-490b-4eee-9804-b17cc5764d0c_2150x38.png 1272w, https://substackcdn.com/image/fetch/$s_!sjLe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7982e1db-490b-4eee-9804-b17cc5764d0c_2150x38.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">Note the seed number</figcaption></figure></div><p>Then, you can reuse the same fingerprint with:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python"># Same fingerprint every time
with InvisiblePlaywright(seed=1561645783) as browser:
    page = browser.new_page()
    # Browser automation logic....</code></pre></div><p>The seed-based fingerprint generation adopted by invisible_playwright is deterministic. In other words, you are not rolling the dice on every run. Instead, the fingerprint is derived from a known seed value, making the same fingerprint reproducible across runs. This makes bugs easier to reproduce and debug, which is particularly useful when <a href="https://www.scraping.club/p/python-async-for-faster-scraping">scraping at scale</a>.</p><h3>Proxy Support</h3><p>invisible_playwright natively supports SOCKS5, SOCKS4, HTTP, and HTTPS proxies. You can configure a proxy as follows:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">proxy = {
    "server": "http://your-proxy-domain.com:1234",
    "username": "my_wonderful_username",
    "password": "my_very_secret_password",
}
with InvisiblePlaywright(proxy=proxy) as browser:
    page = browser.new_page()
    page.goto("https://example.com")</code></pre></div><p>DNS traffic is routed through the proxy by default, helping prevent local DNS leaks. The timezone is automatically derived from the proxy&#8217;s exit IP, keeping the browser configuration consistent with the proxy location.</p><p>If needed, you can override it by specifying a custom timezone:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">with InvisiblePlaywright(proxy=proxy, timezone="Europe/Rome") as browser:
    #  Browser automation logic....</code></pre></div><div><hr></div><blockquote><p><em>Reliable access is the foundation of any scraping project. <strong><a href="https://iproyal.com/pricing/residential-proxies/?r=380132">IPRoyal</a></strong> offers 64M+ residential IPs to help you reach any website without blocks or bans.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://iproyal.com/pricing/residential-proxies/?r=380132" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!uAvv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!uAvv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png" width="1024" height="512" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:512,&quot;width&quot;:1024,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:506839,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://iproyal.com/pricing/residential-proxies/?r=380132&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/215976609?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!uAvv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!uAvv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3991eff8-5610-424e-a59f-fa2f006ee3f3_1024x512.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://iproyal.com/pricing/residential-proxies/?r=380132&quot;,&quot;text&quot;:&quot;Visit IPRoyal&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://iproyal.com/pricing/residential-proxies/?r=380132"><span>Visit IPRoyal</span></a></p></blockquote><div><hr></div><h3>Custom Fingerprint Fields</h3><p>invisible_playwright lets you pin specific fingerprint fields while keeping the remaining fields derived from the seed. You can do that through the <code>pin</code> parameter:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">with InvisiblePlaywright(
    seed=1561645783,
    pin={
        "gpu.renderer": "ANGLE (AMD, AMD Radeon RX 7900 XTX Direct3D11)",
        "gpu.vendor": "Google Inc. (AMD)",
        "screen.width": 1920,
        "screen.height": 1080,
        "hardware.concurrency": 8,
    },
) as browser:
    page = browser.new_page()
    # Browser automation logic...</code></pre></div><p>In this example, the GPU, screen resolution, and CPU core count remain fixed, while other fingerprint fields continue to be generated from the seed. That&#8217;s useful when you want to target a specific device configuration without manually defining the entire fingerprint.</p><h3>CLI Usage</h3><p>The <a href="https://pypi.org/project/invisible-playwright/">PyIP </a><code>invisible-playwright</code><a href="https://pypi.org/project/invisible-playwright/"> package</a> installs an <code>invisible-playwright</code> CLI command. The main supported commands are:</p><ul><li><p><code>invisible-playwright fetch</code>: Downloads the patched browser engine if it&#8217;s not already installed. The download is approximately 240&#8211;265 MB, depending on your platform. The engine is cached locally after the first download, and the command also verifies its SHA-256 hash.</p></li><li><p><code>invisible-playwright version</code>: Displays version information for the patched browser engine, the invisible_playwright library, and other information.</p></li></ul><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!eV3A!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd77f4bd4-9b81-4998-bb82-3b863145e566_2760x184.png 424w, https://substackcdn.com/image/fetch/$s_!eV3A!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd77f4bd4-9b81-4998-bb82-3b863145e566_2760x184.png 848w, https://substackcdn.com/image/fetch/$s_!eV3A!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd77f4bd4-9b81-4998-bb82-3b863145e566_2760x184.png 1272w, https://substackcdn.com/image/fetch/$s_!eV3A!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd77f4bd4-9b81-4998-bb82-3b863145e566_2760x184.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!eV3A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd77f4bd4-9b81-4998-bb82-3b863145e566_2760x184.png" width="1456" height="97" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d77f4bd4-9b81-4998-bb82-3b863145e566_2760x184.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:97,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;The output of the &#8220;invisible-playwright version&#8221; command&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="The output of the &#8220;invisible-playwright version&#8221; command" title="The output of the &#8220;invisible-playwright version&#8221; command" srcset="https://substackcdn.com/image/fetch/$s_!eV3A!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd77f4bd4-9b81-4998-bb82-3b863145e566_2760x184.png 424w, https://substackcdn.com/image/fetch/$s_!eV3A!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd77f4bd4-9b81-4998-bb82-3b863145e566_2760x184.png 848w, https://substackcdn.com/image/fetch/$s_!eV3A!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd77f4bd4-9b81-4998-bb82-3b863145e566_2760x184.png 1272w, https://substackcdn.com/image/fetch/$s_!eV3A!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd77f4bd4-9b81-4998-bb82-3b863145e566_2760x184.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">The output of the &#8220;invisible-playwright version&#8221; command</figcaption></figure></div><h3>MCP Server</h3><p>invisible_playwright can also run as an <a href="https://github.com/feder-cr/AIHawk/wiki/mcp-server">MCP server</a>, giving Claude Code, Claude Desktop, Cursor, Codex, and all other MCP-compatible clients access to a real stealth Firefox browser.</p><p>The exposed MCP server supports navigation, page reading, clicking, typing, screenshots, and JavaScript-based data extraction. You can register it in Claude Code with:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;bash&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-bash">claude mcp add stealth -- uvx aihawk</code></pre></div><p>The server is bundled via <a href="https://github.com/feder-cr/AIHawk">AIHawk</a>, which also provides a UI and OpenRouter support for users who prefer not to work directly from the command line.</p><h3>Multiple Integrations</h3><p>invisible_playwright works with a long list of Python frameworks:</p><ul><li><p>Scrapy, via the <code>scrapy-playwright</code> middleware.</p></li><li><p>Crawlee for Python, with a plugin subclass or single-line setup.</p></li><li><p>Playwright MCP, with two flags on the Microsoft server.</p></li><li><p>CodeceptJS, through the <code>firefox</code> block configuration.</p></li><li><p>Robot Framework Browser, with named keyword arguments.</p></li><li><p><a href="https://github.com/feder-cr/invisible_playwright/tree/main/docs/integrations">And more&#8230;</a></p></li></ul><h2>How invisible_playwright Avoids Anti-Bot Triggering</h2><p>Anti-bot systems generally rely on two main signals to detect automated browsers:</p><ol><li><p>Browser identity, via fingerprinting signals.</p></li><li><p>Interaction behavior, such as whether mouse movements and clicks resemble those of a real user.</p></li></ol><p>invisible_playwright addresses both areas by patching Firefox at the source level and supporting human-like interactions. Discover how!</p><h3>Browser Identity and C++ Level Patching</h3><p>Vanilla Playwright relies on the host machine&#8217;s native browser fingerprint and the default configuration of a fresh browser session. Signals like GPU, installed fonts, screen resolution, canvas rendering, and other browser properties can make automated sessions easier to identify.</p><p>invisible_playwright addresses this by <a href="https://www.scraping.club/p/fingerprint-injection-playwright">patching Firefox at the source-code level</a>. Instead of injecting spoofed values through page-level JavaScript, it sets fingerprint data inside the browser engine before it becomes accessible to scripts.</p><p>Rather than <a href="https://www.scraping.club/p/browser-api-an-introduction">simply spoofing </a><code>navigator.webdriver</code> or injecting modified canvas data, the library configures several fingerprint surfaces, including:</p><ul><li><p>GPU and WebGL values that are coherent and matching.</p></li><li><p>Canvas and WebGL rendering that&#8217;s deterministic per seed.</p></li><li><p>Fonts bundled with the browser for consistency across platforms.</p></li><li><p>Audio context data from the profile instead of the host device.</p></li><li><p>WebRTC that offers a synthetic ICE candidate.</p></li><li><p>Timezone and DNS routed through the proxy.</p></li><li><p>Input events with proper <code>isTrusted</code> values.</p></li></ul><p>These properties are derived from a single seed, helping keep different fingerprint surfaces consistent with one another. Anti-bot systems can often identify spoofed environments when individual signals contradict each other, such as a browser reporting Windows-related properties alongside macOS-specific characteristics.</p><h3>Human-Like Input and Event Fields</h3><p>Most humanization libraries rely on <a href="https://pomax.github.io/bezierinfo/">B&#233;zier curves</a> to generate smooth mouse movements from point A to point B instead of instantly teleporting the cursor. That&#8217;s an improvement, but it may not be enough on its own. After all, a perfectly smooth curve with uniform timing can still look artificial.</p><p>invisible_playwright goes further by generating B&#233;zier-curve mouse paths from the selected seed. This makes movement patterns vary between browser identities while remaining reproducible for debugging.</p><p>It also patches the browser&#8217;s native input handling so pointer events contain more realistic values, including:</p><ul><li><p><strong>Variable timing</strong>: Movement timing adapts to the device&#8217;s refresh rate instead of following a uniform schedule.</p></li><li><p><strong>Appropriate pressure</strong>: Pressure is set to <code>0</code> while hovering and to a non-zero value when clicking.</p></li><li><p><strong>Correct pointer type</strong>: <code>pointerType</code> is consistently set to <code>mouse</code>.</p></li><li><p><strong>Accurate movement deltas</strong>: <code>movementX</code> and <code>movementY</code> reflect the actual pointer displacement.</p></li><li><p><strong>Sub-pixel coordinates</strong>: Pointer positions can account for fractional coordinates on scaled displays.</p></li></ul><p>These properties are determined by the browser&#8217;s native input-handling code, rather than being added or modified by the automation driver. Such a level of control requires patching the browser binary.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><h2>invisible_playwright Against Popular Bot Detection Tests</h2><p>In this section, I&#8217;ll show you how to set up invisible_playwright and use it to score its performance against well-known bot detection tests:</p><ul><li><p><em><a href="https://antcpt.com/score_detector/">AntCpt.com&#8217;s Score Detector for reCAPTCHA v3</a></em></p></li><li><p><em><a href="https://www.apivoid.com/tools/bot-detection-test/">APIVoid&#8217;s Bot Detection Test</a></em></p></li><li><p><em><a href="https://cleantalk.org/am-i-a-bot">CleanTalk&#8217;s Am I a Bot?</a></em></p></li><li><p><em><a href="https://pixelscan.net/bot-check">Pixelscan&#8217;s Bot Detection Test</a></em></p></li></ul><h3>Prerequisites: Project Setup and Script Overview</h3><p>To use <code>invisible_playwright</code>, you need:</p><ul><li><p>Python 3.11 or newer.</p></li><li><p>Windows x86_64 or Linux x86_64/ARM64.</p></li></ul><p>In your Python project, install invisible_playwright with:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;bash&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-bash">pip install invisible-playwright</code></pre></div><p>Then, download the patched browser engine using the CLI:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;bash&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-bash">invisible-playwright fetch</code></pre></div><p>You can now use invisible_playwright much like standard Playwright. For example, the following script opens a bot detection test page, waits 10 seconds for the test to complete, and takes a screenshot of the result:</p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;python&quot;,&quot;nodeId&quot;:null}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-python">from invisible_playwright import InvisiblePlaywright

# Create a new InvisiblePlaywright instance
with InvisiblePlaywright() as browser:
    page = browser.new_page()

    # Visit the bot detection test and wait for 10 seconds
    page.goto("&lt;BOT_DETECTION_PAGE_TEST_URL&gt;")
    page.wait_for_timeout(10000)

    # Take a screenshot of the entire page
    page.screenshot(path="antibot-challenge.png")</code></pre></div><h3>Vs Fingerprint</h3><p>The result was:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!5Pf3!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5Pf3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png 424w, https://substackcdn.com/image/fetch/$s_!5Pf3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png 848w, https://substackcdn.com/image/fetch/$s_!5Pf3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png 1272w, https://substackcdn.com/image/fetch/$s_!5Pf3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!5Pf3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png" width="1456" height="718" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:718,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;invisible_playwright vs Fingerprint&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="invisible_playwright vs Fingerprint" title="invisible_playwright vs Fingerprint" srcset="https://substackcdn.com/image/fetch/$s_!5Pf3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png 424w, https://substackcdn.com/image/fetch/$s_!5Pf3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png 848w, https://substackcdn.com/image/fetch/$s_!5Pf3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png 1272w, https://substackcdn.com/image/fetch/$s_!5Pf3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6750bd78-4ba2-420e-9307-e6291263b053_1920x947.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">invisible_playwright vs Fingerprint</figcaption></figure></div><p>Note the &#8220;Confidence Score&#8221; of 1 from Fingerprint, one of the most complete and popular device intelligence and visitor identification platforms.</p><h3>Vs AntCpt.com&#8217;s Score Detector for reCAPTCHA v3</h3><p>The result was:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ldPx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ldPx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png 424w, https://substackcdn.com/image/fetch/$s_!ldPx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png 848w, https://substackcdn.com/image/fetch/$s_!ldPx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png 1272w, https://substackcdn.com/image/fetch/$s_!ldPx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ldPx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png" width="1456" height="809" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:809,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;invisible_playwright vs AntCpt.com's reCAPTCHA v3 score detector&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="invisible_playwright vs AntCpt.com's reCAPTCHA v3 score detector" title="invisible_playwright vs AntCpt.com's reCAPTCHA v3 score detector" srcset="https://substackcdn.com/image/fetch/$s_!ldPx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png 424w, https://substackcdn.com/image/fetch/$s_!ldPx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png 848w, https://substackcdn.com/image/fetch/$s_!ldPx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png 1272w, https://substackcdn.com/image/fetch/$s_!ldPx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fab63008d-b1af-46fc-96a5-cffb72e8437c_1920x1067.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">invisible_playwright vs AntCpt.com&#8217;s reCAPTCHA v3 score detector</figcaption></figure></div><p>A score of 0.9 is excellent and matches the score I achieved in my regular browser session.</p><h3>Vs APIVoid&#8217;s Bot Detection Test</h3><p>The result was:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!1tre!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!1tre!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png 424w, https://substackcdn.com/image/fetch/$s_!1tre!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png 848w, https://substackcdn.com/image/fetch/$s_!1tre!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png 1272w, https://substackcdn.com/image/fetch/$s_!1tre!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!1tre!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png" width="1456" height="743" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:743,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;invisible_playwright vs APIVoid's Bot Detection Test&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="invisible_playwright vs APIVoid's Bot Detection Test" title="invisible_playwright vs APIVoid's Bot Detection Test" srcset="https://substackcdn.com/image/fetch/$s_!1tre!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png 424w, https://substackcdn.com/image/fetch/$s_!1tre!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png 848w, https://substackcdn.com/image/fetch/$s_!1tre!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png 1272w, https://substackcdn.com/image/fetch/$s_!1tre!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdd6b4871-0ab4-44f3-8271-1a8f1212c8dc_2560x1307.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">invisible_playwright vs APIVoid&#8217;s Bot Detection Test</figcaption></figure></div><p>The detected risk score was 0, indicating a very low likelihood that the session came from a bot.</p><h3>Vs CleanTalk&#8217;s Am I a Bot?</h3><p>The result was:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Ts73!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ts73!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png 424w, https://substackcdn.com/image/fetch/$s_!Ts73!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png 848w, https://substackcdn.com/image/fetch/$s_!Ts73!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png 1272w, https://substackcdn.com/image/fetch/$s_!Ts73!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ts73!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png" width="1456" height="718" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:718,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;invisible_playwright vs CleanTalk's Am I a Bot?&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="invisible_playwright vs CleanTalk's Am I a Bot?" title="invisible_playwright vs CleanTalk's Am I a Bot?" srcset="https://substackcdn.com/image/fetch/$s_!Ts73!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png 424w, https://substackcdn.com/image/fetch/$s_!Ts73!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png 848w, https://substackcdn.com/image/fetch/$s_!Ts73!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png 1272w, https://substackcdn.com/image/fetch/$s_!Ts73!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F84488754-1e34-481a-ad31-7eef51982bf3_1920x947.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">invisible_playwright vs CleanTalk&#8217;s Am I a Bot?</figcaption></figure></div><p>The test returned a 100/100 score with a clear &#8220;Looks human&#8221; result.</p><h3>Vs Pixelscan&#8217;s Bot Detection Test</h3><p>The result was:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!9uml!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!9uml!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png 424w, https://substackcdn.com/image/fetch/$s_!9uml!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png 848w, https://substackcdn.com/image/fetch/$s_!9uml!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png 1272w, https://substackcdn.com/image/fetch/$s_!9uml!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!9uml!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png" width="1456" height="718" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/adfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:718,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;invisible_playwright vs Pixelscan's Bot Detection Test&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="invisible_playwright vs Pixelscan's Bot Detection Test" title="invisible_playwright vs Pixelscan's Bot Detection Test" srcset="https://substackcdn.com/image/fetch/$s_!9uml!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png 424w, https://substackcdn.com/image/fetch/$s_!9uml!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png 848w, https://substackcdn.com/image/fetch/$s_!9uml!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png 1272w, https://substackcdn.com/image/fetch/$s_!9uml!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadfe365c-b59c-4880-836b-4c673f2524ff_1920x947.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">invisible_playwright vs Pixelscan&#8217;s Bot Detection Test</figcaption></figure></div><p>The &#8220;You&#8217;re finally a Human&#8221; message is a strong indication that the tool successfully passed the test.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://substack.thewebscraping.club/p/consulting&quot;,&quot;text&quot;:&quot;Need help with your scraping project?&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://substack.thewebscraping.club/p/consulting"><span>Need help with your scraping project?</span></a></p><div><hr></div><h3>Anti-Bot Performance Benchmarks</h3><p>To evaluate invisible_playwright, I ran a simple script against a chosen website protected by each of the major anti-bot systems. The results are summarized in the following table:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!qWan!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!qWan!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png 424w, https://substackcdn.com/image/fetch/$s_!qWan!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png 848w, https://substackcdn.com/image/fetch/$s_!qWan!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png 1272w, https://substackcdn.com/image/fetch/$s_!qWan!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!qWan!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png" width="1456" height="402" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:402,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;invisible_playwright vs anti-bot performance benchmarks&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="invisible_playwright vs anti-bot performance benchmarks" title="invisible_playwright vs anti-bot performance benchmarks" srcset="https://substackcdn.com/image/fetch/$s_!qWan!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png 424w, https://substackcdn.com/image/fetch/$s_!qWan!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png 848w, https://substackcdn.com/image/fetch/$s_!qWan!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png 1272w, https://substackcdn.com/image/fetch/$s_!qWan!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1f92d05f-3dff-4f68-b07b-1dd038f66bca_1920x530.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption">invisible_playwright vs anti-bot performance benchmarks</figcaption></figure></div><p><strong>Note</strong>: All tests were performed locally using my ISP-provided residential IP address.</p><p>In this limited experiment, invisible_playwright successfully loaded all five test sites, achieving a 100% success rate. By comparison, vanilla Playwright consistently failed in headless mode and also struggled with Kasada in headful mode.</p><h2>Comparisons with Other Stealth Browser Automation Tools</h2><p>This is how invisible_playwright compares with other <a href="https://www.scraping.club/p/browser-automation-landscape-2025">anti-detect browser automation tools</a>:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mboM!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a6b3613-5e06-42ae-9e7b-9a2af7042141_1921x443.png 424w, https://substackcdn.com/image/fetch/$s_!mboM!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a6b3613-5e06-42ae-9e7b-9a2af7042141_1921x443.png 848w, https://substackcdn.com/image/fetch/$s_!mboM!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a6b3613-5e06-42ae-9e7b-9a2af7042141_1921x443.png 1272w, https://substackcdn.com/image/fetch/$s_!mboM!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a6b3613-5e06-42ae-9e7b-9a2af7042141_1921x443.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mboM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a6b3613-5e06-42ae-9e7b-9a2af7042141_1921x443.png" width="1456" height="336" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7a6b3613-5e06-42ae-9e7b-9a2af7042141_1921x443.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:336,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;invisible_playwright vs other stealth browser automation tools&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="invisible_playwright vs other stealth browser automation tools" title="invisible_playwright vs other stealth browser automation tools" srcset="https://substackcdn.com/image/fetch/$s_!mboM!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a6b3613-5e06-42ae-9e7b-9a2af7042141_1921x443.png 424w, https://substackcdn.com/image/fetch/$s_!mboM!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a6b3613-5e06-42ae-9e7b-9a2af7042141_1921x443.png 848w, https://substackcdn.com/image/fetch/$s_!mboM!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a6b3613-5e06-42ae-9e7b-9a2af7042141_1921x443.png 1272w, https://substackcdn.com/image/fetch/$s_!mboM!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7a6b3613-5e06-42ae-9e7b-9a2af7042141_1921x443.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption">invisible_playwright vs other stealth browser automation tools</figcaption></figure></div><p>Each approach involves trade-offs. Firefox has a smaller market share than Chromium, which may itself be a detection signal. At the same time, it&#8217;s currently the only browser engine that can be patched at the native binary level.</p><p>Chromium-based tools typically patch the browser driver or inject JavaScript into the page, leaving more detectable surface area exposed.</p><p>playwright-stealth is lightweight and works with multiple browsers, but it relies on injected JavaScript, which anti-bot systems may detect. <a href="https://www.scraping.club/p/is-camoufox-still-effective-scraping">Camoufox uses a fingerprint database</a> in place of seed-derived values, so it doesn&#8217;t provide the same reproducibility across runs.</p><p>invisible_playwright, by contrast, focuses on reproducible fingerprints and deep, native-level browser patching.</p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>Final Comment</h2><p>Let me get straight to the point: invisible_playwright looks like a very solid choice for web scraping projects where bot detection is a concern. The library not only performed well in my tests and delivered on its promises, but is also exceptionally well documented, with a long list of guides, examples, and low-level explanations of how it works.</p><p>There is one potential downside, though. Because of that, and the fact that the project is fully open source, anti-bot providers could potentially reverse-engineer its implementation as it gains adoption within the community.</p><p>That&#8217;s probably my biggest concern with the library, along with its reliance on a single main maintainer. However, both are challenges that apply to many open-source stealth browser automation tools.</p><p>I also want to add a personal note and give a shout-out to Federico Elia, the author of the library. In the era of AI, where almost anyone can vibe-code a product or library in a matter of hours, or even minutes, it&#8217;s genuinely refreshing to see this level of robustness and documentation. In my experience, that combination is still far from common. Well done!</p><p>The biggest advantages of invisible_playwright are definitely its full Playwright compatibility and straightforward setup. The seed-based fingerprinting is another particularly interesting feature that not all anti-detect browser automation libraries offer.</p><p>Keep in mind that the tool requires Firefox and Python. If you need Chromium or Node.js support, you&#8217;ll have to look elsewhere. Otherwise, I strongly recommend giving this project a try and seeing how it performs for yourself!</p><div><hr></div><p><em>Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/p/invisible-playwright-beat-bot-detection-scraping?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.scraping.club/p/invisible-playwright-beat-bot-detection-scraping?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[What breaks in a long-running authenticated scraper session, and how to keep it logged in]]></title><description><![CDATA[Most scrapers work fine for a few hours, but not if you keep one logged in for a few weeks.]]></description><link>https://www.scraping.club/p/authenticated-scraper-sessions</link><guid isPermaLink="false">https://www.scraping.club/p/authenticated-scraper-sessions</guid><dc:creator><![CDATA[Joel Griffith]]></dc:creator><pubDate>Tue, 29 Sep 2026 08:01:47 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!G3Em!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p><em>This is a guest post written by <span>Joel Griffith, Founder of </span><a href="https://www.browserless.io/?utm_source=thewebscrapingclub&amp;utm_medium=guest-post&amp;utm_campaign=authenticated-scraper-sessions"><span>Browserless</span></a>.</em></p><div><hr></div><p><span>Most authenticated scrapers I&#8217;ve seen don&#8217;t fail at login. They fail when one piece of state stops matching everything else, even while the login still seems valid.</span></p><p><span>That causes problems if you&#8217;re scraping account-gated dashboards, portals, or other authenticated applications over long periods. Saving a cookie can work well enough for a short-lived job. But if you run the same job for weeks, the login &#8220;state&#8221; becomes something you have to manage as well.</span></p><p><span>In my experience, you really have two options: save authenticated state and replay it into new browsers, or keep the same browser session alive. Both still end in re-authentication eventually.</span></p><p><span>So, what do you need to preserve, and what can safely be rebuilt?</span></p><h2><span>What does being logged in actually look like?</span></h2><p><span>It&#8217;s tempting to think of an authenticated session as one cookie, even though sometimes it is just that.</span></p><p><span>But more often, the browser ends up carrying authentication-related &#8220;signals&#8221; in several other places as well. You might have an HttpOnly session cookie, a token in localStorage, another value in IndexedDB, and a temporary token associated with an OAuth flow someplace else.</span></p><p><span>The issue is that all these technologies don&#8217;t share the same eviction timeframe, and sometimes those timeframes don&#8217;t align.</span></p><ul><li><p><span>A cookie could still exist locally after the server has invalidated the session behind it.</span></p></li><li><p><span>An access token may expire while a refresh token remains usable.</span></p></li><li><p><span>A refresh token may be rotated, meaning the old value stops working as soon as the new one is issued.</span></p></li><li><p><span>There&#8217;s another interim state that has changed: maybe an IP address or even geolocation changes.</span></p></li></ul><p><span>Some applications also make risk decisions using signals outside the state you saved, such as network location or characteristics of the browser making the request. This is the part that trips people up most: a person logs in by hand, and then a script picks up where they left off. Anti-bot detection loops run passively all the time, and if something looks suspicious, they&#8217;re quick to revoke access.</span></p><p><span>I&#8217;ve stopped thinking of it as logged in or logged out. It&#8217;s closer to three states: valid, stale, and recovering. The middle one is where I&#8217;ve lost the most time.</span></p><h2><span>Why a saved storageState snapshot always breaks eventually</span></h2><p><span>For most scraping jobs, replaying auth state into a fresh browser beats logging in on every run. Here&#8217;s what that looks like with Playwright:</span></p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;javascript&quot;,&quot;nodeId&quot;:&quot;2a57431f-b33e-468e-be91-e4109040e7c6&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-javascript">const context = await browser.newContext({
  storageState: &#8220;auth.json&#8221;,
});

const page = await context.newPage();
await page.goto(PROTECTED_URL);</code></pre></div><p><span>You authenticate once, persist the relevant state, then use it to bootstrap later browser contexts. Re-applying prior auth is much faster than repeating the login flow, for obvious reasons, and puts less load on the target. This means you don&#8217;t have to automate passwords or manage multi-factor authentication unnecessarily.</span></p><p><span>But there are a few gotchas, the biggest being: the state file only contains IndexedDB if you asked for it when you saved it.</span></p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;javascript&quot;,&quot;nodeId&quot;:&quot;beef4536-a8db-40a1-8c31-debc38f9f054&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-javascript">await context.storageState({ path: &#8220;auth.json&#8221;, indexedDB: true });</code></pre></div><p><span>The option was introduced in Playwright v1.51. On older versions, apps that keep tokens in IndexedDB (like Firebase Authentication) will silently lose them.</span></p><p><span>I think of auth.json as a photo of your login on the day you took it, not the login itself.</span></p><p><span>Imagine the scraper starts on Monday with a working token. On Wednesday, the application rotates that token during normal use. Your live browser now contains the new token, but the state file you created on Monday doesn&#8217;t.</span></p><p><span>Restart the worker on Thursday, and you restore Monday&#8217;s version of reality. You can start to see how this problem plays out, and why its nondeterministic state can cause a lot of confusion. To make matters worse, this &#8220;invalidation&#8221; can happen out of nowhere. If you&#8217;re paying per run, that isn&#8217;t just downtime. It&#8217;s money spent on scrapes that bring back a login page.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!G3Em!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!G3Em!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!G3Em!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!G3Em!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!G3Em!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!G3Em!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/dccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!G3Em!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!G3Em!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!G3Em!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!G3Em!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdccf93e3-9993-48a0-9958-049d4fbeddc7_2048x1152.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p><span>Cookies run into the same kind of trouble. Their local expiry tells you when the browser considers a cookie expired. It doesn&#8217;t tell you whether the server still accepts the session behind that cookie. In a perfect world, they&#8217;d always be in sync, but the web is anything but perfect!</span></p><p><span>Temporary state is another failure point. sessionStorage, for example, is deliberately scoped to a browser tab. Turn every piece of temporary browser state into something permanently replayable, and you can make authentication less reliable, particularly around redirects and CSRF protection.</span></p><p><span>My rule: assume any saved auth state is already going stale. Authenticated scraping is fundamentally a high-fault workflow, so take great care to ensure the system behaves as expected in production.</span></p><h2><span>Why not just keep the browser running?</span></h2><p><span>If replaying a snapshot loses state, the obvious alternative is to keep the original browser alive. For some workflows, that&#8217;s exactly what you need, but it can really depend on how long you need this transient state for.</span></p><p><span>A running browser preserves far more than a cookie file. Open tabs and JavaScript state survive, and storage updates happen naturally. If the application refreshes a token in the background, the browser keeps the new value rather than reverting to an older snapshot on the next run.</span></p><p><span>While trading off persistence for long-running processes, you&#8217;ve traded one problem for another, </span><em><span>and</span></em><span> you still haven&#8217;t solved it! Instead of asking whether your authentication state can survive for three weeks, you&#8217;re asking whether a Chrome process can. I wouldn&#8217;t bet on the Chrome process.</span></p><p><span>Long-running browser processes fail in ordinary ways:</span></p><ul><li><p><span>Pages leak memory.</span></p></li><li><p><span>Connections disappear.</span></p></li><li><p><span>Hosts restart.</span></p></li><li><p><span>Deployments happen.</span></p></li><li><p><span>Chrome crashes.</span></p></li><li><p><span>Your automation code crashes even when Chrome doesn&#8217;t.</span></p></li><li><p><span>... and all these problems compound with one another!</span></p></li></ul><p><span>You&#8217;re also limited when you need to scale. One persistent browser is one mutable environment. If ten workers need the same account, letting all ten operate against one live browser introduces coordination problems. With a saved state snapshot, each worker gets its own isolated copy instead.</span></p><p><span>A long-lived browser is useful when you need continuity inside the browser, but it isn&#8217;t a general replacement for saved state.</span></p><h2><span>How can you detect an expired session?</span></h2><p><span>The worst recovery strategy is waiting for the scraper to throw an exception, because authentication failures often don&#8217;t throw one.</span></p><p><span>A protected URL might respond with a 302 to /login. Another site returns 200, but the HTML is now a sign-in screen. An API request might start returning 401. Some applications send you to an account verification flow instead.</span></p><p><span>I normally want an authenticated scraper to validate application state explicitly after restoring a session. Here&#8217;s a quick example:</span></p><div class="highlighted_code_block" data-attrs="{&quot;language&quot;:&quot;javascript&quot;,&quot;nodeId&quot;:&quot;1cc29ba3-334e-4235-841d-47aad909ed34&quot;}" data-component-name="HighlightedCodeBlockToDOM"><pre class="shiki"><code class="language-javascript">await page.goto(PROTECTED_URL, {
  waitUntil: &#8220;domcontentloaded&#8221;,
});

const authenticated =
  !page.url().includes(&#8221;/login&#8221;) &amp;&amp;
  (await page.locator(&#8221;[data-account-id]&#8221;).count()) &gt; 0;

if (!authenticated) {
  await reauthenticate(page);
  await saveFreshState(page);
}</code></pre></div><p><span>Here we&#8217;re using other signals on the page to determine if it&#8217;s properly authenticated or not. We don&#8217;t ask whether the cookie exists; instead, we ask whether the application still considers the browser authenticated. You </span><em><span>could </span></em><span>script up a cookie check, but from what we&#8217;ve seen running this at Browserless, cookie checks are brittle and throw a lot of false positives.</span></p><p><span>In production, I&#8217;d also distinguish auth failure from ordinary scraping failure. A changed selector shouldn&#8217;t trigger a login. Neither should a 500, a network timeout, or a missing asset. Otherwise, a minor site change becomes hundreds of unnecessary login attempts, which is a much worse failure mode. That&#8217;s a mistake you only need to make once. You have to pick one or two of the </span><em><span>strongest </span></em><span>signals you can see on the page and use those for your authentication condition check.</span></p><h2><span>When is re-authentication the right answer?</span></h2><p><span>Re-authentication tends to get treated as something a good persistence system should eliminate. I&#8217;d argue the opposite: it&#8217;s a recovery path you should design deliberately. I&#8217;ve never run a system that works 100% of the time, even ones we own end to end. I don&#8217;t expect it from a script running against someone else&#8217;s site.</span></p><p><span>If login is cheap and stable, periodically rebuilding state can be simpler than maintaining a browser indefinitely. Preserving authenticated state becomes much more valuable when login involves MFA, email links, or aggressive rate limits. Either way, you want re-authentication to happen only after you&#8217;ve positively identified expiry.</span></p><p><span>Your choice also hinges on concurrency. If many workers need the same authenticated starting point, a reusable snapshot lets each one launch its own browser from it. Keeping the browser itself alive makes more sense when a workflow relies on in-progress UI state.</span></p><p><span>In practice, reliable systems often use both. Start sessions from a known authenticated snapshot, let each browser mutate normally while it is alive, and validate authentication before doing expensive work. When the saved snapshot is no longer accepted, run the login flow again and replace it.</span></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!LPBy!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!LPBy!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!LPBy!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!LPBy!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!LPBy!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!LPBy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png" width="1456" height="819" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:819,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!LPBy!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png 424w, https://substackcdn.com/image/fetch/$s_!LPBy!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png 848w, https://substackcdn.com/image/fetch/$s_!LPBy!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png 1272w, https://substackcdn.com/image/fetch/$s_!LPBy!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1902392c-5f28-4659-84bb-a50df3758fc0_2048x1152.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><h2><span>Saved state and live sessions solve different needs</span></h2><p><span>We ran into this distinction while building authenticated session support at Browserless.</span></p><p><span>Our </span><a href="https://docs.browserless.io/baas/features/authenticated-profiles"><span>Authenticated Profiles</span></a><span> feature stores cookies, localStorage, and IndexedDB, then loads a copy of that state into a fresh browser session. We deliberately skip sessionStorage, since replaying temporary tab state can break short-lived flows such as OAuth redirects and CSRF handling.</span></p><p><span>Changes made inside one of those browsers don&#8217;t rewrite the original saved profile. Think of it as a fresh copy of the authentication state, or a reference by value and not address. If the underlying authentication state expires, the profile needs to be refreshed.</span></p><p><span>That is deliberately separate from session persistence, which comes in two flavors. A standard session flags the running browser as reconnectable mid-script, so you can drop back into the same process, same page, and same in-memory state within a short window.</span></p><p><span>Persisting state goes further: the Session API keeps cookies, localStorage, and cache in an isolated userDataDir that survives full browser restarts for days, and processKeepAlive decides whether the process stays up in between. The two combine: pass a profile name when you create the session, and the browser starts from that saved auth state and then persists normally.</span></p><p><span>One caveat for Playwright users: both of the keep-the-process-alive paths rely on browser.disconnect(), which Playwright doesn&#8217;t expose, so with Playwright you get the on-disk state but not the live process.</span></p><p><span>This distinction forces you to decide what you actually need. Do you need the same authentication state, or do you need the same browser?</span></p><p><span>Once your scraper runs for weeks rather than minutes, those stop being interchangeable.</span></p><h2><span>Plan for the logout</span></h2><p><span>If I could give one piece of advice to anyone building an authenticated scraper, it&#8217;s to stop trying to keep the session alive forever. We spent a lot of time on this problem while building Browserless, and the biggest shift for us was treating a logout as a state to plan for and not a bug to get rid of.</span></p><p><span>Every session goes stale eventually. What separates a scraper that runs for weeks from one that falls over on day three is whether it notices. So I save the full picture: cookies, localStorage, and IndexedDB, but not sessionStorage. I ask the app whether I&#8217;m still logged in instead of trusting a cookie. And I only log in again once I&#8217;ve confirmed the session is gone, then replace the snapshot so the next worker isn&#8217;t starting from last Monday.</span></p><p><span>As for snapshots versus live browsers, I&#8217;d stop thinking of it as a choice. Most setups I&#8217;d put into production use both.</span></p><p><span>You won&#8217;t build a session that never expires, and I wouldn&#8217;t try. Build one that knows when it&#8217;s expired and fixes itself before anyone gets paged. And if you&#8217;ve found a pattern that holds up longer than this, I&#8217;d like to hear about it!</span></p>]]></content:encoded></item><item><title><![CDATA[How to Use Scraping to Analyze Social Media Trends]]></title><description><![CDATA[A deep dive int how to build a multi-platform social media scraping pipeline that turns posts, hashtags, and engagement data into meaningful trend insights.]]></description><link>https://www.scraping.club/p/how-to-scrape-social-media</link><guid isPermaLink="false">https://www.scraping.club/p/how-to-scrape-social-media</guid><dc:creator><![CDATA[Federico Trotta]]></dc:creator><pubDate>Sun, 27 Sep 2026 15:54:23 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/350aa11a-91df-4766-854f-0f7243d47c19_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Social media trend analysis starts with a simple question: &#8220;What are people starting to talk about, and how do I catch it before it becomes obvious?&#8221;</p><p>You could scroll through feeds, follow hashtags, and check what each platform labels as trending. But that only gives you part of the picture. This happens because conversations move across platforms, engagement changes over time, and what looks like growing interest might just be the same content reposted over and over.</p><p>That&#8217;s where social media scraping comes in and makes the game easier. If you&#8217;re wondering why, the answer is simple: it lets you collect posts and engagement metrics systematically, then track how conversations evolve. However, collecting the data isn&#8217;t enough. You need a pipeline that separates meaningful signals from noise, handles differences between platforms, and accounts for the legal requirements of collecting and using social data.</p><p>In this article, we&#8217;ll discuss how to choose your sources, build the collection infrastructure, extract and clean trend signals with Python, and turn them into useful analysis.</p><blockquote><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/rE0PQB">Decodo&#8217;s Web Scraping API</a></strong> turns any site into clean, structured data your models can actually use. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/rE0PQB" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aRfI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png" width="724" height="362" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:512,&quot;width&quot;:1024,&quot;resizeWidth&quot;:724,&quot;bytes&quot;:716985,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/rE0PQB&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://substack.thewebscraping.club/i/204450518?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!aRfI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/rE0PQB&quot;,&quot;text&quot;:&quot;Activate starter plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/rE0PQB"><span>Activate starter plan</span></a></p><div><hr></div></blockquote><h2>Understanding the Social Media Data Landscape</h2><p>Before writing code, you need to map out the platform landscape you&#8217;re targeting. Social media scraping isn&#8217;t a one-size-fits-all discipline because each platform exposes data differently, enforces its own limits, and deploys different anti-bot measures.</p><p>So, you can&#8217;t just pick a social media platform at random and think to apply a scraping strategy to other platforms. You must align your platform choice with your trend-analysis goals and with the technical features each platform has.</p><h3>Checking for Platform-Specific Data Structures</h3><p>Once you&#8217;ve picked your social media targets, you&#8217;ll quickly notice that no two platforms structure their data the same way:</p><ul><li><p>Some platforms expose posts and engagement metrics through JSON-based GraphQL responses.</p></li><li><p>Others return JSON too, but through obfuscated and session-dependent internal APIs. And these are harder to reverse-engineer.</p></li><li><p>Others rely on server-rendered HTML. And this requires a more fragile extraction, as it is selector-based. But remember that, in such cases, a possible solution is <a href="https://www.scraping.club/p/scraping-with-llms-gpt-vision">using GPT vision to extract the data</a>.</p></li></ul><p>To make an example, X (formerly Twitter) fetches data through internal GraphQL endpoints visible in the browser through the DevTools:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!gTzT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gTzT!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png 424w, https://substackcdn.com/image/fetch/$s_!gTzT!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png 848w, https://substackcdn.com/image/fetch/$s_!gTzT!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png 1272w, https://substackcdn.com/image/fetch/$s_!gTzT!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!gTzT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png" width="1456" height="722" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:722,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!gTzT!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png 424w, https://substackcdn.com/image/fetch/$s_!gTzT!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png 848w, https://substackcdn.com/image/fetch/$s_!gTzT!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png 1272w, https://substackcdn.com/image/fetch/$s_!gTzT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F95b9e415-333f-404f-91d6-d741d951dd2b_1573x780.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Instead, <a href="https://www.scraping.club/p/scraping-linkedin-public-data">LinkedIn fetches feed and post data through internal JSON API calls</a> using obfuscated endpoint paths, rather than readable REST routes:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!z5LN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c249304-9968-4d90-af98-caf16674c707_1273x707.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!z5LN!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c249304-9968-4d90-af98-caf16674c707_1273x707.png 424w, https://substackcdn.com/image/fetch/$s_!z5LN!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c249304-9968-4d90-af98-caf16674c707_1273x707.png 848w, https://substackcdn.com/image/fetch/$s_!z5LN!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c249304-9968-4d90-af98-caf16674c707_1273x707.png 1272w, https://substackcdn.com/image/fetch/$s_!z5LN!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c249304-9968-4d90-af98-caf16674c707_1273x707.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!z5LN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c249304-9968-4d90-af98-caf16674c707_1273x707.png" width="1273" height="707" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7c249304-9968-4d90-af98-caf16674c707_1273x707.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:707,&quot;width&quot;:1273,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!z5LN!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c249304-9968-4d90-af98-caf16674c707_1273x707.png 424w, https://substackcdn.com/image/fetch/$s_!z5LN!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c249304-9968-4d90-af98-caf16674c707_1273x707.png 848w, https://substackcdn.com/image/fetch/$s_!z5LN!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c249304-9968-4d90-af98-caf16674c707_1273x707.png 1272w, https://substackcdn.com/image/fetch/$s_!z5LN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c249304-9968-4d90-af98-caf16674c707_1273x707.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>This means your social media scraping pipeline can&#8217;t use a single generic extraction strategy. You need tailored logic for each source, focusing on the specific social media.</p><h3>API vs. Direct Scraping Trade-offs</h3><p>You&#8217;ll also have to decide early on whether to lean on official APIs or go the direct scraping route. APIs give you stability and predictable schemas, but they often come with strict rate limits, high costs, and severe limitations regarding the data you can retrieve.</p><p>For instance, <a href="https://docs.x.com/developer-guidelines">X is very specific about how developers should use their APIs</a> and what happens if they don&#8217;t:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!_P8M!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_P8M!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png 424w, https://substackcdn.com/image/fetch/$s_!_P8M!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png 848w, https://substackcdn.com/image/fetch/$s_!_P8M!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png 1272w, https://substackcdn.com/image/fetch/$s_!_P8M!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!_P8M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png" width="1456" height="471" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:471,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!_P8M!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png 424w, https://substackcdn.com/image/fetch/$s_!_P8M!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png 848w, https://substackcdn.com/image/fetch/$s_!_P8M!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png 1272w, https://substackcdn.com/image/fetch/$s_!_P8M!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93dd4bc1-977e-4adc-876c-c7e135efa146_1873x606.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Even the <a href="https://learn.microsoft.com/en-us/linkedin/?context=linkedin/context">LinkedIn API</a> is very restrictive on use cases. For example, the public API doesn&#8217;t expose feed/post/engagement data for general use. This means that access is essentially locked to partners (limited marketing/talent solutions):</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XAb6!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8131c8e1-367a-404d-952c-669391593c1e_1867x805.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XAb6!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8131c8e1-367a-404d-952c-669391593c1e_1867x805.png 424w, https://substackcdn.com/image/fetch/$s_!XAb6!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8131c8e1-367a-404d-952c-669391593c1e_1867x805.png 848w, https://substackcdn.com/image/fetch/$s_!XAb6!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8131c8e1-367a-404d-952c-669391593c1e_1867x805.png 1272w, https://substackcdn.com/image/fetch/$s_!XAb6!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8131c8e1-367a-404d-952c-669391593c1e_1867x805.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!XAb6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8131c8e1-367a-404d-952c-669391593c1e_1867x805.png" width="1456" height="628" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/8131c8e1-367a-404d-952c-669391593c1e_1867x805.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:628,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!XAb6!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8131c8e1-367a-404d-952c-669391593c1e_1867x805.png 424w, https://substackcdn.com/image/fetch/$s_!XAb6!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8131c8e1-367a-404d-952c-669391593c1e_1867x805.png 848w, https://substackcdn.com/image/fetch/$s_!XAb6!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8131c8e1-367a-404d-952c-669391593c1e_1867x805.png 1272w, https://substackcdn.com/image/fetch/$s_!XAb6!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8131c8e1-367a-404d-952c-669391593c1e_1867x805.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>Direct scraping, on the other hand, gives you far more flexibility and access to richer data, but it raises the stakes. Basically, you&#8217;re taking on:</p><ul><li><p>Greater legal exposure.</p></li><li><p>A heavier maintenance burden since platforms change their front-end structure without warning.</p></li><li><p>Infrastructure management. You need to account for: geo-targeted proxies, a wide pool of residential proxies, CAPTCHA-solving capabilities, <a href="https://www.scraping.club/p/scraping-high-frequency-python">throttle and rate limiting management as these are high-traffic websites</a>, and anything in between.</p></li></ul><div><hr></div><blockquote><p>Only pay for data you actually get &#8212; one request, one credit, every time. No surcharge for JS rendering, no charge when a call gets blocked. <em><strong>Claim your free 10,000 requests with coupon code</strong></em> <strong>WEBSCR</strong></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!sxTx!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!sxTx!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png 424w, https://substackcdn.com/image/fetch/$s_!sxTx!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png 848w, https://substackcdn.com/image/fetch/$s_!sxTx!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png 1272w, https://substackcdn.com/image/fetch/$s_!sxTx!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!sxTx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png" width="1456" height="764" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:764,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:403485,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/210089674?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!sxTx!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png 424w, https://substackcdn.com/image/fetch/$s_!sxTx!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png 848w, https://substackcdn.com/image/fetch/$s_!sxTx!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png 1272w, https://substackcdn.com/image/fetch/$s_!sxTx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F942adb29-f668-42e1-ad1b-a13438b78669_2400x1260.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scrapeunblocker.com/?utm_source=wsclub&amp;utm_medium=email&amp;utm_campaign=free-10k&quot;,&quot;text&quot;:&quot;Grab the deal!&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.scrapeunblocker.com/?utm_source=wsclub&amp;utm_medium=email&amp;utm_campaign=free-10k"><span>Grab the deal!</span></a></p></blockquote><div><hr></div><h2>Legal and Ethical Considerations When Scraping Social Media for Trends</h2><p>Before moving into code and infrastructure, you need to think through the legal terrain you&#8217;re operating in. Otherwise, the risk is getting into legal trouble once you get the data you need.</p><p>If you&#8217;ve been web scraping for a while, you know that accounting for <a href="https://www.scraping.club/p/best-practices-for-ethical-web-scraping">ethical web scraping is a best practice</a> for any scraping case. But note that social media scraping is a delicate case. This is because it sits at the intersection of platform terms of service, privacy regulations like GDPR and CCPA, and copyright law that governs user-generated content.</p><p>This means that if you want to stay safe, you need a compliance framework in place from the beginning. And this matters even if you&#8217;re only collecting public posts for trend analysis. For this specific case, this is what matters most:</p><ul><li><p><strong>Compliance with platform terms of service:</strong> Most platforms explicitly ban automated access in their terms of service, and they don&#8217;t always distinguish between a well-intentioned research project and a malicious bot. If you violate those terms, you&#8217;re risking account bans, IP blocks, or even legal action from the platform itself. That&#8217;s why you want to do a proper risk assessment before you start scraping.</p></li><li><p><strong>Handling personally identifiable information (PII):</strong> Even when you&#8217;re only after trend data, you&#8217;ll almost certainly capture PII (usernames, bios, locations, and sometimes more) along the way. That&#8217;s where things get tricky under regulations like GDPR, which don&#8217;t care whether the data was &#8220;public&#8221; when you scraped it. If you think you need this kind of data for some reason, you better build in an anonymization layer and apply it to the data.</p></li><li><p><strong>Respecting robots.txt</strong>: That&#8217;s probably the evergreen rule. And at The Web Scraping Club, we deeply discussed the <a href="https://www.scraping.club/p/understanding-robotstxt-and-its-implications">implications of the robots.txt file</a>. Remember that it&#8217;s not legally binding on its own, but ignoring it undermines your compliance framework. And if you do that, it can be used as evidence of bad faith if a platform pursues legal action over your social media scraping activity. So, treat it as a baseline signal of the platform&#8217;s intent, then layer your ToS and privacy review on top of it rather than relying on robots.txt alone.</p></li></ul><p>And for the sake of being tied to reality, let&#8217;s check <a href="https://www.linkedin.com/robots.txt">LinkedIn&#8217;s robots.txt file</a>:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SNIK!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3570238-cbf6-4fbf-a74e-253eb7054a44_1813x361.png 424w, https://substackcdn.com/image/fetch/$s_!SNIK!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3570238-cbf6-4fbf-a74e-253eb7054a44_1813x361.png 848w, https://substackcdn.com/image/fetch/$s_!SNIK!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3570238-cbf6-4fbf-a74e-253eb7054a44_1813x361.png 1272w, https://substackcdn.com/image/fetch/$s_!SNIK!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3570238-cbf6-4fbf-a74e-253eb7054a44_1813x361.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!SNIK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3570238-cbf6-4fbf-a74e-253eb7054a44_1813x361.png" width="1456" height="290" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a3570238-cbf6-4fbf-a74e-253eb7054a44_1813x361.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:290,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!SNIK!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3570238-cbf6-4fbf-a74e-253eb7054a44_1813x361.png 424w, https://substackcdn.com/image/fetch/$s_!SNIK!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3570238-cbf6-4fbf-a74e-253eb7054a44_1813x361.png 848w, https://substackcdn.com/image/fetch/$s_!SNIK!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3570238-cbf6-4fbf-a74e-253eb7054a44_1813x361.png 1272w, https://substackcdn.com/image/fetch/$s_!SNIK!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa3570238-cbf6-4fbf-a74e-253eb7054a44_1813x361.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div></div></div></div></figure></div><div><hr></div><p>The above image reports only the initial part of it. But, as you can see, it&#8217;s very clear: you can not use automated software to access LinkedIn unless you ask for specific permission via e-mail. And you&#8217;re not sure you&#8217;ll get permission from LinkedIn, of course.</p><h2>Building a Scalable Scraping Architecture</h2><p>A trend-analysis pipeline isn&#8217;t a simple scraper running on a cron job. Once you&#8217;re dealing with high-velocity, high-volume streams of short-lived social content, you need distributed crawling, proxy rotation, and storage systems built to keep up with the momentum.</p><p>So, let&#8217;s consider the case where you decided to go with your custom scraper because the platform APIs don&#8217;t cover your case. In that case, after assessing the legal risk and defining the legal framework you&#8217;ll work on, you need to build the infrastructure.</p><h3>Proxy Rotation and Anti-Detection Techniques</h3><p>Social platforms don&#8217;t take kindly to automated traffic, and their <a href="https://www.scraping.club/p/machine-learning-for-detecting-bots">anti-bot systems have become aggressive because they use machine learning to intercept detection triggers</a>. To stay under the radar, you need to master at least the following:</p><ul><li><p><a href="https://www.scraping.club/p/the-engineers-guide-residential-proxies">Residential proxies</a>, since they&#8217;re far less likely to get flagged than datacenter IPs.</p></li><li><p>Header randomization, so your requests don&#8217;t all look like they&#8217;re coming from the same bot.</p></li><li><p><a href="https://substack.thewebscraping.club/p/understanding-browser-fingerprint">Browser fingerprint spoofing</a>, which helps you avoid the CAPTCHAs and soft bans that kick in the moment a platform notices a suspicious pattern.</p></li></ul><p>None of these alone is a silver bullet. But combined, they buy you the runway you need to keep collecting data without getting IP-banned every few hours.</p><h3><strong>Headless Browsers vs. Lightweight HTTP Clients</strong></h3><p>Here&#8217;s where you need to be strategic about resource allocation. Dynamic, JS-rendered feeds (think <a href="https://substack.thewebscraping.club/p/handling-infinite-scrolling-python-js">infinite-scroll timelines</a>) often leave you no choice but to use headless browsers like <a href="https://playwright.dev/">Playwright</a> or <a href="https://playwright.dev/">Puppeteer</a>, since the content simply isn&#8217;t there until JavaScript executes. But that comes at a cost: headless browsers are slow and resource-hungry.</p><p>So, for static or semi-static endpoints, you&#8217;re better off with lightweight HTTP-based scrapers. They&#8217;re faster, cheaper to run at scale, and don&#8217;t need a full browser context spun up just to grab a JSON payload. The trick is knowing which endpoints actually need the heavy tooling and which don&#8217;t.</p><h3>Scheduling and Real-Time Data Pipelines</h3><p>Trend detection is a race against freshness. And a trend you catch six hours late isn&#8217;t a trend anymore. That&#8217;s why your scheduling strategy matters as much as your extraction logic. The rule of thumb is the following:</p><ul><li><p>Cron-based batch jobs work fine if you&#8217;re tracking slower-moving trends and can tolerate some lag.</p></li><li><p>Streaming architectures (Kafka, message queues) are what you need if you&#8217;re after near-real-time detection. This is because, in that case, you need to process data as it arrives instead of waiting for the next batch window.</p></li></ul><p>Which one you pick really comes down to how fast you need to know something&#8217;s trending, and how much infrastructure complexity you&#8217;re willing to take on to get there.</p><h2>What If You Don&#8217;t Want to Manage The Infrastructure?</h2><p>If you&#8217;re a scraping expert, you certainly know the burden of managing the scraping infrastructure: residential proxies are very expensive, CAPTCHAs block you, selectors change every one in a while, parallel requests take time to execute, and so on. Furthermore, the legalities of social media scraping get you up at night.</p><p>So&#8230;are there any other alternatives other than using official platform APIs and scraping on your own? The answer is yes!</p><p>As you know, the scraping market is strong, and lots of companies provide several services in it. Among the companies that provide scraping services, some offer social media scraping services. I don&#8217;t want to end up in a list here because everything is easily searchable on your favourite provider&#8217;s website. Just note that the majority of them provide pre-collected datasets and/or scraping APIs to retrieve data from social media.</p><p>In simple terms, you just need to make an API call and collect posts from Instagram, Facebook, TikTok, or whatever social media you&#8217;re targeting. The advantages of this approach are the following:</p><ul><li><p>The scraping service manages the infrastructure for you.</p></li><li><p>You don&#8217;t need to manage the legalities behind the data extraction part. But, yes: you still need to account for the legalities of anyting related, like data retention, storage, and usage.</p></li><li><p>If you&#8217;re ambitious and want to scale, you can do it along with the scraping service you&#8217;re using.</p></li><li><p>You don&#8217;t need to think about how the platform displays the data. Either they expose APIs or render HTML, you make an API call, and you&#8217;re good to go.</p></li><li><p>The price is generally fair. I mean: if you don&#8217;t own proxies or any other scraping infrastructure, API calls for retrieving data from social media platforms need just a few dollars. Furthermore, depending on the scale of your project, you may simply benefit from the service&#8217;s free trial.</p></li></ul><p>So, if you&#8217;re scraping social media to intercept trends, the suggestion is to take a look at these solutions before writing your own custom code. You may be surprised by how you can find the right solution for your case.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><h2>Data Management Strategies for Trend Signals</h2><p>Grabbing raw posts from social media is only half the job. And that&#8217;s it, regardless of the data extraction method you used. To actually detect trends, your social media scraping pipeline needs to extract structured signals from content noise.</p><p>In social media posts, the fields that actually tell you if something is trending are hashtags, topics, engagement velocity, and mention frequency. So, let&#8217;s see how to manage these.</p><h3>Extracting Hashtags, Mentions, and Keywords</h3><p>Post text is messy. So, before you can count anything meaningful, you need to isolate the actual trend indicators from the surrounding noise. A &#8220;traditional&#8221; approach to do that is using regex.</p><p>That method gets you most of the way for hashtags and mentions. For example, consider the code below:</p><pre><code><code>import re  
from collections import Counter  
  
def extract_signals(text):  
    hashtags = re.findall(r"#(\w+)", text)  
    mentions = re.findall(r"@(\w+)", text)  
    return hashtags, mentions  
  
posts = [  
    "AI is eating the world #AI #MachineLearning cc @openai",  
    "New drop today #sneakers #AI incoming",  
]  
  
hashtag_counter = Counter()  
mention_counter = Counter()  
  
for post in posts:  
    tags, mentions = extract_signals(post)  
    hashtag_counter.update(tags)  
    mention_counter.update(mentions)  
  
print(hashtag_counter.most_common())  </code></code></pre><p><strong>NOTE</strong>: For the sake of simplicity, the <code>posts[]</code> list reports short content. An example from a complete scraping scenario works by opening a CSV or JSON file, extracting the content, and applying regex.</p><p>The result from such a basic example is the following:</p><pre><code><code>[('AI', 2), ('MachineLearning', 1), ('sneakers', 1)]</code></code></pre><p>The above result is a list of tuples. Each tuple shows the hashtag and how many times it appears in the analyzed content. This is an elegant result because regex gets you hashtags and mentions cleanly, but it won&#8217;t catch implicit topics people are talking about without tagging them. For that case, you need more than regex. A solution is to use <a href="https://www.scraping.club/p/using-nlp-scraped-data">NLP-based tokenization for entity extraction</a> on top of your script, like the following:</p><pre><code><code># pip install spacy
# python -m spacy download en_core_web_sm

import spacy  
  
nlp = spacy.load("en_core_web_sm")  
  
def extract_keywords(text):  
    doc = nlp(text.lower())  
    return [  
        token.lemma_ for token in doc  
        if not token.is_stop and not token.is_punct and token.pos_ in ("NOUN", "PROPN")  
    ]  
  
keywords = extract_keywords("New drop today, the sneakers are insane")  
print(keywords)   </code></code></pre><p>And the result is:</p><pre><code><code>['drop', 'today', 'sneaker']</code></code></pre><p>In that case, this content can tell you:</p><ul><li><p>drop &#8594; Something has it or is hitting the market.</p></li><li><p>today &#8594; That something occurred today.</p></li><li><p>sneaker &#8594; The actual product.</p></li></ul><p>Combine both approaches, and you&#8217;ve got a keyword pool you can feed into frequency counting or topic clustering downstream, without hand-tagging every post yourself.</p><p>As a final consideration, note that the NLP-based approach also has limitations. The main argument we could discuss is that this solution is based on the vocabulary you use. So, another technique is to use AI to extract entities and patterns from text. If you want to deepen that, we already covered this in the article &#8220;<a href="https://www.scraping.club/p/detect-pattern-scraped-data-with-ai">Using AI to Detect Patterns in Scraped Data</a>&#8221;.</p><h3>Capturing Engagement Metrics Over Time</h3><p>Hashtags, mentions, and keywords tell you what people are talking about. Engagement metrics tell you how fast the news is spreading. And that&#8217;s a moving target. This means that you can&#8217;t scrape a post once and call it a day. You need to hit the same post at certain time intervals and build a time series out of it. The following code helps you do that:</p><pre><code><code>import time  
import csv  
from datetime import datetime  
  
def scrape_engagement(post_id, fetch_fn):  
    data = fetch_fn(post_id)  
    return {  
        "post_id": post_id,  
        "timestamp": datetime.utcnow().isoformat(),  
        "likes": data["likes"],  
        "shares": data["shares"],  
        "comments": data["comments"],  
        "views": data.get("views", 0),  
    }  
  
def track_post(post_id, fetch_fn, interval_sec=900, iterations=8):  
    with open(f"{post_id}_engagement.csv", "a", newline="") as f:  
        writer = csv.DictWriter(f, fieldnames=["post_id", "timestamp", "likes", "shares", "comments", "views"])  
        if f.tell() == 0:  
            writer.writeheader()  
  
        for _ in range(iterations):  
            row = scrape_engagement(post_id, fetch_fn)  
            writer.writerow(row)  
            f.flush()  
            time.sleep(interval_sec)  </code></code></pre><p>In this example code:</p><ul><li><p><code>fetch_fn</code> is whatever extraction logic you&#8217;re using for that specific platform.</p></li><li><p>The <code>interval_sec</code> and <code>iterations</code> are up to you. Tighter intervals catch virality earlier, but they also burn through your proxy pool faster.</p></li></ul><p>Once you&#8217;ve got this data logged, you can compute engagement velocity (delta between snapshots) and feed it straight into the time-series analysis you&#8217;ll run downstream. Then, <a href="https://www.scraping.club/p/building-a-scraper-dashboard-streamlit">build a dashboard - for example, using Streamlit</a> - and use <a href="https://www.scraping.club/p/analyzing-scraped-data-pandas-matplotlib">Pandas, Matplotlib, and related libraries to analyze the scraped data</a> and get trends out of your time series.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://substack.thewebscraping.club/p/consulting&quot;,&quot;text&quot;:&quot;Need help with your scraping project?&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://substack.thewebscraping.club/p/consulting"><span>Need help with your scraping project?</span></a></p><div><hr></div><h2>Data Cleaning and Normalization</h2><p>Social media trends don&#8217;t live in a single platform. A topic can break on Reddit, spike on X six hours later, and only show up on TikTok in a couple of days. So, if you&#8217;re only scraping one source, you risk tracking a fragment of the whole trend. As a consequence, if raw scraped data is messy by default when you get it from a single source, you can imagine the chaos with multiple sources.</p><p>This means your pipeline needs to standardize data across platforms. Otherwise, you&#8217;ll be comparing apples to oranges without realizing it. So let&#8217;s see some techniques to do so.</p><h3>Deduplication and Bot/Spam Filtering</h3><p>Bots and spammers are a real trouble for the web (they were so even in the &#8220;forums era&#8221;, if you&#8217;re old enough to remember it &#128513;). In the context of trend detection, if you consider them and reposts, you get inflated engagement. In other words, if you don&#8217;t filter them out, you&#8217;ll end up &#8220;detecting&#8221; trends that are really just coordinated inauthentic behavior.</p><p>Deduplication is the easy part. For that, content hashing gets you most of the way there as follows:</p><pre><code><code>import hashlib

posts = [
    {"text": "AI is eating the world #AI #MachineLearning"},
    {"text": "AI is eating the world #AI #MachineLearning"},  # exact duplicate
    {"text": "  ai IS eating the world #AI #MachineLearning  "},  # duplicate after normalization
    {"text": "New drop today #sneakers #AI incoming"},
    {"text": "Another unrelated post about crypto"},
]

def content_hash(text):
    normalized = text.strip().lower()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

seen_hashes = set()
unique_posts = []

for post in posts:
    h = content_hash(post["text"])
    if h not in seen_hashes:
        seen_hashes.add(h)
        unique_posts.append(post)

print(f"Original posts: {len(posts)}")
print(f"Unique posts: {len(unique_posts)}")
for p in unique_posts:
    print("-", p["text"])</code></code></pre><p>This code:</p><ul><li><p>Defines a <code>content_hash()</code> function that strips whitespace and lowercases the post text, then computes a SHA-256 hash of it. Note that normalizing before hashing means that formatting differences (extra spaces, capitalization) don&#8217;t prevent identical content from being detected as duplicates.</p></li><li><p>Iterates over the list of <code>posts</code>, computing a hash for each one.</p></li><li><p>Keeps a <code>seen_hashes</code> set to track which content hashes have already been encountered.</p></li></ul><p>So, if a post&#8217;s hash is new, it&#8217;s added to both <code>seen_hashes</code> and the <code>unique_posts</code> list; if the hash already exists, the post is skipped as a duplicate. The result, <code>unique_posts</code>, contains only the first occurrence of each distinct piece of content. When runing it, you&#8217;ll see the following:</p><pre><code><code>Original posts: 5
Unique posts: 3
- AI is eating the world #AI #MachineLearning
- New drop today #sneakers #AI incoming
- Another unrelated post about crypto</code></code></pre><p>Now, the code above accounts only for exact matches. However, that won&#8217;t catch near-duplicates (slightly reworded reposts). For that, you&#8217;ll want something like MinHash or embedding similarity:</p><pre><code><code># pip install sentence-transformers

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")

posts = [
    {"text": "AI is eating the world"},
    {"text": "Artificial intelligence is taking over the world"},
    {"text": "New sneaker drop today"},
    {"text": "Sneakers just dropped today"},
    {"text": "Completely unrelated topic about cooking pasta"},
]

def find_near_duplicates(posts, threshold=0.6):
    texts = [p["text"] for p in posts]
    embeddings = model.encode(texts, convert_to_tensor=True)
    duplicates = []

    for i in range(len(texts)):
        for j in range(i + 1, len(texts)):
            sim = util.cos_sim(embeddings[i], embeddings[j]).item()
            if sim &gt; threshold:
                duplicates.append((i, j, sim))
    return duplicates

results = find_near_duplicates(posts, threshold=0.6)
for i, j, sim in results:
    print(f"({i}, {j}) sim={sim:.3f} -&gt; '{posts\[i\]['text']}' | '{posts\[j\]['text']}'")</code></code></pre><p>This snippet does the following:</p><ul><li><p>Loads a pre-trained sentence embedding model (<code>all-MiniLM-L6-v2</code>) via <code>SentenceTransformer</code>.</p></li><li><p>Encodes each post&#8217;s text into a dense vector embedding that captures its semantic meaning, not just exact wording.</p></li><li><p>Compares every pair of posts using cosine similarity (<code>util.cos_sim</code>) to measure how semantically close their content is.</p></li><li><p>Flags pairs whose similarity score exceeds the given <code>threshold</code> as near-duplicates. This catches reworded or paraphrased reposts that exact-hash matching (like the previous <code>content_hash</code> example) would miss.</p></li><li><p>Returns a list of tuples <code>(i, j, similarity_score)</code> identifying which post pairs are likely duplicates.</p></li></ul><p>As this solution uses machine learning, the actual similarity scores depend on the model run. So, the score you get can be slightly different from the following:</p><pre><code><code>(0, 1) sim=0.72 -&gt; 'AI is eating the world' | 'Artificial intelligence is taking over the world' (2, 3) sim=0.81 -&gt; 'New sneaker drop today' | 'Sneakers just dropped today'</code></code></pre><p>And, yes: this is a pretty good result!</p><h3>Normalizing Timestamps and Multilingual Text</h3><p>Cross-platform trend detection breaks down fast if you don&#8217;t normalize timestamps. This is because every platform reports time differently: some give you UTC ISO strings, others hand you Unix epochs, and some (looking at you, scraped HTML!) give you relative strings like &#8220;3h ago&#8221; that you have to resolve at scrape time before they become meaningless.</p><p>Timestamp conversion is the easy of the two:</p><pre><code><code>from datetime import datetime, timezone
import dateutil.parser

def normalize_timestamp(raw_ts):
    if isinstance(raw_ts, (int, float)):
        return datetime.fromtimestamp(raw_ts, tz=timezone.utc)
    return dateutil.parser.parse(raw_ts).astimezone(timezone.utc)

print(normalize_timestamp(1735689600))
print(normalize_timestamp("2025-01-01T00:00:00+05:30"))</code></code></pre><p>But relative timestamps need to be resolved against your scrape time, not stored as-is:</p><pre><code><code>import re
from datetime import timedelta

def resolve_relative_time(raw, scraped_at):
    match = re.match(r"(\d+)([hmd])", raw)
    if not match:
        return scraped_at
    value, unit = int(match.group(1)), match.group(2)
    delta = {"m": timedelta(minutes=value), "h": timedelta(hours=value), "d": timedelta(days=value)}[unit]
    return scraped_at - delta</code></code></pre><p>If you think that managing time is hard, consider that language is the harder problem. A trend can absolutely break in a non-English-speaking market first, and if your pipeline only handles English, you&#8217;ll miss it entirely. But the problem is that multilingual analysis can&#8217;t just be &#8220;run everything through a translator&#8221;. That&#8217;s where a lot of &#8220;translation&#8221; pipelines get sloppy.</p><p>To account for that, you first need language detection:</p><pre><code><code># pip install langdetect

from langdetect import detect
def detect_language(text):
    try:
        return detect(text)
    except Exception:
        return "unknown"
texts = [
    "AI is eating the world, and everyone is talking about it.",
    "L'intelligence artificielle est en train de conqu&#233;rir le monde.",
]
for t in texts:
    print(f"'{t}' -&gt; {detect_language(t)}")</code></code></pre><p>And the result you get is as expected:</p><pre><code><code>'AI is eating the world, and everyone is talking about it.' -&gt; en
'L'intelligence artificielle est en train de conqu&#233;rir le monde.' -&gt; fr</code></code></pre><p>Then, you can go through translation. Beyond translation quality, tokenization itself gets messy across languages. <a href="https://www.elastic.co/docs/reference/text-analysis/analysis-whitespace-tokenizer">Whitespace-based tokenizers</a> (fine for English) fall apart on languages without clear word boundaries, like Japanese or Chinese, where you need dedicated segmenters.</p><p>Overall, covering every nuance of multilingual translation and tokenization in depth is beyond the scope of this article. But knowing where these pitfalls lie helps you design a pipeline that&#8217;s ready to handle them as your trend-detection needs grow.</p><div><hr></div><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><div><hr></div><h2>Analyzing Trends from Scraped Data</h2><p>Clean data is a reliable starting point, but it doesn&#8217;t tell you what&#8217;s trending. Now, your social media scraping pipeline needs to distinguish normal conversation from unusual growth, identify what people are actually discussing, and measure how that discussion changes over time.</p><p>And there&#8217;s an important distinction to make: a popular topic isn&#8217;t necessarily an emerging trend. A topic with consistently high mention volume might just be part of the platform&#8217;s normal activity. Meanwhile, a smaller conversation that&#8217;s spreading across independent communities could deserve your attention much earlier.</p><p>Basically, you need two complementary layers of analysis:</p><ul><li><p><strong>Time-series analysis for virality detection:</strong> Measure changes in mention volume, participation, and engagement against an appropriate baseline.</p></li><li><p><strong>Sentiment analysis and topic modeling:</strong> Identify the themes behind those changes and understand how people feel about the specific subjects involved.</p></li></ul><p>Both kinds of analysis can require a lot of work and, surely, require specialized engineers. So, allow me to just give you the high-level ideas that matter.</p><h3>Time-Series Analysis for Virality Detection</h3><p>Start by deciding what you&#8217;re counting and when you&#8217;re counting it. This sounds obvious, but it&#8217;s where you can accidentally turn a collection artifact into a trend.</p><p>For mention volume, you should group posts into fixed time windows using their publication timestamps. For engagement growth, use the timestamps of your repeated metric observations. These aren&#8217;t interchangeable, because a post published yesterday can receive new engagement today.</p><p>So, your time-series dataset should distinguish:</p><ul><li><p><strong>Mention volume:</strong> The number of distinct collected posts mentioning a topic within each window.</p></li><li><p><strong>Unique authors:</strong> The number of distinct participating accounts, measured within each platform unless cross-platform identities are reliably linked.</p></li><li><p><strong>Engagement increments:</strong> The change in a post&#8217;s likes, comments, shares, or views between observations.</p></li><li><p><strong>Collection coverage:</strong> Whether collection succeeded, which sources were available, and whether sampling or query settings changed.</p></li></ul><p>That last point matters more than it might seem because if your scraper stops collecting for an hour, you have missing observations.</p><h3>Sentiment Analysis and Topic Modeling</h3><p>Time-series analysis tells you that something changed. NLP helps you understand what changed.<br>But don&#8217;t make the mistake of collapsing sentiment and topic detection into the same task because:</p><ul><li><p>A topic model groups related discussions.</p></li><li><p>A sentiment model estimates expressed polarity toward something.</p></li></ul><p>This means that neither, on its own, explains why a conversation is growing.</p><p>Also, note that the results - especially related to sentiment analysis - highly depend on the specific neural network you&#8217;re using. This is a wide topic, and we already covered it with a practical tutorial. If you want to deepen it, check out our article &#8220;<a href="https://www.scraping.club/i/170801049/sentiment-analysis-on-scraped-product-reviews-step-by-step-tutorial">Sentiment Analysis on Scraped Product Reviews: Step-by-step Tutorial</a>&#8221;.</p><h2>Putting It All Together: From Social Media Scraping to Trend Insights</h2><p>At this point, you&#8217;ve seen the main pieces of the pipeline. But the actual value comes from connecting them. Social media scraping gets you the posts; extraction, cleaning, and analysis turn those posts into signals you can interpret.</p><p>So, before you start building, here&#8217;s how the whole process fits together:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!jeWa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!jeWa!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png 424w, https://substackcdn.com/image/fetch/$s_!jeWa!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png 848w, https://substackcdn.com/image/fetch/$s_!jeWa!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png 1272w, https://substackcdn.com/image/fetch/$s_!jeWa!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!jeWa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png" width="1186" height="1426" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1426,&quot;width&quot;:1186,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!jeWa!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png 424w, https://substackcdn.com/image/fetch/$s_!jeWa!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png 848w, https://substackcdn.com/image/fetch/$s_!jeWa!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png 1272w, https://substackcdn.com/image/fetch/$s_!jeWa!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa17877a6-6e62-4413-b8bd-ad80718221a8_1186x1426.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>In practical terms, these are the stages your pipeline needs to cover:</p><ul><li><p><strong>Define what you&#8217;re tracking and where you can collect it</strong>: Choose platforms around your trend-analysis goals, then review the relevant access rules and legal requirements. Using a scraping provider changes who operates the infrastructure, but it doesn&#8217;t automatically remove your responsibilities around data collection and use.</p></li><li><p><strong>Collect posts and revisit engagement metrics</strong>: Decide whether to use official APIs, your own collectors, or managed services depending on your requirements. Store source identifiers, publication timestamps, and observation timestamps alongside the content. Remember that a dataset collected once can support historical analysis, but tracking engagement growth requires repeated observations.</p></li><li><p><strong>Extract signals, then clean and normalize them</strong>: Isolate hashtags, mentions, keywords, and engagement counters. Standardize timestamps and handle language differences before comparing sources. Also, distinguish duplicate collection from actual reposting: collecting the same post twice is a data-quality issue, while independent reposts can be evidence of a conversation spreading. Don&#8217;t automatically discard that signal.</p></li><li><p><strong>Analyze changes rather than raw popularity alone</strong>: Compare mention volume and engagement velocity against historical activity, then use topic modeling and sentiment analysis to understand the discussions behind those changes. And keep checking collection coverage.</p></li><li><p><strong>Visualize the results and monitor the pipeline</strong>: We haven&#8217;t explored data visualization in depth here, but it&#8217;s the final layer that makes your results easier to interpret. A <a href="http://thewebscrapingclub.substack.com/p/building-a-scraper-dashboard-streamlit">Streamlit dashboard</a>, for example, can display mention-volume curves, engagement growth, and topic or sentiment breakdowns. Just make sure it also shows data freshness and collection gaps, so readers know what they&#8217;re looking at.</p></li></ul><p>Overall, the thing to remember is this one: you don&#8217;t get reliable trends by only collecting more posts. You get them by keeping the whole pipeline consistent, from source selection to the dashboard. And when a spike appears, you should be able to trace it back to the underlying posts and collection conditions.</p><h2>Conclusion</h2><p>In this article, you learned that social media scraping for trend analysis is about more than collecting posts and counting hashtags. You need a pipeline that brings together platform-specific collection, clean data, repeated engagement measurements, and analysis that distinguishes unusual growth from normal activity.</p><p>The value is in how you connect those pieces. A spike in mentions doesn&#8217;t tell you much if you can&#8217;t tell whether it came from independent conversations, repeated content, or a change in your collection coverage. But a system that helps you understand what&#8217;s gaining attention, where it&#8217;s spreading, and how the discussion is changing? That&#8217;s where the real value is!</p><p>So, let us know: Are you using social media scraping to track trends? Which platforms are you collecting from, and what&#8217;s been the hardest part of turning those posts into useful signals? Let&#8217;s discuss in the comments!</p><div><hr></div><p><em>Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.</em></p><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scraping.club/p/how-to-scrape-social-media?utm_source=substack&utm_medium=email&utm_content=share&action=share&quot;,&quot;text&quot;:&quot;Share&quot;,&quot;action&quot;:null,&quot;class&quot;:null}" data-component-name="ButtonCreateButton"><a class="button primary" href="https://www.scraping.club/p/how-to-scrape-social-media?utm_source=substack&utm_medium=email&utm_content=share&action=share"><span>Share</span></a></p><p></p>]]></content:encoded></item><item><title><![CDATA[The LAB #116: Warming an Imperva reese84 cookie, and replaying it without a browser]]></title><description><![CDATA[A tenth of a cent buys thirteen minutes of scraping, from any IP I tried]]></description><link>https://www.scraping.club/p/the-lab-116-warming-an-imperva-reese84</link><guid isPermaLink="false">https://www.scraping.club/p/the-lab-116-warming-an-imperva-reese84</guid><dc:creator><![CDATA[Pierluigi Vinciguerra]]></dc:creator><pubDate>Thu, 24 Sep 2026 15:15:17 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/013af3b9-d3cb-4631-a5e1-ba1cbd3d9c6a_1456x1048.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Imperva is one of the anti-bots we didn&#8217;t write a lot about in the past, probably because, for some reason I ignore, I&#8217;ve never encountered it in my career of scraping ecommerce websites (or maybe, if I encountered, it was so easy to bypass that I didn&#8217;t notice that). </p><p>To fill this gap, today I&#8217;m creating a case study for smythstoys.com, the Irish site of Smyths Toys. I picked it because a reader described the exact problem on the r/webscraping subreddit: Smyths runs Imperva, the product pages are server-rendered HTML, and the one internal endpoint that returns stock is protected too. The idea in that thread was to keep one hot cookie per proxy, forged using a browser, and replay it over cheap datacenter IPs. I wanted to check three things: does that actually work, how long does the cookie last, and when does it eventually break?</p><p>This time I worked two ways at once. I built a bypass that warms a session with a browser, solves the challenge, and then runs on a plain HTTP client. And I ran Manus against the same site to read the protection from the inside (so I could explain why the bypass keeps working, instead of only showing that it does). The bypass is mine, while the deep read of the sensor, the token, and the renewal protocol is Manus&#8217;s, checked against my own captures where the two overlap.</p><blockquote><p>Give your AI a web data layer &#8211; <strong><a href="https://visit.decodo.com/rE0PQB">Decodo&#8217;s Web Scraping API</a></strong> turns any site into clean, structured data your models can actually use. </p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://visit.decodo.com/rE0PQB" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!aRfI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png" width="724" height="362" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;normal&quot;,&quot;height&quot;:512,&quot;width&quot;:1024,&quot;resizeWidth&quot;:724,&quot;bytes&quot;:716985,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:&quot;&quot;,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:&quot;https://visit.decodo.com/rE0PQB&quot;,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://substack.thewebscraping.club/i/204450518?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:&quot;center&quot;,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!aRfI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 424w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 848w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1272w, https://substackcdn.com/image/fetch/$s_!aRfI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F33ef41f8-3e73-4fa1-a516-a17d90e53e20_1024x512.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://visit.decodo.com/rE0PQB&quot;,&quot;text&quot;:&quot;Activate starter plan&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://visit.decodo.com/rE0PQB"><span>Activate starter plan</span></a></p></blockquote><h2>Manus, and how I used it here</h2><p>Manus is an agent that pairs a language model with a sandboxed Linux VM and a Chromium browser. It can drive a page, run Node and Python, and instrument the browser while it executes, which is what makes it useful here. You may have noticed that in the latest The Lab issues I used that a lot:  in <a href="https://www.scraping.club/p/deobfuscating-kasada-with-manus-ai">THE LAB #114</a> against Kasada and <a href="https://www.scraping.club/p/datadome-on-leboncoin-scraping">THE LAB #115</a> against DataDome. <br>This is mainly for two reasons: it uses the sandboxed VMs, so it can spawn a browser and run tests in its virtual environment, and, at the same time, it&#8217;s not so heavily guardrailed, so these tasks can be performed with ease.</p><p>For this article, Manus hooked <code>JSON.stringify</code> and the <code>btoa</code> boundary before the page scripts ran, so it logged the exact sensor request and the clear-text fingerprint the moment they were built. Then it ran cold, warm and mutation matrices against the token endpoint. Its run was on 18 September 2026, in headless Chromium 151.0.7922.71, anonymous, thirteen page navigations, no credentials and no CAPTCHA solving. One detail from that setup matters later: its browser reported <code>navigator.webdriver=true</code>, so it announced itself as automated.</p><p>I trust the numbers Manus reported across those runs, and I checked the ones that agree with my own captures. Where I quote a Manus figure I say so, because I did not reproduce its in-browser instrumentation myself. Manus also did not vary the IP, the User-Agent or the TLS fingerprint, and it did not solve a CAPTCHA, so those parts of the story are mine.</p><h2>The protection at a high level</h2><p>Before jumping into the details, let&#8217;s see what happens if we cURL the target website:</p><pre><code><code>$ curl -s -D - -A "Mozilla/5.0 ... Chrome/140.0.0.0 Safari/537.36" https://www.smythstoys.com/ie/en-ie/
HTTP/2 200
x-cdn: Imperva
x-iinfo: 10-32110005-0 NNNN RT(...) ...
set-cookie: visid_incap_2483049=...; Domain=.smythstoys.com; ...
set-cookie: incap_ses_416_2483049=...; Domain=.smythstoys.com; ...</code></code></pre><p>The <code>x-cdn: Imperva</code> header and the <code>visid_incap</code> and <code>incap_ses</code> cookies are the Incapsula signature. Of course, what we receive it&#8217;s not the home page of the website but an error page. It&#8217;s a 1,158 byte HTML shell with an iframe that says &#8220;Request unsuccessful. Incapsula incident ID: ...&#8221;. So the edge already decided my plain client is not welcome, before any product page is served.</p><p>There are five pieces here, and it helps to name them before going deep on any one. The edge WAF answers the navigation with the interstitial (the &#8220;Pardon Our Interruption&#8221; page that sits between you and the shop), or with the real shop once you are cleared. A throwaway fingerprint cookie called <code>___utmvc</code> is built in the browser and lives for about twenty seconds. The reese84 sensor is the real collector, two large obfuscated scripts of around 800 KB each loaded from paths that look like random english words. The token exchange is the POST that sends the sensor result and gets the reese84 token back. And the GeeTest CAPTCHA is what Imperva falls back to when the sensor result alone is not enough.</p><div><hr></div><blockquote><p><em>Tired of getting blocked while scraping the web?<br>ScrapingBee handles proxies, browsers, anti-bot systems, and retries so you can focus on your data. Get clean Markdown, JSON, or HTML from the web with up to a 99.9% success rate. ScrapingBee is SOC 2 Type II and GDPR compliant, and trusted by 4,000+ developers.</em></p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Pj1_!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 424w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 848w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png" width="1456" height="437" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:437,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:128210,&quot;alt&quot;:&quot;&quot;,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/216891176?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" title="" srcset="https://substackcdn.com/image/fetch/$s_!Pj1_!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 424w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 848w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1272w, https://substackcdn.com/image/fetch/$s_!Pj1_!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F355793ca-3a3b-4369-8c43-0f54aaa79b69_2000x600.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"></figcaption></figure></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club&quot;,&quot;text&quot;:&quot;Discover ScrapingBee&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.scrapingbee.com/?utm_source=sponsored_newsletter&amp;utm_medium=email&amp;utm_campaign=the_web_scraping_club"><span>Discover ScrapingBee</span></a></p></blockquote><div><hr></div><p>Manus drew the whole thing as one diagram, which is the clearest way to see how these pieces connect.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Fr5X!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Fr5X!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png 424w, https://substackcdn.com/image/fetch/$s_!Fr5X!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png 848w, https://substackcdn.com/image/fetch/$s_!Fr5X!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png 1272w, https://substackcdn.com/image/fetch/$s_!Fr5X!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!Fr5X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png" width="1456" height="2660" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/caf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:2660,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:535321,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://www.scraping.club/i/216891176?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Fr5X!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png 424w, https://substackcdn.com/image/fetch/$s_!Fr5X!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png 848w, https://substackcdn.com/image/fetch/$s_!Fr5X!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png 1272w, https://substackcdn.com/image/fetch/$s_!Fr5X!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcaf72c38-005e-41ce-87ac-4a0549f4b059_3120x5700.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><em>The reese84 flow end to end, mapped from the Manus run. It shows the edge challenge and the two browser collectors, the short-lived </em><code>___utmvc</code><em> helper and the reese84 interrogator. It then splits into the cold, warm and replacement token paths, and ends at the edge policy check. That last box is where a CAPTCHA-solved jar and an interrogation-only jar go different ways.</em></figcaption></figure></div><p>The server-side scoring is out of scope, for the simple reason that nobody outside Imperva can see it. Everything I claim below is something one of us saw live, in the cookie jar, or in the sensor code as it ran. We have covered the general idea before: cookies and sessions as a clearance store in <a href="https://www.scraping.club/p/the-lab-94-using-cookies-and-session">THE LAB #94</a>, and a browser-to-HTTP handoff in <a href="https://www.scraping.club/p/hybrid-scraping-camoufox-curl-cffi">THE LAB #100</a>. This is the same shape, applied to Imperva, with the extra question of what the token is tied to.</p><h2>The bypass: warm a session, then run without a browser</h2><p>The bypass has three tools, each for one job.</p><p><a href="https://github.com/D4Vinci/Scrapling">Scrapling</a> does the warming. Its <code>StealthySession</code> launches a Chromium browser and keeps cookies across requests. I use it because it gives me a <code>page_action</code> callback with the real Playwright page inside, so I can watch the network and drive the challenge frame by hand.</p><p><a href="https://www.capsolver.com/">CapSolver</a> solves the CAPTCHA. Imperva shows a GeeTest challenge when the sensor alone is not enough, and CapSolver has a GeeTest task type that returns the three values GeeTest wants back.</p><p><a href="https://github.com/yifeikong/curl_cffi">curl_cffi</a> does the replay. It sends HTTP requests with the TLS fingerprint of a real Chrome, without a browser, so I can test whether the warmed cookie works on its own.</p><p>All of it runs in the shared venv on Python 3.11. My code is in the <code>116-Imperva-Reese84</code> directory of <a href="https://github.com/TheWebScrapingClub/thelab">The Lab repository</a>, with the scripts and the captured results, so you can rerun every number that carries my name below.</p><p>When I warm a session with Scrapling and log every request on smythstoys.com, the order is always the same. The browser loads the home page and gets the interstitial. The sensor scripts load. The sensor POSTs its blob to a random path, and a second POST returns the token:</p><pre><code><code>POST /mbit-And-Dirers-him-Face-and-sure-such-Parry-qui?d=www.smythstoys.com
{"token":"3:nHkG6Ry2bxW+gbWD+elLTg==:RZgGQ1gd...","renewInSec":699,"cookieDomain":"www.smythstoys.com"}</code></code></pre><p>That <code>token</code> value becomes the <code>reese84</code> cookie. The <code>renewInSec</code> is 699, so the browser is told to refresh the token in about eleven and a half minutes. The <code>cookieDomain</code> is the host, so the cookie is scoped to <code>www.smythstoys.com</code>. On Smyths this token alone was never enough for my headless Chromium. Every warming run ended on a GeeTest challenge, so I had to solve it.</p><div><hr></div><p class="button-wrapper" data-attrs="{&quot;url&quot;:&quot;https://www.youtube.com/@thewebscrapingclub&quot;,&quot;text&quot;:&quot;Check the TWSC YouTube Channel&quot;,&quot;action&quot;:null,&quot;class&quot;:&quot;button-wrapper&quot;}" data-component-name="ButtonCreateButton"><a class="button primary button-wrapper" href="https://www.youtube.com/@thewebscrapingclub"><span>Check the TWSC YouTube Channel</span></a></p><div><hr></div><p>The challenge flow is readable in the interstitial frame. A GET to <code>/_Incapsula_Resource?SWCNGEEC=...</code> returns the GeeTest parameters:</p><pre><code><code>{"challenge": "230a350828feeac904b737841e83ecd8", "gt": "ce33de396f8d04030f6eca8fbd225070", "new_captcha": true, "success": 1}</code></code></pre><p>The <code>gt</code> was the same across every run, <code>ce33de396f8d04030f6eca8fbd225070</code>, which is the site&#8217;s GeeTest id. The <code>challenge</code> changes each time. When the widget is solved, the page POSTs the result to <code>/_Incapsula_Resource?SWCGHOEL=gee&amp;...</code> with <code>geetest_challenge</code>, <code>geetest_validate</code> and <code>geetest_seccode</code>, then reloads the parent frame.</p><p>My <code>warm_session.py</code> does exactly that, from inside the frame. It reads the two URLs out of the frame&#8217;s HTML, calls the GET to get <code>gt</code> and <code>challenge</code>, sends those to CapSolver as a <code>GeeTestTaskProxyLess</code> task, and gets back the three values. Then it POSTs them to the <code>SWCGHOEL=gee</code> URL using the frame&#8217;s own <code>fetch</code>, so the request carries the session cookies, and reloads the page.</p><p>CapSolver returned a solution in about ten to twelve seconds each time. It&#8217;s not perfectly reliable: I saw it fail with <code>ERROR_CAPTCHA_SOLVE_FAILED</code> code 1001 on some challenges, so the script retries up to three times. GeeTest solves cost one dollar per thousand on CapSolver when I ran this, so a failed attempt followed by a good one is still a fraction of a cent. After the solved reload, the page title finally read &#8220;Smyths Toys Superstores | Buy Toys for Kids&#8221;, which is the real shop. Here is the tail of a real run, warmed through an Irish residential proxy:</p><pre><code><code>attempt 1: challenge 6abbf70a... gt ce33de396f8d04030f6eca8fbd225070
capsolver solved in 10.3s
imperva captcha submit -&gt; 200 '&lt;html&gt;&lt;head&gt;&lt;script&gt;window.parent.location.reload(true);&lt;/script&gt;&lt;/head&gt;...'
SAVED status=REAL_AFTER_CAPTCHA cookies=[incap_ses, reese84, incap_sh, visid_incap, nlbi, nlbi]</code></code></pre><p>Now the part that makes the whole thing worth doing: does the clearance survive without the browser? I take the cookie jar from that warmed session (the full set of cookies the browser collected) and replay it with curl_cffi, no browser, impersonating Chrome. I fetch the home page and the Magic The Gathering category page the reader mentioned:</p><pre><code><code>jar: warm_pub01.json  status=REAL_AFTER_CAPTCHA  minted 86s ago
warm proxy (same exit)   | home      | 200  603618 REAL | 'Smyths Toys Superstores | Buy Toys for Kids'
warm proxy (same exit)   | category  | 200  471472 REAL | 'Magic The Gathering | Smyths Toys Ireland'
replay proxy             | home      | 200  603752 REAL | 'Smyths Toys Superstores | Buy Toys for Kids'
replay proxy             | category  | 200  471605 REAL | 'Magic The Gathering | Smyths Toys Ireland'
direct (no proxy)        | home      | 200  603751 REAL | 'Smyths Toys Superstores | Buy Toys for Kids'
direct (no proxy)        | category  | 200  471605 REAL | 'Magic The Gathering | Smyths Toys Ireland'</code></code></pre><p>The warmed jar returns the real page from the same Irish residential IP that created it, from a datacenter IP in Ohio, and from my own connection in Italy. I ran the same jar from a second Irish residential IP too, with the same result. So, at least in my tests, the clearance is not bound to the minting IP. This is what makes the reader&#8217;s architecture work: you warm on a good residential IP, then you replay over whatever is cheapest.</p><p>I went further and varied the HTTP client itself, same warmed jar, direct connection:</p><pre><code><code>chrome136 + Chrome UA (match)      | 200  467891 REAL
firefox135 + Firefox UA            | 200  467891 REAL
chrome136 + Firefox UA             | 200  467891 REAL
no impersonation + Chrome UA       | 200  467891 REAL</code></code></pre><p>On this site, once the jar is warmed, the TLS fingerprint and the User-Agent stop mattering. Even curl_cffi with no impersonation, which sends a TLS handshake no browser would send, gets the real page. So the warmed reese84 works as a bearer token (a token that works for whoever presents it, with nothing tying it to one client): any client that presents the jar gets the real page back.</p><p>That is the working bypass. The rest of the article explains why it holds, using the Manus report.</p><h2>Why the replay works</h2>
      <p>
          <a href="https://www.scraping.club/p/the-lab-116-warming-an-imperva-reese84">
              Read more
          </a>
      </p>
   ]]></content:encoded></item></channel></rss>