Back in November 2023, I published The true costs of a web scraping project, in which I wanted to understand whether, when we need a scraper, it is worth buying one (by which I mean buying the bypass technique using a web unblocker) or creating it in-house.
Given that there are so many aspects to consider, such as your knowledge of web scraping techniques and experience in the field, I tried to put some numbers to that.
What I carved out was a general good practice: if you need something just once, buy it. If you need a feed that runs for years, build it yourself, since you can spread out the setup cost and keep the running costs low. This assumption was based on the fact that, at that time, the most important voice cost was a developer's hourly rate.
Three years later, AI and LLMs changed these assumptions. Once you’ve properly set up your coding environment with skills and best practices, the cost of creating and even maintaining a scraper is near 0.
This makes the idea of building everything in-house more attractive again, and in this article we’ll dig into some numbers to understand if this is really true or not.
Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.
The first problem is the question itself. “Do we build or buy our scraping?” contains three layers, each with a different answer.
There are the code production costs, given by who writes the scraper and who fixes it when the site changes. There are infrastructure costs, which include proxies, stealth browsers, anti-bot bypass, and captcha solving. And there are the execution costs: who runs the scrapers and who keeps them running.
Until 2023, the code layer was the first cost to consider, followed by the infrastructure ones. Depending on the size of the scraping project, the costs of building could lean towards the first or the second layer. The execution layer, instead, depends on the tool: do you need a browser for scraping? If so, you need way more CPU and memory capacity compared to a browserless scraper.
The code layer in 2026
It would be easy to say that now code is “free”, so this layer went down to zero.
The reality is that the cost is near zero (you still buy tokens and have a human in a seat, for much less time), just for experienced teams that spent weeks on building the harness for their scraping agents.
I can share my example: I’ve spent some time on this task, creating a skill for every open source tool I’m using and also for the ones I’m not, but they could be useful for building scrapers on the hardest targets. Today there are 32 of them, about 33000 words, covering Camoufox, Patchright, nodriver, rnet, curl_cffi, Scrapling, RayoBrowse, Kameleo, Ghidra, Frida, and many others. Each skill says what the tool does, which parameters matter, and when to avoid it (don’t shoot a fly with a bazooka; it’s the first rule for a successful scraping project). My actual problem is that these skills are going to be stale soon, so the next step is to set up a monitor for each of these items and check the newest versions on GitHub, so skills can be updated automatically.
That’s, in my opinion, the best way to leverage LLMs to create and maintain in-house scraping projects.
If you’re starting today and you’re not interested in learning about scraping, since you have just a bunch of random requests, then scraping APIs are the tool for you. You just set them up and get the data, usually with no hassle even from medium-protected websites. You save money and time, unless the projects requires a huge amount of results to be scraped. The breaking point between buy and sell here depends strictly on your expertise and the size of the project.
Oxylabs Web API — built for agentic search
AI agents hallucinate, fresh data doesn’t. Our new Web API delivers fresh, real-time web data so your agents stay accurate, relevant, and ready to scale.Become an early user, try our new Web API, and share feedback so we can build a solution that better fits your needs.
Let’s say you decided to build your scraper in-house and wrote a working scraper using LLMs that returns a bunch of records. Now you have to launch a full run and maybe do it recurrently. That’s where things usually break, and the costs rise.
The buy vs. build on this layer in this case means: should we glue together in-house a stack of tools (proxies, TLS fingerprinting spoofing, browser), with the related running costs?
Buying here means getting a proxy provider, a web unblocker, or a browser-as-a-service. You pay per gigabyte or per request, according to what you’re buying. A vendor sells you a success rate, and it is worth more the less you want to think about the target.
While for the code layer, a structured team-building approach seems the obvious choice, here the choice is not so easy. There are more factors to keep in mind, apart from costs and development time: how critical is this scraping project? How much should I pay for the peace of mind of delegating this project to a vendor? How do I keep up with which tools are now bypassing anti-bot X? (spoiler: an auto-updated knowledge base like The Web Scraping Club Wiki).
The interesting part compared to 2023 is that now that almost every big name in the proxy industry is moving towards browser scraping, which can be used both for tackling hard websites and for agentic automation, unblocker APIs are getting a bit overshadowed. I’ve found that their prices have lowered to 1-2 USD per 1k requests, which makes the costs comparable to a high-priced residential proxy in some cases (which are usually sold per GB).
This is another challenge in answering the question of buy vs. build. How do I know the total price of my scraping project? I should consider the bandwidth for the proxies, the total number of requests in case of an unblocker API or Scraping Browser.
The execution layer, where your scraper actually runs, depends mainly on the tools you chose earlier. You can manage your own cloud setup, go serverless, or deploy the spider to a platform and forget the machine, sized properly. I went through the trade-offs in Optimizing costs for large-scale scraping operations: serverless is cheap for short, sporadic jobs but expensive when something runs all day. This is the layer that costs the least and, even with LLMs, can be managed more easily than before, so I won’t spend too much time on it.
Running the numbers again
Let me redo the 2023 exercise at the same $20 an hour, so the two can be compared. The hours in both columns are my estimates. For the model cost, I use Claude Opus 5 at its list price today: $5 per million input tokens and $25 per million output. A build session on an easy site burns something like 1M input and 100K output tokens once you count the retries, three times that on a medium site, fifteen times on a hard one. Batch requests cost half the list price, and cache reads cost a tenth of the input price, so the table shows the expensive version of the model's cost.
I added a traffic column to the calculation. When you buy an external tool, usually proxies are included, while when you build, the proxies are a separate invoice, and usually not a marginal one.
So I measured it. I took four ecommerce listing pages, on ikea.it, mediaworld.it, etsy.com and asos.com, and counted the bytes twice. A median to use in this calculation is 100 KB for an HTML fetch and 1.7 MB for a full render, with residential proxies at $4 per GB, a scraping browser at $8 per GB, and an unblocker at $1.50 per 1,000 requests with the proxies included.
I’ve made several assumptions for creating the following table, on top of the token calculated before:
A structured web scraping team is working on it, so we’ve all our skills and guidelines in place
I decided to compare build costs with unblockers for websites with no or light anti-bot protection, since we assume the HTML parsing phase is done in-house. This favors the buy solution, since scraping APIs are generally more expensive, but it’s also true that HTML parsing incurs a marginal cost when building a scraper with LLMs.
I categorized websites between light anti-bot (don’t require a browser for being scraped) and hard anti-bot (requiring a browser in-house or a scraping browser to buy).
Infrastructure costs are not included, since they’re relevant only in the hard anti-bot scenario.
Considering the three oversimplified scenarios in the table, a website without bot protection is not worth buying, since the costs of writing and running a scraper are near 0.
The middle scenario is the most ambiguous: depending on your time constraints, your real-world tool pricing, and LLM plans, it can be more convenient to buy or build.
On the hardest tier, both sides have to run a browser and carry the same 1.7 MB, so proxy and execution costs are way higher than in previous scenarios, and building should be convenient only after more than 100k requests are made.
Generally speaking, the break-even fell everywhere: by about two-thirds on the easy sites, by more than half in the middle, and by barely a tenth at the top. The cheaper the code got, the less it helped on the targets where the code was never the expensive part.
Maintenance moves the same way. In 2023 I budgeted two hours a month per scraper. When the fix is a selector or a changed JSON path, an agent proposes the patch and I confirm it, so half an hour is closer to reality. On the hard tier, nothing improves, because there the work is usually understanding how to bypass the website again, not just a selector change.
The commercial tools cover most of the web, and they cover it well. Then there’s a top slice of sites, few of them and usually the ones the business cares about most. There, the problem is a virtual machine inside an obfuscated script, or a sensor reading properties straight from the browser engine. No vendor publishes a price list for a specific interpreter on a specific domain, with an SLA.
Chatting around at the latest Oxycon, despite the fall in code creation costs, the R&D time spent on certain websites (especially Chinese ecommerces with custom-built solutions) makes them non-lucrative for companies selling scraping data.
Other considerations
Building now is cheap to start now and expensive to leave. It may take an afternoon to build fifty scrapers, but managing and checking their results require a small, dedicated team. The per-scraper cost fell, and the number of scrapers went up, so the total can easily be higher than before.
And there is the failure mode described in Stuck? More of the Same Won’t Do, published here in 2025: teams that build keep building well past the point where it pays. Agents make that easier, because progress feels continuous even when you’re stuck. Decide in advance how many weeks you give an internal attempt before you go and buy.
The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Final considerations
Build the code because that cost has collapsed, and what you get is yours. Buy the infrastructure, if you don’t want scraping to be your main business. The execution layer follows whatever ops capacity you already have.
The time you save goes into checking that the data is right. No model does that part for you yet.
Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.
