August 2022. Inside a hotel room in Tuscany, quite bored because of the rain, I started a project that I had been thinking about for some time. A blog about web scraping.
Never imagined that someone would have read it. It was just an excuse for writing down my scraping notes in a more structured way.
Four years later, both TWSC and the industry look very different.
Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.
Thank you to all the readers who think I’m saying something interesting enough to be subscribed to.
In this era, when we’re all fighting for attention, having 7,000+ people who choose to receive this newsletter and open it every week seems like a miracle.
This is despite no marketing efforts, 0 ads, and an issue with Google SEO that made the whole domain disappear from the results from one day to the next.
In the next few days, to solve this issue, there will be a new domain address that, according to my experiments, should fix this.
Let me also thank you, all the backers, paying readers, and companies that make this possible.
While still very lean, The Web Scraping Club in its current format could not run without sponsors and readers who choose to support it. And about the companies backing TWSC, let me spend a special word on the NetNut team, who are having a hard time now. I’ve met great people there, and they were among the first believers in this newsletter.
Behind the scenes, TWSC is far from perfect, but I’m doing my best to keep things running smoothly. I’ve just launched a partner portal, where sponsors can monitor their campaign. For the readers, this year we’ve released the new version of the Proxy Price benchmark tool, which I promise will be updated more often in the future. Also, the Scraping Wiki, our repo with all the findings of The Web Scraping Club organized as a second brain, is a recent new entry. Please give it a look and share your feedback.
What’s next
Apart from fixing the Google SEO issue in the immediate term, I’ve got a couple of ideas that still need time to be implemented, but hopefully they’ll see the light before the end of the year. Something I’ve been asked for a long time is a Job Board dedicated to open scraping positions and the great talent to fill them. Is this something you’d like to have here?
Ideally, I’d like to revamp the YouTube channel. And last but not least, I’m working on a web-scraping event in Italy in October. More details soon, but be ready for it!
The industry over the last four years looks very different.
We got LLMs; we know it, so the coding part is way faster. But what they brought to the industry is the legitimacy of web scraping.
It seems the world has recognized the role of web data and browser automation in innovation. While web data was once seen as something shady for spying on your competitors, it is now the main moat for training LLMs, and browser automation is the key tool for giving your agents real-time data.
This changed the perception of the industry as a whole, and the recent funding rounds (Nimble, Oxylabs) are the proof.
The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Think about it: have you ever heard of a funding round for a proxy company before 2023?
Conclusion
It’s an exciting time for the industry, and I’m glad to work at The Web Scraping Club right now. I’ll be at conferences around Europe (Oxycon in September and the Extract Summit in Dublin), which will allow me to meet great professionals and learn about the latest trends.
While both TWSC and the industry have changed a lot over the years, the spirit of this newsletter hasn't: no marketing fluff, honest reviews, some hot takes, and my personal opinions. I know that in four years, not all articles can be a 10 out of 10, but I can assure you that every article written here was the result of sweat and hard work. And it will be the same for the future.
Thanks for reading The Web Scraping Club! This post is public so feel free to share it.
Glad to be part of the project! ❤️