Welcome back to the Crawl of Fame; in this October edition, we’ll talk about social networks but from different angles. Let’s start from Reddit, which announced the end of its RSS feed and its public API, making it more difficult to scrape.
A few days later, a small company published the metadata of 5.6 billion TikTok videos on Hugging Face, which is remarkable since TikTok is known for being a hard target to scrape.
Then, from the legal perspective, LinkedIn closed another lawsuit against a scraping company, and we’ll dive a bit into it.
Give your AI a web data layer – Decodo’s Web Scraping API turns any site into clean, structured data your models can actually use.
Reddit closes RSS on November 13 and the public API in March 2027
On September 30, Reddit published two posts, one in r/modnews for moderators and one in r/redditdev for developers. Starting from October 31, Reddit won’t accept new requests for public API access, and from November 13, RSS feeds will stop working. Then, starting January 12, 2027, Reddit begins removing API access for apps and users that haven’t registered, and in March 2027, public access closes to everyone.
The reason is obvious and clearly stated by the moderators: “Because RSS is now a common surface for large-scale scraping and automated abuse, we’ll stop supporting RSS feeds on November 13, 2026”. This means that apps and bots should move to the Developer Platform (Devvit) managed by Reddit itself, rather than being hosted on third-party servers. And there will be no replacement for RSS feeds.
In addition, in the past months, Reddit already required logging in to the “old” Reddit endpoint, which was used as a backdoor by scrapers.
What’s left for people who collect Reddit data
After March 2027, the official paths are three: a Devvit app, a data licensing deal, or nothing. Devvit is made to build apps that run inside Reddit communities; it uses TypeScript, and Reddit told PPC Land that “adding additional language support has not been a scalable thing for us,” which is bad news for the many bots written with PRAW.
The licensing deals, of course, are where Reddit is pushing. A look at their balance sheet shows that the “other revenue” line, which includes data licensing, grew 24% year over year to $43 million in Q2 2026, according to TechCrunch. This is the way Google and OpenAI gather data, and they’re already paying for it.
For everyone else, the web version is what remains, and today it’s already behind a JavaScript check. In my opinion, from next spring Reddit scraping becomes a browser job with good residential proxies, and probably with logged-in accounts for anything that goes beyond the public listing. That’s way more expensive per page than an RSS feed, so I expect many small social listening tools and research projects to pay for a vendor or to stop collecting.
5.6 billion TikTok videos on Hugging Face
On October 3, a user called datasocial published tiktok-5.6B-videos on Hugging Face. Yes, it’s a dataset about 5.6 billion videos of TikTok, but there are just the metadata, not the videos themselves.
This means we have ID, publish date, country, language, caption, hashtags, mentions, on-screen text, sound ID, ad and TikTok Shop flags, the AI-generated flag when TikTok provides it, and the counts of views, likes, comments, shares, saves and downloads, with the date when they were read. The files are 148 parquet files, one per month from July 2014 to October 2026, for 460.5 GB in total.
How it was collected
The dataset card links to a long technical guide on datasocial.ai, and for us this is the most interesting part. Instead of scraping the website, they used the private API of TikTok’s Android app, with 24 endpoints. To make it work, they register fake devices built from a catalog of about 250 real Android profiles, they sign every request with the same headers the app uses (X-Argus, X-Ladon and the others), and they copy the TLS fingerprint of Android with uTLS. There’s no login anywhere, so everything behind an account is out of reach.
Two details from the guide are useful for anyone who works on mobile APIs. The first is that a failure is often a clean HTTP 200 with an empty body, and the guide counts four unrelated causes that produce it. The second is the proxy: since the rate limit is per exit IP, they use a rotating datacenter proxy at about $150 per month, with success rates between 88.2% and 99.3% depending on how you rotate and retry. With this setup, they write they collected 3.23 billion creator profiles, 5.94 billion videos and 2.8 billion comments in three weeks. I didn’t verify these numbers beyond the 5.6 billion rows of the dataset, but if they’re correct, it’s an impressive result for the cost of a datacenter proxy plan.
The guide is definitely worth reading, it could be helpful for other apps too.
LinkedIn wins against ProAPIs, and the scraper must delete everything
On September 16, Judge P. Casey Pitts of the Northern District of California signed a final judgment on consent in the case LinkedIn Corporation v. ProAPIs Inc. (I found it through The Record, then I read the documents on CourtListener). The defendants are ProAPIs, a Delaware company, Netswift, a Pakistani company founded in 2022, and their CEO Rehmat Alam.
LinkedIn filed the complaint on October 2, 2025, so the case lasted less than a year and never went to trial. The defendants never filed an answer and still deny liability, but they agreed to the judgment as part of a settlement.
The complaint is the interesting part for us, because it describes the business in detail. ProAPIs sold the iScraper API, which promised to “scrape LinkedIn data efficiently in real-time and at scale” at up to “150 requests / second”, with a plan at $15,000 per month for 5,000,000 API calls. According to LinkedIn, the data came from fake accounts: they created email addresses with automated tools, registered LinkedIn accounts with false names and stock photos, and used them to scrape while logged in. LinkedIn writes that it detects and restricts these accounts “within a few hours”, but in that time each account could scrape hundreds of profiles, and ProAPIs registered “hundreds if not thousands of new accounts per day”. In total, LinkedIn says they used more than a million fake accounts. They also scraped the profiles that members set as visible only to logged-in users, which contradicts the claim of ProAPIs that they only collected “data that’s public on personal profiles”.
It’s a clear case of massive scraping behind the login, something that is dangerous to do from a legal perspective, and this case confirms it.
The defendants are permanently barred from accessing LinkedIn with fake accounts, scrapers or bots, from circumventing any technical measure, and from selling or giving away LinkedIn data or any information derived from it (the text says “including for free”). They also can’t use any third-party service that provides LinkedIn data or accesses LinkedIn in an automated way, so they can’t simply become customers of another vendor. They certified, under penalty of perjury, that they deleted all the data, the reports made with it, the scraping code and the SDKs and APIs used to deliver it, and that they removed from their website every reference to LinkedIn scraping, “regardless of whether LinkedIn is referred to by name”.
For me, this case follows a pattern we have already seen. In the hiQ case, the court found a breach of the User Agreement for the same reason: the fake accounts used to scrape behind the login (I wrote about it in my article on LinkedIn’s hidden APIs). Proxycurl was sued by LinkedIn in January 2025 and shut down on July 4, 2025, and its founder explained that even winning wouldn’t pay the legal fees, while “LinkedIn, owned by Microsoft, has more or less an unlimited war chest”. ProAPIs settled in less than a year. In all three cases, the weak point is the same: when you create an account, you accept the User Agreement, and from that moment scraping becomes a breach of contract, before any discussion about public data. So if you buy LinkedIn data from a vendor today, ask them where it comes from, because a judgment like this one can close that source in a few months.
The Web Scraping Club is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
For this month, that’s it; see you in November with the next edition of the Crawl of Fame.
This matches what we see at Kameleo. Many of our users already run Reddit this way: real browser profiles, residential proxies, and logged-in accounts for anything beyond the public listing. The hard part is keeping those accounts alive for weeks, which comes down to a consistent fingerprint and persistent profiles, and that works well on-premise.
This matches what we see at Kameleo. Many of our users already run Reddit this way: real browser profiles, residential proxies, and logged-in accounts for anything beyond the public listing. The hard part is keeping those accounts alive for weeks, which comes down to a consistent fingerprint and persistent profiles, and that works well on-premise.