Web scraping is often framed only as a technical challenge: Retrieve the records you need, parse them, store the results, and your job is done. But when those records relate to identifiable people, the scraper is also processing personal data—and the fact that the source is publicly accessible does not, by itself, make that processing lawful.
This guide takes a practical, engineering-first approach to GDPR-aware scraping. It follows personal data through detection, transformation, storage, and data-subject rights, with particular attention to the links that can survive hashing or redaction. The central principle is simple: A transformation is not a compliance decision. You need to assess the whole record, control the routes that could identify someone, and make sure those decisions shape what your pipeline actually does.
NOTE: Every snippet presented in this article is an illustrative extract: Just the minimum to show the mechanism and the idea. Complete implementations are in the companion GitHub repository. Where a snippet ends, the repo picks up.
Give your AI a web data layer – Decodo’s Web Data API turns any site into clean, structured data your models can actually use.
What Makes Your Scrape a GDPR Problem
Web scraping often starts with a technical question: can you extract the data you need from a website? However, once those records contain information about identifiable people, you also need to consider whether you can lawfully collect and use it.
That applies even when the website is publicly accessible, and your scraper never encounters a login page. Let’s see why.
Identifiability, Not Visibility
Under Article 4(1) of the GDPR, personal data includes information relating to a person who can be identified directly or indirectly. In practice, that means your scraper doesn’t need to collect someone’s full name for a record to contain personal data. Some examples worth considering are:
Owner names: Identify the individuals behind business listings.
Direct email addresses: Can identify a particular person, including when used for professional purposes.
Directory IDs: Can indirectly identify someone when they resolve to a personal profile or can be matched against another dataset.
As the EDPB explains, an individual’s client number or booking reference can be personal data even though it doesn’t directly display their identity. The same reasoning applies to a directory identifier linked to an individual.
Now, public availability is a different question. It tells you where the information came from, not whether your intended processing has a lawful basis under Article 6. Being able to read someone’s email address on a website doesn’t automatically authorize collecting it for your own database.
Keep in mind that collection itself is processing under Article 4(2). So, every scrape run must remain covered by an appropriate lawful basis and a specified purpose, with documentation supporting your compliance. That doesn’t mean choosing a new basis every time the script runs. An existing assessment may cover recurring collection, provided the purpose, scope, and relevant circumstances remain within it.
And remember that lawful processing is one part of responsible scraping. For a broader overview, read our article on ethical web scraping best practices. Also, whether a site’s robots.txt instructions affect your scraping is a related but separate question from whether you have a GDPR lawful basis.
Datafuel is The Fuel for Web Scraping & AI Agents, with proxies and scraping APIs.
Jurisdiction Travels with the Data
Running your scraper on a server outside the EU doesn’t automatically put the processing outside the GDPR.
Under Article 3(1), the GDPR applies when processing takes place in the context of the activities of an EU establishment, regardless of where the processing actually happens. For that processing, the people in your records don’t need to live in the EU.
For organizations without an EU establishment, Article 3(2) provides two other routes into scope:
Offering goods or services to people who are in the EU.
Monitoring their behavior, insofar as that behavior takes place within the EU.
The distinction matters. Simply collecting a record about someone in the EU doesn’t automatically satisfy either condition.
Similarly, building a database or profile isn’t automatically monitoring. The purpose of the collection and subsequent use matters. The Greek authority’s Clearview decision, citing the EDPB’s territorial-scope guidelines, explains that not every online collection or analysis of personal data qualifies as monitoring.
Also, when Article 3(2) applies, Article 27 generally requires a written appointment of an EU representative, subject to its exceptions. So, this isn’t just a theoretical obligation. However, Italian and Greek authorities found that Clearview had failed to appoint one. Meanwhile, the Dutch authority subsequently found the same infringement, but did not impose another fine for that violation because the Italian and Greek authorities had already done so.
Business Names Are Indirect Identifiers You Leak
Suppose you hash an owner’s email address but leave their business name untouched. If that name identifies the owner or leads back to their directory profile, you’ve preserved a route to the person.
This is particularly relevant for sole traders and freelancers whose trading names include their own names. A fictional listing such as “Jane Smith — Freelance Translator” doesn’t stop identifying Jane Smith because your database stores it in a business-name field.
That’s why Recital 14 excludes information concerning legal persons as such from the GDPR’s protection. However, that is not a blanket exclusion for anything labeled “business data.” You still need to assess whether the information relates to an identifiable individual.
For this reason, your detection pass should also inspect extracted business names before they reach storage. Here’s a deliberately conservative example using fictional data:
SOLE_TRADER_HINTS = re.compile(
r"\b(?:sole trader|freelance|freelancer)\b|\btrading as\b|\bt/a\b",
re.IGNORECASE,
)
def detect_sole_trader(business_name: str) -> list[str]:
flags = []
if SOLE_TRADER_HINTS.search(business_name):
flags.append("possible_sole_trader")
return flags
# "Jane Smith — Freelance Translator" → ["possible_sole_trader"]
# "Alex Morgan trading as Riverside Repairs" → ["possible_sole_trader"]
# "Example Manufacturing Ltd" → []
# No match is not clearance. Apply person-name detection next.This snippet catches explicit wording that deserves further inspection. But it doesn’t determine someone’s legal status, and it will miss names without those phrases. In practice, you’ll want to combine these rules with person-name detection and checks against the rest of the record.
The important point is that hashing one field doesn’t remove the identifying information left in the others. Your detection needs to consider the record as a whole, not just the fields that look obviously personal.
Reliable access is the foundation of any scraping project. IPRoyal offers 64M+ residential IPs to help you reach any website without blocks or bans.
Pseudonymized Is Not Anonymized: The Error That Breaks Your Storage Design
Hashing an email address changes what your scraper stores. However, it doesn’t necessarily change whether that record relates to an identifiable person.
If you can match the hash against a list of known addresses, follow a retained directory ID, or search for the business name you left untouched, you still have a way back to the individual. So, calling the destination an “anonymized” dataset doesn’t remove those connections.
This matters because your storage design usually determines what happens next, who can access the records, how long you retain them, and which downstream systems receive them.
The EDPB Three-Criteria Test
The EDPB’s Guidelines 02/2026 on Anonymisation, adopted on 7 July 2026 for public consultation, provide a practical framework for assessing anonymity. The version referenced here is the consultation draft, not final guidance. The framework uses three criteria:
No record isolation: The data doesn’t contain a unique combination of attributes relating to a single individual.
No linkage: A person’s record cannot be linked to a record in another dataset relating, with certainty or high likelihood, to the same individual.
No inference: The data doesn’t support specific and meaningful inferences about an identifiable individual, as defined in the guidelines.
Now, here’s an important qualification: These aren’t three independent switches where one failure automatically proves that the dataset is personal. In other words:
Passing all three supports a conclusion of anonymity.
Failing one means further analysis is needed, including consideration of the relevant entity’s capabilities and the means reasonably likely to be used.
Now, for a scraper, linkage is often the most immediate problem…
Suppose you remove the owner’s name and hash their email address, but retain a trading name that resolves to their public directory listing. If that listing identifies the same individual, the business name provides a working connection between your stored record and the source. That means you haven’t satisfied No Linkage simply because the email column contains a hash.
The same applies to a documented re-fetch route. If your system deliberately retains enough information to retrieve the identifying source record again, that route belongs in your anonymity assessment. It isn’t merely an accidental possibility you can ignore: Your design already anticipates using it.
This means that the EDPB’s framework considers both intentional and unintentional identification techniques. For this reason, review not just the values you store, but also the lookup mechanisms your application preserves.
Low-Entropy Fields Make Hashing Enumerable
Cryptographic hashes are designed to resist reversal. However, an attacker doesn’t always need to reverse anything. Instead, they can take a candidate value, hash it using the same procedure, and compare the result with the stored digest. Repeat that over a relevant list of addresses, and a match reveals which candidate produced the hash.
In this context, consider that email addresses don’t occupy a universally small input space. The practical issue is that your relevant candidate space may be small and readily available. A directory’s published contact list, for example, may already contain the address you hashed. Phone numbers also have structured formats and can become easier to enumerate when the country, area, or likely number range is known.
This means that adding a salt prevents straightforward reuse of precomputed hash tables, but it doesn’t prevent candidate testing when the salt is available. Here’s a small demonstration using fictional addresses:
def hash_email(email: str, salt: bytes) -> bytes:
return hashlib.sha256(salt + email.encode("utf-8")).digest()
salt = secrets.token_bytes(16)
stored_digest = hash_email("jane@example.com", salt)
for candidate in ["alex@example.com", "jane@example.com", "sam@example.com"]:
if hmac.compare_digest(hash_email(candidate, salt), stored_digest):
print(f"Recovered: {candidate}") # Recovered: jane@example.com
break
# Elapsed: 0.00009 seconds. No SHA-256 reversal required.The script measures the recovery time when you run it rather than assuming a benchmark. Notice that the code never reverses SHA-256: It checks which known input produces the stored output. A unique salt per record makes each record require separate work, but it doesn’t eliminate this approach.
Under Article 4(5) and Recital 26 of the GDPR, the relevant issue is whether information can be attributed to an individual using additional information and means reasonably likely to be used. But cryptographic irreversibility alone doesn’t answer that question.
Also, hashing one field doesn’t necessarily establish compliant pseudonymization. If the rest of the record still directly identifies the person, the transformation hasn’t achieved the separation described in Article 4(5).
Rename the Sinks You Write To
Once you recognize these connections, your storage destinations should reflect them. A practical naming scheme is:
non-personal: Records assessed as not relating to an identifiable individual in the relevant context.pseudonymized: Personal records whose attribution requires separately held additional information, with appropriate safeguards.quarantine: Uncertain records, failed transformations, or records that still contain identifying information requiring further handling.
Consider that these are engineering categories, not GDPR-defined storage destinations. In particular, quarantine is a restricted processing area.
Here’s an illustrative routing function. Note that its arguments represent assessment results supplied by your pipeline, not fields assumed to exist in your source data:
CONFIDENCE_FLOOR = 0.95 # illustrative policy threshold, not a GDPR requirement
def route_record(
*, confidence: float, non_personal_assessment_passed: bool,
pseudonymization_assessment_passed: bool, unresolved_identifiers: bool,
) -> Sink:
if not isfinite(confidence) or not 0.0 <= confidence <= 1.0:
return "quarantine"
if confidence < CONFIDENCE_FLOOR or unresolved_identifiers:
return "quarantine"
if non_personal_assessment_passed:
return "non-personal"
if pseudonymization_assessment_passed:
return "pseudonymized"
return "quarantine"Keep in mind that a confidence score doesn’t establish anonymity. The assessments behind these flags need to evaluate the whole record, including public-directory linkage, retained lookup information, and accessible re-fetch routes. For pseudonymization, they also need to verify the separation and protection of additional identifying information.
Finally, use the same destination names in your retention schedule, access-control matrix, and reviewer documentation. Where you maintain an Article 30 record of processing activities, map the relevant personal-data flows to those destinations as well.
Ultimately, the name should describe the record’s assessed status, not the transformation you hoped would make it anonymous.
DRR in Python
Detecting personal data is only useful if the result changes what your scraper does with it. To do so, you need to identify the relevant spans, remove or replace them, and route the resulting record to the appropriate destination before raw values reach storage or logs.
Here, we’ll build that detection, redaction, and routing flow in Python. The detection runs locally, with custom recognizers for identifiers that a general-purpose model doesn’t understand.
Detection That Never Leaves Your Infrastructure
Presidio Analyzer combines pattern matching, named entity recognition, and other detection logic. Paired with a locally installed spaCy model, it can analyze scraped text without sending that text to an external inference service.
Both Presidio Analyzer and the [en_core_web_lg](https://huggingface.co/spacy/en_core_web_lg) model used here are MIT-licensed. However, the license isn’t what keeps your data local: Your deployment does. So, install the dependencies and model beforehand, avoid remote recognizers, and restrict outbound network access in the processing environment.
Start by installing the packages in an activated virtual environment:
pip install presidio-analyzer
python -m spacy download en_core_web_lgNext, build the analyzer once and reuse it:
def build_analyzer() -> AnalyzerEngine:
provider = NlpEngineProvider(nlp_configuration={
"nlp_engine_name": "spacy",
"models": [{"lang_code": "en", "model_name": "en_core_web_lg"}],
})
return AnalyzerEngine(nlp_engine=provider.create_engine(), supported_languages=["en"])
def detect(analyzer, text: str, entities: list[str]) -> list[RecognizerResult]:
return sorted(
analyzer.analyze(text=text, language="en", entities=entities, score_threshold=0.0),
key=lambda r: (r.start, r.end),
)
# entity=PERSON span='Jane Smith' offset=[11, 21) score=0.85
# entity=PERSON span='Email' offset=[23, 28) score=0.85
# entity=EMAIL_ADDRESS span='jane@example.com' offset=[29, 45) score=1.00Each result includes the entity type, matched text, character offsets, and recognition score. The end offset is exclusive, so the detected span is exactly text[start:end].
Pattern-based recognizers work well for structured values such as email addresses and identifiers. Named entity recognition adds contextual detection for names that would be difficult to capture reliably with regex alone. Presidio’s recognizer-development documentation explains how these approaches complement each other.
Keep in mind that a recognition score isn’t necessarily a calibrated probability of correctness. A score of 0.85 doesn’t automatically mean “85% likely to be personal data.” You need to evaluate detection on representative examples from your sources.
Also, printing matched spans is useful when testing with fictional fixtures. Doing the same with live records would create another copy of the personal data you’re trying to control.
Custom Recognizers and the Entity-Scope Decision
A general-purpose analyzer won’t know that your directory uses a particular membership-ID format. For that, you can register a PatternRecognizer:
def register_site_recognizers(analyzer: AnalyzerEngine) -> None:
analyzer.registry.add_recognizer(
PatternRecognizer(
name="ExampleMembershipRecognizer",
supported_entity="MEMBER_ID",
supported_language="en",
patterns=[Pattern(name="example_member_id", regex=r"\bMEM-\d{6}\b", score=0.98)],
)
)
SITE_ENTITIES = [*BASE_ENTITIES, "MEMBER_ID"]
# 0.98 is an illustrative starting value, not a measured accuracy claim.
# Document the format, positive examples, and observed errors before
# using any score operationally.The assigned scores are illustrative starting values, not measured accuracy claims. Before using them operationally, document the source format, representative positive and negative examples, observed errors, and why the chosen score is appropriate.
Now, entity scope matters too. But detecting every supported entity can return technically accurate findings that don’t help your decision. For example, ordinary dates in a product description. So, request the entity types your processing assessment identifies as relevant. However, don’t confuse “we don’t need this field” with “we don’t need to detect it”. If an unnecessary identifier can enter your extracted text, you may need detection precisely so you can remove it.
To redact findings, first merge overlapping spans. Otherwise, overlapping recognizers can produce incorrect replacements:
def redact(text: str, findings: list[RecognizerResult]) -> str:
merged: list[tuple[int, int]] = []
for result in sorted(findings, key=lambda item: item.start):
if merged and result.start <= merged\[-1\][1]:
merged\[-1] = (merged[-1\][0], max(merged\[-1\][1], result.end))
else:
merged.append((result.start, result.end))
for start, end in reversed(merged):
text = text[:start] + "[REDACTED]" + text[end:]
return text
You can then connect detection to the route_record() function from the preceding section. This demonstration keeps the record in quarantine until both detection coverage and the separate pseudonymization assessment have been approved:
def inspect_and_route(
text: str, *, coverage_approved: bool, pseudonymization_approved: bool
) -> tuple[str, str]:
findings = detect(analyzer, text, SITE_ENTITIES)
transformed = redact(text, findings)
destination = route_record(
confidence=1.0 if coverage_approved else 0.0,
non_personal_assessment_passed=False,
pseudonymization_assessment_passed=(coverage_approved and pseudonymization_approved),
unresolved_identifiers=not coverage_approved,
)
return transformed, destinationUnder these explicitly supplied assumptions, the destination changes from quarantine to pseudonymized. Registering the recognizers alone doesn’t authorize that change.
The approval must consider the entire record, including retained business names, directory links, and any additional information used for attribution. No detection results - or no results after redaction - don’t establish that a record is non-personal.
Transient Raw Text Is Still Processing
Keeping raw text in memory is preferable to scattering copies across files, but it doesn’t put the operation outside the GDPR.
Under Article 4(2), processing includes operations such as collection, consultation, use, and alteration. Detecting and transforming personal data in memory therefore remains processing, even when it lasts only milliseconds. That processing also needs an applicable Article 6 basis.
In practice, the less obvious persistence paths deserve just as much attention:
Retries: Job queues and dead-letter records may serialize the original payload.
Exceptions: Error messages can include input values, URLs, or response bodies.
HTTP logging: Client instrumentation may record identifying query parameters, headers, or bodies.
Error monitoring: Debugging integrations may capture local variables and request context.
For this reason, avoid logging exception messages directly. If you inspect them for redaction, do so before logging:
@contextmanager
def transient_text(text: str):
holder = {"text": text}
del text
try:
yield holder
finally:
holder.clear() # drops references; not secure memory erasure
def log_processing_failure(exc: Exception) -> None:
# Automated detection can miss identifiers in exception messages.
# Suppress entirely rather than risk logging raw personal data.
logger.warning(
"DRR processing failed; exception text suppressed", exc_info=False
)
# Never fall back to raw exception logging on redaction failure.This handler suppresses exception text rather than trusting automated redaction to make arbitrary diagnostics safe. It also avoids attaching a traceback to the log event.
The context manager has a narrower purpose: It drops the references held in its container when the block exits. However, it does not securely erase a Python string from memory. That’s because strings are immutable, other references may exist, and HTTP buffers, tracebacks, or NLP objects may retain copies.
Furthermore, you still need deployment controls for crash dumps, swap, request tracing, and monitoring integrations. Similarly, returning redacted text doesn’t approve it for storage: It must still pass your routing assessment.
Ultimately, local detection is one part of the boundary. The rest is making sure raw personal data cannot quietly bypass that boundary through retries, diagnostics, or an overly optimistic storage decision.
Finally, the content you collect raises a separate issue from personal-data protection: Copyright. Our article on how to avoid copyright violations while scraping discusses why collecting and retaining page content deserves its own assessment.
Operators, and the Legal Status of Each Output
An operator tells Presidio how to transform a detected span. But it doesn’t tell you whether the resulting record is anonymous.
Replacing a name, masking a phone number, and hashing an email solve different engineering problems. Their legal effect depends on what remains, what additional information you hold, and whether the record still points to an individual.
So your operator configuration needs a companion assessment: What does this transformation actually leave behind?
A Corrected Operator Table
Start with the information your purpose needs. If the useful fact is simply that a listing named a person, replace PERSON with a fixed placeholder. A stable name hash would introduce matching capability you don’t need.
For phone numbers and membership IDs, retain an area code or source prefix only when it serves a documented purpose. Masking the remaining characters can support the minimisation principle in Article 5(1), but partial masking doesn’t itself establish anonymity. Below is a table you can use as guidance:
Pseudonymized data is also personal data. The distinction in the last column describes how attribution works, not two mutually exclusive legal categories.
Here’s an OPERATORS dictionary implementing those choices for fixed-format fictional fixtures:
OPERATORS = {
"DEFAULT": OperatorConfig("redact", {}), # unmapped entities removed, not retained
"PERSON": OperatorConfig("replace", {"new_value": "[PERSON]"}),
"PHONE_NUMBER": OperatorConfig("mask", {"masking_char": "*", "chars_to_mask": 9, "from_end": True}),
"MEMBER_ID": OperatorConfig("mask", {"masking_char": "*", "chars_to_mask": 6, "from_end": True}),
# EMAIL_ADDRESS is added after the secret salt is loaded — see load_hash_salt()
}
OUTPUT_STATUS = {
"PERSON": "Placeholder only; assess the remaining record",
"PHONE_NUMBER": "Minimized; retain personal-data controls",
"MEMBER_ID": "Minimized; assess remaining linkage",
"EMAIL_ADDRESS": "Pseudonymized if separation is effective",
}Note that Presidio’s operator documentation defines chars_to_mask as a character count, not a count of digits. Spaces, brackets, and punctuation therefore matter.
Notice the DEFAULT operator too. Unmapped detected entities are removed rather than retained. It does nothing about identifiers the analyzer misses.
The status notes belong beside the configuration because they explain its limits. They should never become a shortcut that routes every successfully transformed record to non-personal.
Your Salt Is Re-Identification Material You Hold
Stable hashes are useful when you need to recognize the same address across records. That usefulness is also the linkage you must account for.
According to the current Presidio documentation, the hash operator generates a random salt per entity by default starting with version 2.2.361. Repeated values therefore produce different outputs. Supplying a consistent salt restores deterministic matching.
That default removes equality matching through the hash alone. It doesn’t remove linkage through the rest of the record, and it doesn’t prove anonymity.
A terminology distinction matters here: A salt is not inherently a secret cryptographic key. Conventional salts can be public. If you deliberately keep a shared salt secret to restrict candidate testing, however, it becomes security-sensitive re-identification material. Treat it accordingly. For a purpose-built keyed hashing design, HMAC is generally preferable to treating salted SHA-256 as though it were HMAC.
The following extract loads the secret and wires it into the OPERATORS map. Provision PII_HASH_SALT through your deployment’s secret-management system:
def load_hash_salt() -> str:
try:
salt = os.environ["PII_HASH_SALT"]
except KeyError:
raise RuntimeError("Required hashing secret is unavailable") from None
if len(salt.encode("utf-8")) < 16:
raise RuntimeError("Hashing secret fails minimum length policy")
secret_audit.info("PII hashing secret loaded") # value never logged
return salt
salt = load_hash_salt()
OPERATORS["EMAIL_ADDRESS"] = OperatorConfig(
"hash", {"hash_type": "sha256", "salt": salt}
)
# Fixed-salt outputs are deterministic: same input + same salt = same digest,
# across process restarts and pipeline runs.With the documented implementation, default (unsalted) outputs differ across calls, while fixed-salt outputs match. Across separate runs, the fixed-salt digest remains stable only if the input, salt, and hashing implementation remain unchanged.
So, if you retain the means to test candidate addresses against stored digests, document that capability. Changing the column from an email address to a hash hasn’t removed it.
Article 9 Fields Get Halted, Not Tokenized
Some findings need a stop decision before ordinary operators run. Article 9 prohibits processing specified special categories of personal data unless an applicable exception under Article 9(2) applies.
For this pipeline, the conservative policy is therefore: Halt suspected special-category records unless the processing has been separately approved. But that is an engineering policy, not a claim that Article 9 universally requires dropping every such record.
Presidio’s NRP entity covers nationality, religious groups, and political groups. So, it is a useful warning signal, not a legal classifier:
Nationality alone is not automatically an Article 9 category.
Mentioning a religious or political group doesn’t necessarily reveal an individual’s beliefs.
Health, ethnicity, and other special-category information can appear without any
NRPfinding.
To implement this, add NRP to the earlier detection scope, and supplement it with source-specific review or validated detectors where needed. The guard below accepts an additional review signal without inventing unsupported Presidio entity labels:
GUARD_ENTITIES = [*SITE_ENTITIES, "NRP"]
def guard_special_categories(
findings: list[RecognizerResult], *, source_review_flags_sensitive: bool
) -> bool:
suspected = source_review_flags_sensitive or any(r.entity_type == "NRP" for r in findings)
if suspected:
decision_log.warning("Record halted by special-category screening policy")
return False # separate metadata-only log: no span, URL, or inferred category
return True
def transform_or_halt(raw: dict[str, str], *, source_review_flags_sensitive: bool) -> str | None:
findings = detect(analyzer, raw["text"], GUARD_ENTITIES)
if not guard_special_categories(findings, source_review_flags_sensitive=source_review_flags_sensitive):
raw.clear() # discard the entire payload, not just the detected span
return None
return anonymizer.anonymize(text=raw["text"], analyzer_results=findings, operators=OPERATORS).textCall this inside the earlier transient_text() context. A None result means stop: No normal storage write, payload retry, or raw quarantine copy.
Configure scrape.sensitive_decisions with its own restricted handler and retention policy; a separate logger name alone doesn’t create separate storage. Even decision metadata can become sensitive when linked back to an identifiable record.
Also, the guard deliberately discards the whole payload rather than deleting one detected word and forwarding the rest. Removing a religious-group name, for example, may leave surrounding text that reveals the same belief.
Overall, the practical rule is simple: Operators transform values; assessments authorize processing and determine where the resulting record may go.
Rights You Have to Ship as Code
The DRR pipeline earns you a legal basis to collect. What it does not do automatically is honour the rights that kick in after collection. In other words, erasure, rectification, and objection are architectural constraints that determine whether your pseudonymisation actually holds up under audit. Build them wrong and every deletion you’ve ever logged is theatre.
Erasure and Rectification Keyed on the Pseudonym
Here’s the elegant part of a keyed-hash scheme: You never stored the raw identifier, so you can’t search by it directly. What you can do is re-derive the token from the identifier the data subject gives you, then use that token as the lookup key across every sink. Since the hash is deterministic, the token you compute at erasure time matches the token you computed at ingestion time, exactly.
Erasure takes the raw identifier, recomputes the hash, and deletes every row in every table where pseudonym = token. Then it writes a tombstone so the identifier can never be reingested. Rectification is the same lookup, but instead of deleting, it re-derives a new token from the corrected input and rewrites both the stored pseudonym and any downstream aggregates that reference it.
Here’s an example you can implement:
def _derive_token(raw_identifier: str) -> str:
return hmac.new(SALT, raw_identifier.encode(), hashlib.sha256).hexdigest()
def handle_erasure(raw_identifier: str, reason: str, db_path: str = "pipeline.db") -> dict:
token = _derive_token(raw_identifier)
now = datetime.now(timezone.utc).isoformat()
with sqlite3.connect(db_path) as conn:
for row in conn.execute("SELECT payload FROM scrape_records WHERE pseudonym = ?", (token,)):
conn.execute( # quarantine before deleting — 30-day backup window
"INSERT INTO quarantine (pseudonym, payload, quarantine_at, quarantine_expires_at) "
"VALUES (?, ?, ?, datetime('now', '+30 days'))", (token, row["payload"], now))
conn.execute("DELETE FROM scrape_records WHERE pseudonym = ?", (token,))
conn.execute("DELETE FROM aggregates WHERE pseudonym = ?", (token,))
conn.execute( # suppress — prevents re-scraping this identifier
"INSERT OR IGNORE INTO suppression_list (pseudonym, reason, added_at) VALUES (?, ?, ?)",
(token, f"Art.17 erasure: {reason}", now))
conn.execute( # audit log — token only, never the raw identifier
"INSERT INTO erasure_log (pseudonym, erased_at) VALUES (?, ?)", (token, now))
conn.commit()The quarantine table is not optional. Without it, a deletion that gets challenged in an audit leaves you with nothing but a log entry. With it, you have the payload for 30 days, the deletion timestamp, and a clear expiry: That is precisely what an Art. 17(3) exception window looks like on paper. The suppression write is baked into the erasure handler rather than being a separate step because a deletion without suppression is an erasure you will silently undo the next time your scrape cycle hits the same source.
Rectification follows the same token-derivation logic, but issues UPDATE instead of DELETE:
def handle_rectification(old_raw: str, new_raw: str, db_path: str = "pipeline.db") -> dict:
old_token, new_token = _derive_token(old_raw), _derive_token(new_raw)
if old_token == new_token:
return {"status": "noop", "reason": "tokens_identical"} # idempotent
with sqlite3.connect(db_path) as conn:
conn.execute("UPDATE scrape_records SET pseudonym = ? WHERE pseudonym = ?", (new_token, old_token))
conn.execute("UPDATE aggregates SET pseudonym = ? WHERE pseudonym = ?", (new_token, old_token))
conn.execute("UPDATE suppression_list SET pseudonym = ? WHERE pseudonym = ?", (new_token, old_token))
conn.commit()
return {"old": old_token[:8] + "…", "new": new_token[:8] + "…"}The tokens_identical early return matters: If the corrected input hashes to the same token as the old one, every stored row already carries the right pseudonym. No rewrite needed, and the response is safe to call repeatedly from a retry queue.
Objection and the Re-Scrape Loop That Defeats It
Art. 21(1) gives individuals the right to object to processing based on legitimate interests. When they do, you must stop immediately. The suppression list the erasure handler writes to is the same list your scrape loop must consult, and it must do so before the fetch, not after.
The failure mode here is subtle. You run an erasure. The identifier goes into the suppression list. Next week your scraper hits the same domain, encounters the same profile URL, and happily fetches it again. This way, you’ve recreated deleted data. And under GDPR, that’s a fresh collection event, and your legal basis evaporates.
Use the following to solve this:
def is_suppressed(raw_identifier: str, db_path: str = "pipeline.db") -> bool:
token = _derive_token(raw_identifier)
with sqlite3.connect(db_path) as conn:
return conn.execute(
"SELECT 1 FROM suppression_list WHERE pseudonym = ? LIMIT 1", (token,)
).fetchone() is not None
for url in session.urls:
identifier = extract_identifier_from_url(url)
if is_suppressed(identifier):
session.suppression_hits += 1
logger.warning("suppression_hit url=%s domain=%s — skipping fetch", url, session.source_domain)
continue # ← enforcement point: never reaches the network
content = fetch(url)
yield url, contentThe suppression_hits counter matters beyond the individual scraper. That’s because if you’re consistently hitting suppressions on a particular domain, that domain is re-publishing profiles of people who’ve already exercised rights against you.
One implementation note: The suppression check runs on the identifier you would have extracted from the page, derived before any network call. That means you need a stable identifier in the URL. If your URLs are opaque (session tokens, integer IDs with no semantic content), you’ll need a lightweight pre-fetch to resolve the identifier, then suppress-check before the full crawl. That’s slower, but it’s the correct order of operations.
The Art. 14 Notice Nobody Ever Schedules
Art. 14 applies whenever you collect personal data indirectly — meaning from a source other than the data subject themselves. Web scraping is indirect collection by definition, so here’s the obligation: Inform the data subject of your identity, your legal basis, the categories of data you’ve collected, and their rights.
Note that the exemption in Art. 14(5)(b) — “disproportionate effort” — sounds like a get-out clause, but it isn’t. The GDPR requires that you document why individual notice is disproportionate and implement alternative transparency measures (a public notice page, for example).
In practice, that means your pipeline must emit, for every domain it scrapes, structured metadata about what was collected and when. Here’s a snippet that does so:
@dataclass
class CollectionEvent:
source_domain: str
collection_timestamp: str
legal_basis: str # "Art. 6(1)(f) — legitimate interest"
purposes: list[str] # ["business contact enrichment"]
data_categories: list[str] # from DRR pipeline RecognizerResult entity types — not hand-maintained
record_count: int
pseudonymised: bool
controller_identity: str
controller_contact: str # DPO email or privacy@ address
notice_method: str = "pending" # updated by notice scheduler after dispatch
def emit_collection_event(event: CollectionEvent) -> None:
with COLLECTION_LOG_PATH.open("a", encoding="utf-8") as fh:
fh.write(json.dumps({"schema_version": "1.0", **event.__dict__}, ensure_ascii=False) + "\n")The notice_method field starts as "pending" and your notice scheduler updates it once it dispatches. That field, together with the timestamp, is what you produce during a regulator investigation to show the obligation was tracked and acted on, not ignored.
One structural point: The data_categories list feeding into CollectionEvent should come directly from your DRR pipeline’s RecognizerResult output. That closes the loop between detection and disclosure: You’re notifying on exactly the categories you processed, not a manually-maintained approximation that drifts over time.
The three rights mechanisms now connect to the same pseudonym token the pipeline produces at ingest: Erasure finds rows by token, the scrape loop suppresses by token, and the collection log feeds notices that arrive before the one-month clock runs out. This way, the pipeline isn’t simply a data-collection tool with compliance features bolted on: It’s a rights-management system that happens to also collect data.
Conclusion
A GDPR-aware scraping pipeline is more than a detector followed by a redaction step. It needs to account for the full record, including business names, directory links, retained lookup information, and the ways raw data can persist in retries, logs, or monitoring systems. Hashing and masking can reduce exposure or support pseudonymization, but neither automatically makes a record anonymous.
So, tell us: How are you managing compliance in your scrapers? Let’s discuss that in the comments!
Did you like this article? Share it with someone who might find it useful and get a discount on paid plans.





