Privacy, discovery, subsetting and masking: the concepts

Production is full of real people's private details, which is why teams cannot simply copy it into testing (the why-TDM chapter). Before any tool can protect data, someone has to say what "sensitive" means, which laws care about it, and how a tool can recognise it among thousands of columns.

This is the vocabulary layer of TDM. Discovery, subsetting, masking and all three tools use the words defined here: PII, PHI, PCI, policy, rule, data domain, classification.

2.1 Words to know first

Sensitive data

Any data that could hurt a person or a company if it leaked or was misused: identity details, health records, payment details, passwords, trade secrets. "Sensitive" is a business and legal judgement, not just a technical one.

PII (Personally Identifiable Information)

Data that identifies a person, alone or combined with other data. Examples: name, email, phone number, a government ID number, a home address.

PHI (Protected Health Information)

Health-related data that can be tied to a person: diagnoses, treatments, medical record numbers, insurance member IDs. Informatica describes its PHI pack as aimed at the healthcare and pharmaceutical industries (User Guide 10.5.10, p.79).

PCI (Payment Card Industry) data

Payment card details such as the card number and expiry date. Informatica describes its PCI pack as aimed at banking and finance (User Guide 10.5.10, p.80).

Identifier

A piece of data that points to one person. A direct identifier does it on its own (an SSN). An indirect identifier (also called a quasi-identifier) only does it when combined with others (zip code plus birth date).

Re-identification

Working out who a "nameless" record belongs to by combining it with other information.

Data classification

Sorting data into sensitivity levels (for example public, internal, confidential, restricted) so each level gets the right protection.

Data domain

A named kind of data, such as "Social Security number" or "First name", described by how the data looks and what its column is usually called. Both Informatica and Delphix use this word, with slightly different meanings.

Rule

In Informatica, the logic that changes a value: a masking rule uses a masking technique (such as substitution) with chosen settings. Rules are covered in the masking concepts chapter.

Policy

In TDM, a named bundle that says which kinds of data are sensitive and how to protect each kind. Section 2.8 shows the exact Informatica structure.

Data steward

The person who knows a data set's meaning and quality and answers questions such as "what is in this column and who may see it?".

Compliance officer

The person responsible for making sure the organisation follows laws and internal rules about data.

2.2 What is sensitive data?

Start with the ShopSphere database from the data basics chapter. Three example rows from CUSTOMERS:

Ask one question of each column: if this leaked, could someone be harmed or could ShopSphere be in trouble?

• customer_id is just a number the system made up. On its own it harms nobody. But it links to orders, so it still needs careful handling (more in the masking concepts chapter).
• first_name and last_name tell you who the person is.
• email and phone let a stranger contact, spam, phish or impersonate the person.
• ssn (a government ID number) is the classic example: with a name and a birth date it is enough to open credit in someone's name.
• date_of_birth is used by banks and call centres to "prove" who you are.
• zip_code looks harmless. Section 2.4 shows why it is not.

Delphix's own introduction makes the same point from the tool side: organisations face data breaches and pressure from regulators, and encryption helps protect data on disks and in transit but is, in their words, ill-suited for protecting non-production environments used for development, testing and reporting. Their answer is to replace confidential values with fictitious but realistic ones. (Delphix "Introduction to Delphix Masking", source page ↗).

2.3 The three families: PII, PHI and PCI

People in TDM talk in three abbreviations. They overlap, and a single column can belong to more than one family, but each one points at a different law or standard and a different industry.

2.3.1 PII: personally identifiable information

PII is data that identifies a person. The definition differs a little between countries and laws, but the everyday list is stable:

• Names, email addresses, phone numbers, postal addresses.
• Government-issued numbers: SSN (US), passport number, driver's licence number, national ID. India's Aadhaar and PAN are comparable examples.
• Date of birth, age, gender, place of birth.
• Online identifiers such as IP addresses.

ShopSphere's CUSTOMERS table is almost pure PII: first_name, last_name, email, phone, ssn, date_of_birth, zip_code and tax_id.

Informatica ships a ready-made PII policy pack. It lists 26 data domains, each paired with a default masking rule (User Guide 10.5.10, pp.78 to 79). Reading the list is a quick way to see what a vendor treats as PII:

Notice that the pack includes things you might not call "identity", such as Salary and Grades. A vendor's list is a starting point. Your organisation decides what else counts.

2.3.2 PHI: protected health information

PHI is health data that can be linked to a person. The health data alone ("type 2 diabetes") is not what makes it PHI. The link to an identifiable individual does. A list of diagnoses with no names or IDs and no way to trace back is a different thing from a patient record.

ShopSphere is a retailer, so it has no medical records. To make the idea concrete, imagine a hypothetical ShopSphere feature, "ShopSphere Pharmacy", with a table PRESCRIPTIONS(rx_id, customer_id, diagnosis_code, drug_name, insurer_member_id). Then:

Here diagnosis_code and drug_name are health facts, insurer_member_id is an identifier, and customer_id links them to Asha or Meera. Together they are PHI. The same pack idea exists in Informatica: the PHI policy pack lists 17 domains, among them Medical Record Number, Health Plan Beneficiary Number, Device Identifier Serial Number, Certificate License Number, Account Number, Birth Date, Email Address, IP Address, Web URL, Social Security Number and ZIP (User Guide 10.5.10, p.79).

Delphix organises its detection logic into profile sets. Besides the recommended ASDD Standard set, it ships legacy sets named "Financial - Legacy" and "HIPAA - Legacy" (Delphix "Out of the box profiling settings"). Delphix's standard set also has domains with explicitly medical names, such as MEDICAL_CODES, MEDICAL_DRUG and BLOOD_TYPE, and a domain called HIPAA_DATE (Delphix "ASDD standard profile set" listing). The words "HIPAA Date" hint at a rule HIPAA has about dates, which section 2.5 explains.

2.3.3 PCI: payment card data

PCI refers to the Payment Card Industry. The PCI DSS (Data Security Standard) is a rulebook created by the card brands for anyone who stores, processes or sends card data. It is not a government law. It is a contractual standard: merchants who ignore it can lose the right to take card payments, and can be fined by their payment partners. In TDM talk, "PCI data" means card number, card expiry date, security code, cardholder name.

ShopSphere's shared schema has no card column, but any real online shop would have one (or would store a token from a payment provider). If it did, it would look like 4539 1596 8210 2773. This is the number Informatica's own documentation uses as a masking example (User Guide 10.5.10, p.158).

The Informatica PCI policy pack is short: Account Number, Birth Date, Credit Card, Expiration Date, First Name and Last Name (User Guide 10.5.10, p.80).

2.4 Direct and indirect identifiers: why "just remove the names" fails

A common beginner idea is: "We will delete the name column and then the data is anonymous." It is not, and the reason is the most important privacy idea here.

2.4.1 Two kinds of identifier

Research on public records, most famously by the computer scientist Latanya Sweeney, has found that a large share of people in the United States can be singled out from just three facts: zip code, full date of birth and gender. The exact percentage depends on the data set and the year, so no figure is quoted here. Remember the shape of the result: three boring columns, one unique person.

2.4.2 A worked example with ShopSphere

Imagine a careless tester shares an "anonymised" extract with a contractor. Names, emails, phones and SSNs were deleted. What remains:

Now suppose the contractor also has an outside list: a public club membership page, a leaked mailing list, a social media profile that mentions "born 12 April 1990" and a city. Matching the outside list against the extract on birth date plus zip code finds exactly one row. The contractor now knows that Asha spent 41,250 at ShopSphere. No name was ever in the extract.

1. Take the extract.

   Birth date and zip code are in it.

2. Take any outside list that has the same two facts plus a name.

   The linking columns do not have to be sensitive on their own.

   Result: the outside list's "Asha Menon" matches one row of the extract.

3. Read everything else in the matched row.

   Everything in the row now belongs to a named person.

2.5 Privacy laws and standards in plain words

Why should a tester care about laws? Because when a court or regulator looks at a data leak, they do not ask "was it production or test?". They ask "was personal data properly protected?". A copy in a test database is still personal data.

2.5.1 GDPR in plain words

The GDPR (General Data Protection Regulation) is a European Union law that has applied since May 2018.

• What it protects: "personal data", meaning any information about an identifiable person. That is broader than a name and number: online identifiers and location data can count.
• Who it applies to: organisations that handle the personal data of people in the EU, even if the organisation sits outside Europe. A ShopSphere selling to customers in Germany would likely be in scope wherever its servers are.
• Ideas behind it: there must be a lawful reason to use the data (consent is one such reason); use it only for the purpose it was collected for; collect and keep no more than needed ("data minimisation"); keep it secure; let people see and delete their data.
• Fines at a high level: the most serious breaches can lead to fines of up to 20 million euros or 4 percent of worldwide yearly turnover, whichever is higher. Lesser breaches have a lower ceiling.

The consequence for test data. Two GDPR ideas hit TDM directly. First, purpose: a customer who gave details so ShopSphere could deliver an order did not necessarily agree to have them copied into a test lab. Second, minimisation and security: copying a whole production database to a loosely guarded test server is hard to justify when a small, masked copy would do. The law also treats truly anonymous data (nobody can be identified any more) as outside its scope, which is one reason masking and subsetting are popular. Whether a particular masked data set counts as anonymous is a question for your compliance officer.

2.5.2 HIPAA in plain words

HIPAA is a United States law from 1996 that, through its Privacy and Security Rules, protects health information.

• What it protects: PHI, health information that identifies a person (section 2.3).
• Who it applies to: "covered entities" (health plans, healthcare providers, healthcare clearinghouses) and their "business associates", meaning vendors who handle PHI for them. A software company testing a hospital's billing system is typically a business associate.
• Minimum necessary: people and systems should receive only the least PHI they need for their job. Testers rarely need the real thing.
• De-identification: HIPAA describes two accepted ways to de-identify health data so it is no longer treated as PHI. One removes a defined list of 18 kinds of identifiers (names, detailed locations, most date elements except the year, phone numbers, record numbers and so on). The other has a qualified expert certify that the risk of identifying anyone is very small. This may be why Delphix has a domain called HIPAA_DATE, and why dates and ZIP codes feature so heavily in PHI discussions.
• Penalties at a high level: civil penalties in tiers depending on how careless the organisation was, and criminal penalties for knowing misuse.

Both tools mention these regimes only briefly. Delphix's introduction lists HIPAA, PCI and GDPR as examples of mandates organisations must stay compliant with, and Informatica packages its PHI and PCI policy packs for healthcare and banking respectively (User Guide 10.5.10, pp.79 to 80). GenRocket's column-profiling page says that identifying PII early helps organisations meet requirements "such as HIPAA".

2.5.3 Other regimes, kept general

There are many more (sector rules for banks, national laws in dozens of countries, contract terms from clients). The practical lesson is the same everywhere: personal data in test environments is a liability, and the standard defence is to make sure the test copy no longer identifies anyone.

2.5.4 Four ideas that repeat across laws

2.6 Data classification: how sensitive is each column?

Data classification means labelling data by how much damage a leak could do, so that you can apply protection in proportion. Organisations choose their own scheme. A typical four-level scheme looks like this:

Two things are worth noticing. First, classification is about columns (and sometimes whole tables), not rows. Second, the result of classifying is a plan: each level maps to an action. A column with no label gets no action, so unlabelled sensitive data is the dangerous kind.

2.6.1 How the tools record sensitivity

• Informatica. When you create a data domain you set a sensitivity level for all its columns, chosen from levels that the Administrator defines (User Guide 10.5.10, p.75). Separately, each column in a project has a Sensitive flag. The guide says the flag is "for reference only" and does not affect processing, and that assigning a data domain to a column marks it sensitive automatically (p.89). A different property, Restricted, does change behaviour: the integration service does not write a restricted column's value to the target (p.89).
• Delphix. A domain ties a category of sensitive data (the terminology page's example is social security numbers) to the way it should be secured. The profiler assigns the domain to a column and, with it, the domain's default masking algorithm (Delphix "Discovering your sensitive data").

Many laws treat some data as especially sensitive: ethnic origin, political opinions, religion, trade union membership, health, sexual orientation, biometrics. The Delphix ASDD Standard profile set has domains for these (for example ETHNICITY, RELIGIOUS_ORIEN, POLITICAL_ORIEN, UNION_MEMBERSHIP_NO, BIOMETRIC and MEDICAL_CODES), and the default masking method it shows for BIOMETRIC is NullValueLookup. For the most delicate data, blanking a value is often better than faking it.

2.7 How sensitive data is found

A real production database can have hundreds of tables and tens of thousands of columns. Nobody can read them all, and column names are often cryptic (COL_17, FLD_X2). So tools look for sensitive data automatically. At concept level they all rely on a small set of signals. The full hands-on process is the subject of the discovery and subsetting concepts chapter, the Informatica discovery and profiling chapter and the Delphix profiling chapter. Here we only need the ideas.

This chain is the same in all three tools, even though each names the boxes differently. Keep it in mind: every feature in the later tool chapters is a way to fill in or control one of these boxes.

2.7.1 Three signals

A fourth signal, data type and length, is used to rule things out: an SSN cannot live in a column that holds one-character flags.

2.7.2 What Informatica does: data patterns and metadata patterns

Informatica describes a data domain using regular expressions. A regular expression (regex) is a compact text recipe for a pattern, for example "three digits, a hyphen, two digits, a hyphen, four digits". The guide separates two kinds (User Guide 10.5.10, pp.74 to 75):

• A data pattern matches the values. Its example for Social Security numbers is [0-9]{3}-[0-9]{2}-[0-9]{4}.
• A metadata pattern matches column names. Its examples for Social Security numbers are .*SSN*, .*SOCIAL* and .*SECURITY*.

A column joins the data domain if it matches any expression in the domain. When you run a data domain profile you choose to search the data only, the column name only, or both, how many rows to read (default 1,000) and a minimum conformance percent, since not every row has to match the pattern (pp.95 and 100).

Data pattern     [0-9]{3}-[0-9]{2}-[0-9]{4}     matches   123-45-6789
Metadata pattern .*SSN*                          matches   SSN, CUST_SSN, SSN_NO

2.7.3 What Delphix does: profile sets, classifiers and domains

Delphix calls the detection step profiling. A profiling job examines metadata (column names and types) and possibly the data itself, and when it decides a column is sensitive it assigns the matching domain and that domain's default masking algorithm. It only covers the tables and files in the chosen rule set (Delphix "Discovering your sensitive data", source page ↗).

The logic is packaged in classifiers, grouped into profile sets. Four classifier frameworks exist, and they map neatly to our signals:

The recommended built-in profile set, the "ASDD Standard" set, includes list classifiers that can detect domains such as FIRST_NAME and LAST_NAME even when column names are meaningless, though data-level detection is limited to English values (Delphix "Out of the box profiling settings").

2.7.4 One real example: how Delphix recognises an SSN

The ASDD Standard listing (Delphix "ASDD standard profile set", source page ↗) shows the SSN domain is defined by three classifiers. In our words:

1. A PATH classifier.

   A regex on the column path catches names like ssn or social security number with variants such as social_security_no. Its match strength is listed as 0.67.

2. A REGEX classifier on the data.

   Patterns for the nine-digit format. The dashed or spaced form (123-45-6789) is listed with match strength 0.7, while nine bare digits are listed with only 0.1.

   The lower number for bare digits fits the idea in the weaknesses table above: nine digits in a row are far more likely to be something other than an SSN.

3. A TYPE classifier.

   The column must be a Number or String of at least 9 characters. This rejects tiny columns.

Not documented: how the strengths are combined into a final decision. Read the product page for your version before relying on the numbers. The design is what matters: several weak clues, each with a weight, are combined to decide.

2.7.5 What GenRocket does: column names only

GenRocket's "Data Column Profiling" scans column headers, not the data. It checks whether each column name contains a keyword from a list: 102 built-in names plus any your organisation adds (for example user_email matches email, client_ssn matches ssn). It then shows you the matches to choose from (GenRocket "PII Discovery with Data Column Profiling", source page ↗). It is the "column name" signal on its own. The upside is that the tool never needs to read real values.

You will read many patterns in policies, so here is the minimum: [0-9] means one digit, {n} repeats the previous item n times, ? makes it optional, * means zero or more, and a backslash makes a special character literal (User Guide 10.5.10, pp.74 to 75). A data pattern says what a value looks like, not what it is: [0-9]{10} matches a phone number and also a ten-digit order reference. That is the source of false alarms (false positives); a real SSN column stored without hyphens that fails the pattern is a false negative. Dialects differ: Delphix patterns start with (?i) for case-insensitive matching.

2.7.6 ShopSphere example: what each signal finds

2.8 Policies, rules and data domains

Now the vocabulary promised at the start. A TDM tool needs one place that records the answer to "what is sensitive here, and how do we protect each kind?". Informatica calls that place a policy. The structure is worth learning exactly, because it explains many menus you will see later. (User Guide 10.5.10, p.71)

2.8.1 What each piece is

Policy (Informatica)

"A data masking component that describes the methods to maintain the privacy of specific types of source data". It contains data domains. You assign a policy to a data source in a project, and one policy can serve several projects (p.71).

Data domain (Informatica)

Describes the functional meaning of a column based on its data or its name. It carries data patterns, metadata patterns, a sensitivity level, a status (enabled or disabled) and, optionally, preferred masking and generation rules (pp.73 to 75).

Rule (Informatica)

"A logic to mask sensitive data or to generate test data": a technique plus its parameters. Rules exist for string, numeric and date data (p.72).

Preferred rule and default rule

When you attach several rules to a domain they are "preferred rules", and one is the default, applied to all columns in the domain unless you pick another. The guide's example: a Last_Name domain with a shuffle rule as default and a substitution rule as another preferred rule (p.73).

Policy pack

A ready-made policy shipped with TDM. There are three: PII, PHI and PCI, visible after installation (p.77).

The order of work does not matter. The guide says you can create policies, rules and domains in any order, then connect them: create masking rules, define domains, assign rules to domains, create a policy, assign domains to it (p.72).

2.8.2 The same ideas in Delphix words

Delphix splits the same job differently. Use this table to translate when you move between chapters (Delphix "Discovering your sensitive data", "Managing domains" and "Delphix masking terminology").

Two behaviours from the Delphix "Managing domains" page are worth knowing: each domain has a single default masking algorithm, so you need a separate domain for each distinct algorithm you want the profiler to assign, and changing a domain's default algorithm only affects future profiling runs, not assignments that already exist in rule sets.

Both tools use the domain idea for a consistency reason as well. Delphix says the domain mechanism helps ensure the same masking algorithm is applied consistently across rule sets whenever a particular type of sensitive data is discovered. Informatica says all columns in a domain are masked with the same rule, so "SSN" and "SOCIAL_SECURITY" in different tables end up masked the same way (p.73). Consistency across tables is a major theme of the masking concepts chapter.

2.9 Who owns what?

Tools do not decide what is sensitive. People do. Titles differ between companies, but these roles appear in most TDM programmes:

The tools reflect the split of duties. In Informatica, you need the View Policies and Manage Policies privileges to view and work with policies (User Guide 10.5.10, p.77), and the Administrator defines the list of sensitivity levels (p.75). Delphix has roles too: an "Editing a user-defined algorithm" permission is described on its algorithms page, and a separate best-practice page covers defining masking roles.

2.10 Walkthrough: from a legal requirement to a policy

Here is the whole chain in one story, so the pieces above fit together. This is the shape of a real engagement, simplified.

1. Start with the requirement, in business words.

   ShopSphere's compliance officer says: "No real customer identity may leave production, in India or Germany. Card data never leaves the payment zone." The reason to write it in plain words is that everything later is traced back to this sentence.

   A one-page statement of scope: which systems, which data subjects, which laws.

2. Find candidate sensitive columns.

   Run discovery (the discovery and subsetting concepts chapter, then the tool chapters). The tool proposes columns and domains from names, patterns and lists.

   A candidate list of columns with a proposed domain and a match score or conformance percent.

3. Review with the data steward.

   Remove false alarms, add misses, resolve doubtful ones such as zip_code and tax_id. Informatica's flow lets you approve each suggested domain (User Guide 10.5.10, p.105).

   An approved list, with a short reason for each yes and no.

4. Classify.

   Give each approved column a level (section 2.6) and decide the action for each level: mask, null out, or exclude.

   A table like the lab answer below, signed off by the compliance officer.

5. Choose a protection for each domain.

   Pick techniques that keep data usable for tests (see the masking concepts chapter). In Informatica, attach them as preferred rules and mark one default per domain; in Delphix, choose the default algorithm for each domain.

   Each domain has one default protection and a written reason.

6. Bundle into a policy and store it.

   In Informatica you add the domains to a named policy and assign the policy to a project's data source (p.71). You can export policies, domains and rules to XML files and import them into another TDM repository, so a tested policy can travel from a test setup to the real one (p.80). Importing fails if the object already has assignments, so clear them or rename the file first (pp.80 to 81).

   A versioned policy and a copy of its exported files.

7. Prove it works.

   Run on a small sample, look at the results, and have a second person check. The masking concepts chapter lists what to check.

   A short report: columns covered, rule used, sample before and after.

8. Repeat on change.

   New columns appear whenever the application changes. Delphix notes that a profiling job only covers tables and files already in the rule set, and that newly reachable objects must be added by hand (Delphix "Discovering your sensitive data"). Re-run discovery on every release.

   A standing step in the release checklist.

2.11 Where test copies leak

Protecting the test database is not enough. Real data also escapes through developer laptops ("quick samples"), bug tickets and screenshots, application logs, backups taken before masking finished, files sent to vendors, free-text notes and the TDM tool's own staging area. A good policy states where unmasked data may exist, for how long and who can reach it. GenRocket's G-Subset documentation shows one design answer: data stays behind the customer's firewall, sensitive columns are never read, and temporary dump files are removed after the run (GenRocket "G-Subset Overview").

2.12 A judgement call: free text

Policies cover the common situations. The interesting work is in the edges.

2.13 Common mistakes and gotchas

2.14 Recap

SELECT o.order_id, o.customer_id
FROM   ORDERS o
LEFT JOIN CUSTOMERS c ON c.customer_id = o.customer_id
WHERE  c.customer_id IS NULL;      -- any row returned is an orphan order

-- duplicates in a column that must be unique
SELECT ssn, COUNT(*) FROM CUSTOMERS GROUP BY ssn HAVING COUNT(*) > 1;

-- any order without a customer (orphan)
SELECT o.order_id FROM ORDERS o
LEFT JOIN CUSTOMERS c ON c.customer_id = o.customer_id
WHERE c.customer_id IS NULL;

-- any SSN-shaped text left in a notes column (pattern syntax varies by database)
SELECT order_id FROM ORDER_NOTES WHERE notes LIKE '%___-__-____%';