GenRocket (1): the building blocks

The first three of the five building blocks are domains (the tables), attributes (the columns) and generators (the recipes that decide each value). Together they are the Model and part of the Design phase. The worked example describes the ShopSphere CUSTOMERS, ORDERS and ORDER_ITEMS tables to GenRocket, links them so every foreign key is valid, and chooses a generator for every column.

The output side, receivers and scenarios, is in the receivers and scenarios chapter. Patterns built with generators (boundary, sequential, random, edge cases, null) are in the data patterns chapter.

6.13 Words to know first

Template domain

A domain as defined in a project version. Think of it as the master copy; a scenario gets a copy called a scenario domain.

Scenario domain

The per-scenario copy of a template domain. Changes to the template flow down to it unless it is locked.

Domain variable

A named setting that belongs to a domain, written global.DomainName.variableName, which generator parameters can refer to. Every domain gets three: id, loopCount and seed.

loopCount

How many rows (records) a domain generates.

Seed

A starting number for random generators. The same seed gives the same "random" data on every run.

Parent, child, sibling

GenRocket's way to express relationships between domains. A child has exactly one parent; "describing" tables are marked as siblings.

Primary attribute

The attribute marked as the key (shown with a key icon). Children refer to it.

Generator parameter

A setting on a generator, such as startRange. Required ones carry an asterisk in the web platform.

Linked generators

Several generators placed in one attribute so that later ones can use the values of earlier ones.

Preview

A sample of generated values shown in the web platform so you can check a configuration before downloading anything.

DDL

Data Definition Language: the SQL text that creates tables (CREATE TABLE ...). See the data basics chapter.

6.14 Domains: the tables of your design

6.14.1 What a domain is

A domain "defines real-world objects", a noun such as a person, place or thing. When you model a database, a domain is a table and its attributes are the columns. The docs' own example has four tables, Company, Department, User and Address, each becoming a domain with its columns as attributes. At least one domain must exist in a project to generate any data.

A domain inside a project version is a template: it holds the attributes, generators and relationships. When you create a scenario, GenRocket copies the template into a scenario domain. Changes you make to the template cascade down to every scenario unless the scenario (or the scenario domain) is locked. It is like a letter template, except that you can pin a printed letter so later edits leave it alone.

The docs also state a rule: a template domain works independently and cannot really reference another domain's attributes. Relationships at the domain level are "just probable". The real references are made at the scenario level, where related domains are put together. You will see this again with ProxyReferenceGen below.

Each time a domain is created or imported, it gets three default variables, all initially 1 (details below): id, loopCount and seed.

The "New Domain" page says a domain name should contain only letters and numbers (spaces are now also allowed), so ORDER_ITEMS becomes a domain named OrderItems with ORDER_ITEMS in the Table Name field. The page does not say what happens with an underscore.

6.14.2 The Domain Dashboard

Click a domain's name and the Domain Dashboard opens. It is the main working screen; the table below explains its areas.

6.14.3 Four ways to create domains

The page "How do I import my data model into GenRocket?" counts seventeen primary methods: DDL (single or multiple), XTS, CSV, CSV for fixed file, JSON, GenRocket JSON schema, standard JSON schema, XSD, YAML, presets, Spark schema, Avro JSON schema, CopyBook, Salesforce, Scratch Pad, Quick Pattern and New Domain. The count changes between pages, so do not memorise it. All are reached from the green New Domain button in the Domains pane.

For ShopSphere, four of these matter most.

6.14.4 Creating a domain by hand

1. Select your project in the Projects pane.

   The Domains pane only appears once a project is selected.

2. Click New Domain, then choose New Domain.

   This is the single-domain method.

3. Type a name, for example Customers, and click Save.

   Use letters and numbers.

   What you should see: the Domain Dashboard for the new domain, with just one attribute called id and three domain variables (global.Customers.id, .loopCount and .seed, each 1).

You would then add attributes. That is fine for two or three columns; for a whole database use an import.

6.14.5 Importing from DDL

You can paste one table's DDL, or upload a file with many. The vendor's sample single-table DDL creates a user table with an id primary key and columns such as first_name varchar(25) not null. Choosing the database vendor from a list tells GenRocket how to read the SQL dialect.

Multi-table import:

1. Open New Domain and choose Import from DDL, then Import Multiple Domains from DDL.

   The other choice in that first dialog is single-table import.

2. Pick the vendor (for example mysql) and choose the DDL file.

   Optionally tick Enforce Type/Limit, which makes every assigned generator respect each column's declared type and size (explained below).

3. Click Save and wait.

   The file goes into a queue (you can watch the Queue Status pane). The user who started the import gets an email when it starts and when it ends.

   What you should see: new domains with their attributes, a primary key marked, and generators assigned automatically.

4. Optionally open the DDL audit from the Project Versions pane.

   The "GenRocket Table Schema (GTS) Audit Details" option shows what was imported and any errors.

6.14.6 Importing a live database schema (XTS)

"XTS" stands for Extract Table Schema. This is the documented way to pull a database's structure straight from the database. Two steps:

1. Extract the schema with the runtime.

   Run the runtime's -xts option with the path to a JDBC properties file and, optionally, the table names (genrocket -xts /path/config.properties company,department,user). With no names, every table is extracted. Options exist to exclude tables (-xt, -xf), and tables in a specific schema are written schema1:tablename. The properties file holds four required entries: driver, user, password and url.

   What you should see: an encrypted schema file, tied to your organisation. The page calls it a schema extract; it does not say more about what is read.

2. Import the file with the XTS Import Wizard.

   Open Self Serve Options then XTS Wizard (or choose Import from XTS in the New Domain menu), choose the file, optionally enable Enforce Type/Limit, click Save and wait. The project is frozen until the import finishes.

   What you should see: domains, attributes and generators created for every extracted table, and the wizard's step 1 ready to be marked completed.

Each step has its own wizard that works on many domains at once, because the setup after importing hundreds of tables is also repetitive; you tick each step as complete so that the next unlocks.

6.14.7 A live Scratch Pad demo from the 2020 webinar

In the 2020 webinar, GenRocket's CTO builds a domain on screen in a few minutes, which shows what the Scratch Pad method feels like. He creates a project called "live demo", opens the domain page and chooses Scratch Pad. He types attributes that nearly every application has (first name, last name, middle initial, password, username), then adds social security number, card number and phone number. GenRocket looks each name up in what he calls its data warehouse, a store of the generators most used for common attribute names, and attaches a likely generator to each. It even matched "middle" after he misspelled it. A preview of sample rows appears at once.

The docs describe the same behaviour: Create with Generators assigns generators, guessing from the attribute name, domain name, industry type and what other users chose (section 6.18.1). They say nothing about tolerating misspellings, so type names carefully. They also say an attribute with "password" in its name is given the EncryptGen generator automatically.

Two details from the same session. He says there were nine ways to set up a project in 2020; the docs now count seventeen (section 6.14.3). And in the Q&A he describes the schema extract (XTS, section 6.14.6) as reading metadata only (table and attribute names, types, foreign keys), after which a wizard lets you mark each domain's one parent and the rest become siblings. The docs match. His claim of 10,000 tables of 100 attributes loaded in about eight minutes is a 2020 vendor figure that the docs do not repeat.

6.15 Relationships: parent, child and sibling

Real tables point at each other: ORDERS.customer_id is a foreign key to CUSTOMERS (data basics chapter). If you generate 50 customers and 80 orders with random customer numbers, some orders will point at customers that do not exist, which breaks referential integrity. GenRocket's answer is to generate data in order, from parent to child, so a child can only pick a key a parent has already made.

The documentation says GenRocket "generates synthetic data with full referential integrity" because every domain has an id or primary attribute, and children refer to the parent's primary attribute. It also says GenRocket looks at relationships differently from a database. A database table can have several foreign keys. A GenRocket domain may have only one parent but many siblings, because generating sequentially along several parents would produce "a large amount of unnecessary data".

The scenario page adds a vocabulary pair: the primary domain is the one a scenario is created for, and the root domain is the first parent up the chain. For an ORDER_ITEMS scenario, ORDER_ITEMS is primary and CUSTOMERS is root, so the scenario creates customers, then orders, then items, and every key matches.

6.15.1 Setting a parent or sibling by hand

1. Open the child domain's dashboard and click the Relationship tab.

   The relationship is stored on the child.

2. Click Add Referential Parent and tick the parent's radio button, then Save.

   Radio buttons enforce the one-parent rule.

3. For a lookup table, click Add Referential Sibling, tick one or more domains, and add them to the scenarios you want.

   What you should see: the Relationship tab draws a small diagram of the parent, the child and its siblings. If scenarios already exist, you are asked which scenarios should receive the new relationship.

6.15.2 The Domain Referencing Wizard

Setting parents one by one is tedious after a big import. The wizard shows every domain with parent radio buttons, sibling checkboxes and a diagram that updates as you click. It opens from the Project Versions pane or step 2 of the XTS wizard.

1. Select a domain on the left.

   Its possible parents appear in the middle pane.

2. Click a parent's radio button.

   Siblings are ticked automatically for the other domains, and the domain is marked Reviewed because you changed it.

3. If a domain is already right, click Mark as Reviewed.

   This is how you track progress when hundreds of tables exist.

   What you should see: the progress pie fills as domains are reviewed. In the XTS wizard, finish by clicking Mark Completed for step 2.

6.16 Domain variables, loopCount and categories

A domain variable is an instance variable of a domain, named global.DomainName.variableName. Generator parameters throughout the project can refer to it, and changing the value instantly changes everything that references it. Three exist by default:

You can add your own, such as global.User.startDate. Defaults cannot be deleted.

6.16.1 Changing how many rows

The docs describe three levels at which the row count can be set, from broad to narrow: the domain (template) level, which applies everywhere; the test data case level, which they recommend as the most flexible because different tests can have different counts; and the scenario level, for special cases. The steps below change it at the domain level.

1. Open the domain and find the loopCount variable in Domain Variables.

   It starts at 1, which is why a first run produces a single row.

2. Click its pencil icon, type the new value and click Save.

   What you should see: the Value column changes (in the vendor example, from 1 to 25).

For many domains at once, the Global Editing menu has an "Edit Domains LoopCount" screen. One quirk: if a domain's first generator is a ListGen with setLoop true, the list length overrides the loop count.

6.16.2 Domain categories

When a version has dozens of domains, open a domain, click Categorize, create a category and save; then filter the Domains pane by category or use Advanced Search. Scenarios, chains and chain sets can be categorised too.

6.16.3 The id variable and loopCount, as shown in the 2020 demo

In the same demo, the CTO notes that a new domain already has three variables and starts with id. He calls it the starting point from which GenRocket guarantees referential integrity. The id attribute reads it, and so does the username generator, which builds its value from the id; when he types 1,000 into the variable, both change together. He adds that the id attribute can be made invisible so receivers do not output it while it keeps working; the docs' Visible setting (section 6.17) is that switch.

The docs agree on the mechanics: the variable is the start of the id range, every parameter that refers to it follows a change, and integrity comes from children referring to the parent's primary attribute (section 6.16). They differ on one point. The demo scenario generated 10,000 users, with the loop count described as the first thing a scenario reads, but the docs say a new domain's loopCount starts at 1. The demo does not show where 10,000 was entered.

6.17 Attributes: the columns

An attribute defines a characteristic of a domain, and a domain needs attributes for any data to exist. Importing creates them from the schema; otherwise you add them by hand. Every new domain starts with an id attribute that produces a unique number per row.

Click an attribute to open the Attribute Dashboard. The top half holds the attribute's own parameters; the bottom half holds its generators.

6.17.1 Adding an attribute

1. In the Domain Dashboard, click New Attribute, type a name and Save.

   The name matters, because GenRocket uses it to guess a generator.

2. Look at the generator that was assigned, change parameters if needed and click Save Generator.

   What you should see: for firstName, the NameGen generator with parameters nameType, caseType, sortOrder, seed and waitAmount, and sample names in the preview.

To add columns that appeared later in the real database, the docs use Scratch Pad: type the existing domain name exactly as it is in GenRocket (not the table name), list the new column names, and choose Create with Generators.

6.17.2 Referencing one attribute from another, and the ordering rule

Many generators have a reference parameter that takes another attribute's value. EmailGen, for example, can reference the id so each email is unique (user1@example.com, user2@example.com...). Double-click the reference field and pick the attribute with the plus button.

Other attribute tasks, in brief: Copy an attribute to another domain or project version (rename it if copying within the same domain); use Group Edit to change one parameter on many attributes, or remove many attributes at once; and use the gear icon (Browse and Manage Attributes) to see which generators are in use.

6.18 Generators: the recipe for each value

A generator creates synthetic data (or queries existing data) for one attribute. Every attribute needs one. GenRocket advertises over 765 generators (a growing number), grouped by purpose and, for many countries, by country. Generators produce raw values only; receivers format them (receivers and scenarios chapter).

6.18.1 How a generator gets assigned automatically

You rarely pick generators from scratch. For each new or imported attribute GenRocket tries these steps in order, according to the "How are Generators Assigned" page:

1. Is there a matching Project generator tag? If yes, use it.
2. Otherwise, a matching Organization generator tag?
3. Otherwise, an intelligent guess from the domain name, attribute name, the project's industry type and usage patterns of other users. If Enforce Type/Limit is on, the column's type is considered too (a date column gets FlexibleDateRangeGen, a uniqueidentifier gets UUIDGen).
4. If nothing fits, SimplePatternGen, which builds a value like name1, name2, name3 from the attribute name and the row's id.

6.18.2 Replacing a generator on one attribute

1. Open the attribute and use the Quick Generator Replacement menu below the generator.

   Typing part of a name filters the list (typing "town" offers BulgariaTownGen and UKTownGen).

2. Pick the generator and click Replace, then set its parameters and click Save Generator.

   What you should see: the new generator's name and parameters, and a fresh preview. Not sure of the name? Use Remove Generator and the Add a Generator dialog, which has categories and a search box.

6.18.3 Assigning one generator to many attributes

If first_name and last_name both need NameGen, do it in one go through the gear icon above the attribute list.

1. Click the gear icon (Browse and Manage Attributes) and tick the attributes.

   Selected rows turn blue.

2. Type the generator name in Select Generator, click Set Generator and confirm with Yes.

   The dialog warns that existing generators on those attributes will be replaced.

3. Click Modify Generator Parameters to set a parameter for several attributes together.

   Here you can give firstName the value firstName in nameType and lastName the value lastName, each in its own row, or tick rows and set a shared parameter such as sortOrder.

The Generator Tuning Wizard (step 3 of the XTS wizard) does the same across the whole project version, with statistics on how well generators match attributes. Copying a configured generator to other attributes is possible too: Simple Copy to chosen attributes (up to 20 per bucket; it replaces the target's generator) and Pattern Copy to all attributes whose names follow a pattern.

6.18.4 Generator tags: steering the automatic choice

A generator tag is a rule "if an attribute's name matches this text, assign this generator". Each has a value, a match type (starts with, ends with, contains or matches) and an optional ignore-case flag. Project-level tags apply to all versions of a project; organisation-level tags apply everywhere. The page's example: every attribute named like firstName in a Canadian project should get CanadaNameGen. Tags live in the Generator Tags tab of the Management pane (project) or the Organization menu.

To see which generators GenRocket thinks fit an attribute, click the small icon beside it in the attribute list. A ranked list appears (in the vendor's example NameGen 84%, AustraliaNameGen 5%, and so on); the top one is the default, and a tag icon lets you promote another.

6.18.5 Generator families

The generator overview pages group generators into families; each generator has a "How do I use..." page with parameters.

Conditional generators can express rules such as "if age is under 18, return Child; between 18 and 60, Adult" or "50% Regular, 30% Premium, 20% VIP". The docs contrast them with test data rules (G-Rules), which attach conditions to cases instead of single attributes. The patterns they enable are taken further in the data patterns chapter.

6.18.6 Seeding: repeatable randomness

If two runs of a "random" generator give different numbers, test failures cannot be reproduced. A seed fixes this: with the same seed, a generator gives the same set every run. A seed can be set on an attribute's generator or through the domain's seed variable (button Global Seed applies it to the whole domain). The data patterns chapter shows the docs' example.

6.18.7 Linked generators

One attribute can hold several generators in a row (gen1, gen2...). They run left to right, a generator may use any generator to its left, and the last one gives the value. The docs' example builds a full name in a single fullName attribute: NameGen, AlphabetGen (middle initial), NameGen (last name), then ConcatGen joining the three. Without linking you would need four attributes and would have to hide three of them with Visible turned off.

6.18.8 ReferenceGen and ProxyReferenceGen: how a child finds its parent key

This pair is the mechanism behind foreign keys, and it confuses everyone once. The docs explain it as follows.

This is why the project version status report flags "foreign key attributes that do not have the ProxyReferenceGen": without the proxy, the child will not find its parent when a scenario is made. The referenced attribute is the parent's primary attribute; to target another column, tick its Primary checkbox.

6.18.9 Enforcing data type and size

A generator does not know that first_name is varchar(5) unless you tell it. The Enforce Type/Limit option makes the generator keep values within the attribute's Type and Size. It is on automatically for generators assigned by a DDL or XTS import, can be enabled for a whole project version when you create it, and can be set manually per generator. The docs list integer limits it applies, for example 127 for tinyint, 32767 for smallint and 2147483647 for int.

1. Open the attribute, set Type (for example varchar) and Size (for example 5) and save.

   These tell GenRocket the limits.

2. Tick Enforce Type/Limit in the generator section and save the generator.

   What you should see: the preview now shows names of at most five characters.

6.18.10 Primary keys after the current maximum

When you insert into a table that already holds rows, new keys must not collide with old ones. The docs' recipe links two generators in the id attribute. gen1 is QueryOnceGen with resourceName pointing at the JDBC resource, query = select max(id) + 1 from <tableName> and a sampleValue (5001 in the example) used for previews. gen2 is RangeGen whose startRange references gen1.

The General FAQ adds a second pattern: write the parent's ids to a delimited file as you insert, then read them back with ListCSVGen at scenario level so the keys match.

6.18.11 Inside a few generators, as shown in the 2020 demo

The CTO opens individual attributes to show that every generator works the same way: parameters decide its behaviour. Each example below is followed by what the docs say.

• A counter (RangeGen). The id uses a range generator with a start, an end and a "jump". Starting at 1,000 it counts 1000, 1001, ...; setting the jump to 3 makes the preview step by 3. The RangeGen page confirms startRange, endRange and jumpAmount, and that with no end it runs to the largest long number.
• Names (NameGen). He says the name generator draws on lists of first, last, male and female names for about 40 million combinations. The NameGen page confirms this and gives the list sizes (1,000 male, 3,000 female, 10,000 last names).
• A constant plus the id (ConcatGen). He replaces the username generator with a concatenation generator, types the constant "user" and picks the id attribute as the second part. The ConcatGen page builds BR01-1000 the same way, from a constant and a RangeGen value.
• A hashed value. He passes the id through an MD5 setting and the hash appears at the end of each output row. A hash is a one-way scramble: the same input always gives the same output, but the output cannot be turned back into the input. The EncryptGen page lists MD5, SHA-1, SHA-256, SHA-384 and SHA-512.
• SSN and card numbers. He says the SSN generator avoids numbers that are never issued, such as area number 666 The SSNGen page does not mention such a rule, so check the preview. For cards he picks Discover, MasterCard and Visa and says the numbers pass the Luhn check, the checksum formula payment forms use to catch mistyped numbers. The CreditCardGen page confirms Luhn-valid output and one on/off parameter per card brand. Luhn-valid does not mean real: no person owns these numbers.

6.18.12 Linked generators and "patterns, not randomness" in the 2020 Q&A

Asked how to generate values defined by a regular expression, the CTO answers in three parts. A regular-expression generator exists for that. More broadly, he says most generators do not produce random data, because random data is not good for testing; they produce patterns, and because linked generators can talk to each other, simple patterns combine into complex data. His examples are if/then/else, case, switch and a multi-way generator that works in percentages. For very complex rules there is a separate feature, test data rules, which takes if/then/else conditions. The docs name matching generators (IfThenElseGen, EvalCaseGen, SwitchGen, MultiWeightGen, StringRegexGen; see section 6.18.5) and describe test data rules (G-Rules) as the condition-based alternative. The data patterns chapter builds these patterns one by one.

6.19 ShopSphere design: generators for every column

The design below uses the documented generators; for StringRegexGen only the overview description is available. Domain names avoid underscores; the table name keeps them. "Domain variable" shows the setting that gives a sensible start.

Row counts: Customers loopCount 50. How many orders per customer and items per order is set per scenario, which is covered next. Because total_amount is generated independently, it will not equal the sum of its items. If a test needs that, use the conditional and calculation generators from the table above, or a G-Rule (the docs describe both).

6.20 Common mistakes and gotchas

6.21 Recap