# Generative Tests Overview

The generative (property-based) tests use Hypothesis to generate query model variable definitions.

There are detailed docstrings in the code, and a description of how to run the tests in the

[developer docs](../../DEVELOPERS.md#generative-tests). This document is an attempt to

provide an overview and introduction to what the tests are doing.

There is one main test, `test_query_model` in [test_query_model.py](test_query_model.py). This

generates a variable definition, executes it using the MSSQL, SQLite and in-memory query engines,

and checks that the results are the same.

Note that because the inputs are automatically generated, we can't test that the results are

*correct*, but we rely on other (unit, integration, acceptance) tests to check the exact outputs

produced by the query engines, and we assume that the engines are sufficiently different in

implementation that it is a sufficient test to check that they always produce the same results

from the same inputs.

## Background

The most difficult part of the generative tests is teaching Hypothesis how to construct queries

out of query model objects. The query model has various constraints or validation rules which tell

you, given a particular object, what sort of operations you can apply to it and what other kinds of

object you can combine it with.

One way of trying to produce valid queries is just to generate lots of different objects, try sticking

them together and reject any structures that are invalid. This is what we did at first. It has the benefit

of being simple, but as the number of different types in the query model grows it becomes harder and harder

to randomly generate valid examples and eventually it stops working altogether.

The approach we take now involves, effectively, applying the query model validation rules backwards. We start

by deciding on the sort of object we want to end up with e.g. a Series of one-row-per-patient integers. Then

we ask, given we've got this object, what sorts of operation could validly produce an object like that? We

then get Hypothesis to pick one. That operation in turn requires other inputs and so, again, we ask what

sorts of operation could validly produce it and get Hypothesis to pick one. This process repeats until we

reach a "terminal node" i.e. an operation which doesn't require any inputs. (And if Hypothesis doesn't

naturally give us a terminal node after reaching a certain depth, we force it to choose one.)

This is very efficient at generating large and complex queries for testing. But applying the validation rules

backwards is fundamentally quite a mind-stretching exercise so don't be surprised if it takes a little while

for everything to fall into place.

## Terminology

We are used to thinking about query model things such as `Series` as representing a concrete column

in a table. The generative tests require a mental shift from thinking in terms of concrete objects to

thinking in terms of  `strategies` - i.e. a `value` strategy is a recipe for generating a `Value`,

rather than a `Value` itself.

### Building strategies

Strategies for query model nodes are built in hypothesis by using

[`hypothesis.strategies.builds`](https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.builds)

or the [`hypothesis.strategies.composite`](https://hypothesis.readthedocs.io/en/latest/data.html#composite-strategies)

decorator.

Calling `builds` with a callable and strategies for the callable's arguments

creates a new strategy that works by drawing the arguments and then

passing them to the callable

e.g. if we have defined strategies for choosing integers and strings, we can create a query model

`Value` strategy with:

```

from hypothesis import strategies as st

integer_strategy = st.integers(min_value=0, max_value=10)

str_strategy = st.text(alphabet=["a", "b", "c"], min_size=0, max_size=3)

value = st.builds(Value, value=st.one_of(integer_strategy, str_strategy))

```

`value` here is still a *strategy*, not a concrete thing.

```

>>> value

builds(Value, value=one_of(integers(min_value=0, max_value=10), text(alphabet=['a', 'b', 'c'], max_size=3)))

```

We can get an actual example by calling `example()` on a strategy:

```

>>> value.example()

Value(value='ca')

```

(The `example()` method is intended to be used for exploration only, and not in tests or strategy

definitions. Hypothesis will complain if you try to do that.)

If we need to reason about the examples being drawn, we can use the `composite` decorator. This gives us a magic `draw` argument that we can use to get examples out of a component strategy:

```

@st.composite

def value(draw, integer_strategy, str_strategy):

    raw_value = draw(st.one_of(integer_strategy, str_strategy))

    if isinstance(raw_value, str):

        # do something

        ...

    return Value(value=raw_value)

```

Note that the function we've written here creates one example `Value`, which we're calling in

the normal way, with a concrete `value` keyword argument. The `composite` decorator works by

taking a function like this which returns *one* example, and converting it into function that

returns a *strategy* that produces such examples.

When we call `value()`, we get the function back:

```

>>> value(integer_strategy, str_strategy)

value(integer_strategy=integers(min_value=0, max_value=10), str_strategy=text(alphabet=['a', 'b', 'c'], max_size=3))

```

As it's a strategy, we can call example() on it to get an actual example:

```

>>> value_st = value(integer_strategy, str_strategy)

>>> value_st.example()

Value(value=5)

```

### Value vs Series

Note that in the query model, a `Value` is a type of `Series` which wraps a single static value

in a one-row-per-patient series which just has that value for every patient. The variable strategies

treat `Value` somewhat differently to other types of `Series`; the overall strategy for a `series`

selects from all possible query model nodes that return a `Series`, *except* `Value`. This is to

avoid generating a lot of examples that do not involve the database at all.

It's import to remember that when a strategy for a node uses input arguments that are

`series` and `value` strategies, those *both* represent `Series` nodes of different types.

## test_query_model

`test_query_model` is the main generative test, and takes a `variable` and `data`, both hypothesis

strategies.

```

@hyp.given(variable=variable_strategy, data=data_strategy)

@hyp.settings(**settings)

def test_query_model(query_engines, variable, data, recorder):

    recorder.record_inputs(variable, data)  # this is used to record and report on some helpful data about the tests

    run_test(query_engines, data, variable, recorder)

```

## data strategy

We define a simplified table schema that contains columns of various types. The [`data_setup`](data_setup.py) uses this schema to setup up a number of patient and event tables (2 of each), and

the [`data_strategy`](data_strategies.py) populates the tables with hypothesis-generated data

for each test.

## value strategies

The value strategies are strategies defining simple types; `int`, `bool`, `date`, `float` and `str`.

```

value_strategies = {

    int: st.integers(min_value=0, max_value=10),

    bool: st.booleans(),

    datetime.date: st.dates(

        min_value=datetime.date(2010, 1, 1), max_value=datetime.date(2020, 12, 31)

    ),

    float: st.floats(min_value=0.0, max_value=11.0, width=16, allow_infinity=False),

    str: st.text(alphabet=["a", "b", "c"], min_size=0, max_size=3),

}

```

The same value strategies are used for defining both data strategies and variable (query model)

strategies. Generally, we try to define these values with narrow enough ranges that they will

sometimes overlap, and we can test equality. e.g. if we are testing an addition operation that

takes 2 ints, we can be reasonably sure that `st.integers(min_value=0, max_value=10)` will

test addition of two ints that are the same at some point.

## variable strategies

The variable strategies are the most complex part of ehrQL's generative test strategies.

A variable is defined by calling `variable()` in [`variable_strategies.py`](variable_strategies.py),

with the tables, schema and value strategies as described above.

`variable` defines lots of inner functions, each of which returns a strategy for creating the

thing it is named for, not the thing itself.

For example, the `value` inner function returns a strategy for creating `Value` objects, and not a `Value` object itself.

A valid variable is a `Series`, chosen by selecting a type (one of the types in `value_strategies`) and a frame

that the variable must be consistent with (with one row per patient, because we require that any

variable on a dataset represents one row per patient). The variable returned will be a

series of the chosen type, consistent with the chosen frame.

```

def variable(patient_tables, event_tables, schema, value_strategies):

   ...

    @st.composite

    def valid_variable(draw):

        type_ = draw(any_type())

        frame = draw(one_row_per_patient_frame())

        return draw(series(type_, frame))

    return valid_variable()

```

Let's take a closer look at the elements of the `valid_variable`.

`type_` is a type, drawn using the `any_type()` strategy, that chooses one of the types used as the keys for `value_strategies`:

```

    def any_type():

        return st.sampled_from(list(value_strategies.keys()))

```

`frame` is a frame with one row per patient, drawn using the `one_row_per_patient_frame()`

strategy. That could be a simple patient table (picked from one of the two

`SelectPatientTable` nodes defined by our data strategies), or a `PickOneRowPerPatient`

node, which is the result of a number of sorting and filtering operations.

Each of the sort and filter operations are themselves defined by strategies which require a

series to sort/filter on, so a frame strategy can quickly become deeply nested. The nesting

can potentially become so deep that it exceeds Hypothesis' max allowed depth, resulting in the

generated example being discarded; to prevent this, various strategies check the depth, and if

it's gone too far, they return a strategy that will pick a "terminal" node; i.e. one that we know

won't recurse any more. In the case of a one-row-per-patient frame, that is a `SelectPatientTable`

node, which can only take one of the two patient tables.

```

    def one_row_per_patient_frame():

        if depth_exceeded():

            return select_patient_table()

        return st.one_of(select_patient_table(), pick_one_row_per_patient_frame())

```

### The `series` strategy

The `series` strategy is where things become particularly complex!

`series()` is a strategy for choosing another strategy. Specifically, it chooses a strategy

that will generate a particular type of `Series` node. Whenever a series is needed, we call `series()`

passing in the type of the series that we want to generate, and a frame that it should be consistent

with. The `series()` strategy does the job of finding all the available strategies, and choosing one.

>**NOTE**

>

>The `type_` and `frame` arguments to `series()` can be thought of as describing properties of the

>series we want to generate from the strategy. So when we pass in `int` as the `type_` and a patient

>frame, it means that we expect to generate an int-type series that has one row per patient. The way

>that series is generated (and whether it uses the `type_` and `frame` arguments itself) is determined

>by the specific strategy that `series()` chooses.

Within `series()`, we define `series_constraints`; this is a dict of all possible strategies for

operations that produce a series. The keys are strategy callables. The values are 2-tuples representing the possible return types of the series that will be generated by that strategy,

and the possible domains of the frame it can be drawn from.

Frame domains indicate whether a particular `Series` node needs to be drawn from a patient table (i.e.one-row-per-patient), a non-patient table, or either.

The `series()` strategy chooses from the possible operation strategies that meet the constraints of

the `type_` and `frame` passed into it.

### A simple example: count

For example, if we only had strategies for `exists` and `count`, and we call `series(int, patient_table)`,

where `patient_table` has been chosen via `one_row_per_patient_frame()`:

```

    def series(type_, frame):

        ...

        # define contraints for possible series strategies

        series_constraints = {

            exists: ({bool}, DomainConstraint.PATIENT),

            count: ({int}, DomainConstraint.PATIENT),

        }

        ...

        def constraints_match(s):

            ...

        # find possible series strategies that match constraints, and choose one

        possible_series = [s for s in series_types if constraints_match(s)]

        series_strategy = draw(st.sampled_from(possible_series))

        # draw a series from the chosen strategy

        return draw(series_strategy(type_, frame))

```

`exists` returns a bool series with one row per patient.

`count` returns an int series with one row per patient.

The `frame` we've passed in matches the domain constriants for both strategies (it's a

one-row-per-patient frame). However, `exists` produces a bool series, and we need a strategy

that produces an int series, so `count` is the only possible strategy that matches. `series()`

will draw from the `count` strategy.

Note that the `type_` and `frame` are always passed on as arguments to the selected series

strategy, so that all series operation strategies have the same function signature.

However, whether they are actually used depends on the individual strategy.

For individual series strategies, it's important to remember that the `type_` argument they

receive represents the type of the **resulting series**; it may be important for that particular

series node that we know what that resulting type is

(e.g. see the example of [`Add`](#a-more-complex-example-add) below).

Following through to the `count` strategy, called by our call to `series(int, frame)`:

```

    def count(_type, _frame):

        return st.builds(AggregateByPatient.Count, any_frame())

```

`count` is passed a type (int) from the `series()` strategy; this is the expected return

type of the series, but it's not required here; it's also not necessary to use the

one-row-per-patient frame we passed in. (By convention, the arguments are named with leading

underscores to indicate this).

`count` generates an `AggregateByPatient.Count` node which can drawn from either patient-level

or an event-level frame, so we let Hypothesis choose one to use as the input to

`AggregateByPatient.Count`. (Note that the `series` returned *from* `AggregateByPatient.Count`

will be a one-row-per-patient series, consistent with `frame`.)

### A more complex example: add

If we now assume we also have a stratgey for `add`, and we again call `series(int, frame)`,

where `frame` has been chosen via `one_row_per_patient_frame()`:

```

    def series(type_, frame):

        ...

        # define contraints for possible series strategies

        series_constraints = {

            exists: ({bool}, DomainConstraint.PATIENT),

            count: ({int}, DomainConstraint.PATIENT),

            add: ({int, float}, DomainConstraint.PATIENT))

        }

        ...

```

`add` returns either an int or a float series, with one row per patient.

This time, the `frame` we've passed in again matches the domain constriants for all strategies.

We need a strategy that produces an int series, so now `series` can select from the `count` and

`add` strategies. Let's assume it chooses `add`.

```

    def add(type_, frame):

        ...

```

`add` is passed a type (int) from the `series()` strategy; this is the **expected return

type** of the series. In the case of the `add` strategy, this IS important. An `add` strategy

produces an `Add` query model node, which takes two series as arguments, and returns another series.

It can return an int or a float, and the aruments can be either int or float, BUT they all must be

the same.

This is the definition of `Add` in [`query_model.nodes`](ehrql/query_model/nodes.py)

```

class Function:

    class Add(Series[Numeric]):

        lhs: Series[Numeric]

        rhs: Series[Numeric]

```

A numeric type `Series` can be float or int, but both arguments to `Add` must be of the same type,

and return another `Series` of that type. In other words, if the expected return type is an int, we

know that we need to build our `add` strategy with int inputs.

So, for each of the lhs and rhs of the `Add` operation, we need a Hypothesis strategy that

produces an int `Series`. We can do that with `series(type_, frame)`.

However, there's a bit more to consider here. As mentioned before, a `Value` is also a type of

`Series`. `Value`s represent constants, and in order to cover the case where the `rhs` or `lhs` is

a constant, we need our strategy to allow a `Value` to be chosen as well. In addition, we have a

`frame` passed to the `add` strategy, but the `lhs` and `rhs` arguments can be chosen from different

tables, so we need our strategy to allow for this too.

So, we have two arguments; we say that one of them MUST be an int series drawn from the provided frame. This ensures that we don't end up with operations on two `Value` series, which don't touch

the database.

For the second argument, this could be a series drawn from the same frame as the first series, or

from a different frame, OR it could be a value.

And finally, these arguments could be either the `lhs` or the `rhs` of the `Add` operation, so we

need the strategy to choose that too.

This logic is implemented in the `binary_operation_with_types()` helper method, which deals with the

many query model nodes operate on two inputs, a `lhs` and `rhs` `Series`.

### Adding a new variable strategy

Let's assume we want to add the `GT` node as a new variable strategy.

First, look at how the `GT` node is implemented:

```

class Function:

    class GT(Series[bool]):

        lhs: Series[Comparable]

        rhs: Series[Comparable]

```

A `GT` node takes two `Comparable` series as inputs, and returns a boolean series. A `Comparable`

series can be of any type except bool, but the `lhs` and `rhs` series must be of the same type.

We can add a new entry to the `series_constraints` dict in `series()`, defining the return type of

the series (bool), and the domain constraint.

A `GT` operation can return a series consistent with *either* a one-row-per-patient frame, or a many-rows-per-patient frame:

```

    def series(type_, frame):

        ...

        # define contraints for possible series strategies

        series_constraints = {

            exists: ({bool}, DomainConstraint.PATIENT),

            count: ({int}, DomainConstraint.PATIENT),

            gt: ({bool}, DomainContstraint.ANY),

        }

        ...

```

Next we define the `gt` strategy. We can use a helper function `any_comparable_type()` to draw a

suitable type. Note that the `_type` passed in to the `gt` function is the expected return type of the series (bool), and in this case it does not have an impact on the implementation of the `gt`

strategy.

This is a binary operation like many other nodes, so the implementation follows the same strategy as

described for the `add` strategy above.

```

    @st.composite

    def gt(draw, _type, frame):

        type_ = draw(any_comparable_type())

        return draw(binary_operation(type_, frame, Function.GT))

```