You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This release adds |settings.observability|, which can be used to enable :ref:`observability <observability>`.
4
+
5
+
When observability is enabled, Hypothesis writes data about each test case to the ``.hypothesis/observed`` directory in an analysis-ready `jsonlines <https://jsonlines.org/>`_ format. This data is intended to help users who want to dive deep into understanding their tests. It's also intended for people building tools or research on top of Hypothesis.
6
+
7
+
Observability can be controlled in two ways:
8
+
9
+
* via the new |settings.observability| argument,
10
+
* or via the ``HYPOTHESIS_OBSERVABILITY`` environment variable.
11
+
12
+
See :ref:`Configuring observability <observability-configuration>` for details.
13
+
14
+
If you use VSCode, we recommend the `Tyche <https://github.com/tyche-pbt/tyche-extension>`__ extension, a PBT-specific visualization tool designed for Hypothesis's observability interface.
Copy file name to clipboardExpand all lines: hypothesis-python/docs/changelog.rst
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -529,7 +529,7 @@ Fixes a bug with solver-based :ref:`alternative backends <alternative-backends>`
529
529
6.137.0 - 2025-08-05
530
530
--------------------
531
531
532
-
Add the |add_observability_callback|, |remove_observability_callback|, |with_observability_callback|, and |observability_enabled| methods to the :ref:`observability <observability>` interface. The previous |TESTCASE_CALLBACKS| is deprecated.
532
+
Add the ``add_observability_callback``, ``remove_observability_callback``, ``with_observability_callback``, and ``observability_enabled`` methods to the :ref:`observability <observability>` interface. The previous ``TESTCASE_CALLBACKS`` is deprecated.
533
533
534
534
This release also adds better threading support to observability callbacks. An observability callback will now only be called for observations generated by the same thread.
535
535
@@ -864,11 +864,11 @@ Further improve the performance of the constants-collection feature introduced i
864
864
6.135.3 - 2025-06-08
865
865
--------------------
866
866
867
-
This release adds the experimental and unstable |OBSERVABILITY_CHOICES| option for :ref:`observability <observability>`. If set, the choice sequence is included in ``metadata.choice_nodes``, and choice sequence spans are included in ``metadata.choice_spans``.
867
+
This release adds the experimental and unstable ``OBSERVABILITY_CHOICES`` option for :ref:`observability <observability>`. If set, the choice sequence is included in ``metadata.choice_nodes``, and choice sequence spans are included in ``metadata.choice_spans``.
868
868
869
869
These are relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence and choice spans.
870
870
871
-
We are actively working towards a better interface for this. Feel free to use |OBSERVABILITY_CHOICES| to experiment, but don't rely on it yet!
871
+
We are actively working towards a better interface for this. Feel free to use ``OBSERVABILITY_CHOICES`` to experiment, but don't rely on it yet!
872
872
873
873
.. _v6.135.2:
874
874
@@ -985,7 +985,7 @@ This patch resolves a Pandas FutureWarning (:issue:`4400`) caused by indexing wi
985
985
6.131.29 - 2025-05-27
986
986
---------------------
987
987
988
-
The observations passed to |TESTCASE_CALLBACKS| are now dataclasses, rather than dictionaries. The content written to ``.hypothesis/observed`` under ``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY`` remains the same.
988
+
The observations passed to ``TESTCASE_CALLBACKS`` are now dataclasses, rather than dictionaries. The content written to ``.hypothesis/observed`` under ``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY`` remains the same.
Copy file name to clipboardExpand all lines: hypothesis-python/docs/reference/integrations.rst
+10-18Lines changed: 10 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -75,17 +75,10 @@ If you're interested in similar questions, `drop me an email`_!
75
75
Observability
76
76
-------------
77
77
78
-
.. note::
78
+
.. tip::
79
79
80
80
The `Tyche <https://github.com/tyche-pbt/tyche-extension>`__ VSCode extension provides an in-editor UI for observability results generated by Hypothesis. If you want to *view* observability results, rather than programmatically consume or display them, we recommend using Tyche.
81
81
82
-
.. warning::
83
-
84
-
This feature is experimental, and could have breaking changes or even be removed
85
-
without notice. Try it out, let us know what you think, but don't rely on it
86
-
just yet!
87
-
88
-
89
82
Motivation
90
83
~~~~~~~~~~
91
84
@@ -108,24 +101,23 @@ debuggers such as `rr <https://rr-project.org/>`__ or `pytrace <https://pytrace.
108
101
because there's no good way to compare multiple traces from these tools and their
109
102
Python support is relatively immature.
110
103
104
+
.. _observability-configuration:
111
105
112
-
Configuration
113
-
~~~~~~~~~~~~~
106
+
Configuring observability
107
+
~~~~~~~~~~~~~~~~~~~~~~~~~
114
108
115
-
If you set the ``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY`` environment variable,
116
-
Hypothesis will log various observations to jsonlines files in the
109
+
The standard way to configure observability is with |settings.observability|.
110
+
111
+
Alternatively, observability can be configured by setting the ``HYPOTHESIS_OBSERVABILITY`` environment variable. If ``HYPOTHESIS_OBSERVABILITY`` is set to one of ``True``, ``true``, or ``1``, |settings.observability| defaults to ``True``. Note that unlike |settings.observability|, ``HYPOTHESIS_OBSERVABILITY`` only configures whether observability is enabled or disabled, not additional options like |ObservabilityConfig|.
112
+
113
+
When observability is enabled, and ``callbacks`` in |ObservabilityConfig| has not been configured to remove the default callback, Hypothesis will log various observations to jsonlines files in the
117
114
``.hypothesis/observed/`` directory. You can load and explore these with e.g.
If you are experiencing a significant slow-down, you can try setting
125
-
``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_NOCOVER`` instead; this will disable coverage information
126
-
collection. This should not be necessary on Python 3.12 or later, where coverage collection is very fast.
127
-
128
-
129
121
Collecting more information
130
122
^^^^^^^^^^^^^^^^^^^^^^^^^^^
131
123
@@ -181,7 +173,7 @@ While the observability format is agnostic to the property-based testing library
181
173
Choices metadata
182
174
++++++++++++++++
183
175
184
-
These additional metadata elements are included in ``metadata`` (as e.g. ``metadata["choice_nodes"]`` or ``metadata["choice_spans"]``), if and only if |OBSERVABILITY_CHOICES| is set.
176
+
These additional metadata elements are included in ``metadata`` (as e.g. ``metadata["choice_nodes"]`` or ``metadata["choice_spans"]``), if and only if observability is configured to include choices (see |ObservabilityConfig|).
Copy file name to clipboardExpand all lines: hypothesis-python/docs/reference/schema_metadata_choices.json
+2-2Lines changed: 2 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,7 @@
3
3
"properties": {
4
4
"choice_nodes": {
5
5
"type": ["array", "null"],
6
-
"description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute may change format or disappear without warning.\n\nThe sequence of choices made during this test case. This includes the choice value, as well as its constraints and whether it was forced or not.\n\nOnly present if |OBSERVABILITY_CHOICES| is ``True``.\n\n.. note::\n\n The choice sequence is a relatively low-level implementation detail of Hypothesis, and is exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence.",
6
+
"description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute may change format without warning.\n\nThe sequence of choices made during this test case. This includes the choice value, as well as its constraints and whether it was forced or not.\n\n.. note::\n\n Only present if observability is configured to include choices (see |ObservabilityConfig|).\n\n.. note::\n\n The choice sequence is a relatively low-level implementation detail of Hypothesis, and is exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence.",
7
7
"items": {
8
8
"type": "object",
9
9
"properties": {
@@ -31,7 +31,7 @@
31
31
"choice_spans": {
32
32
"type": "array",
33
33
"items": {"type": "array"},
34
-
"description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute may change format or disappear without warning.\n\nThe semantically-meaningful spans of the choice sequence of this test case.\n\nEach span has the format ``[label, start, end, discarded]``, where:\n\n* ``label`` is an opaque integer-value string shared by all spans drawn from a particular strategy.\n* ``start`` and ``end`` are indices into the choice sequence for this span, such that ``choices[start:end]`` are the corresponding choices.\n* ``discarded`` is a boolean indicating whether this span was discarded (see |PrimitiveProvider.span_end|).\n\nOnly present if |OBSERVABILITY_CHOICES| is ``True``.\n\n.. note::\n\n Spans are a relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| (and particularly |PrimitiveProvider.span_start| and |PrimitiveProvider.span_end|) for more details about spans."
34
+
"description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute may change format without warning.\n\nThe semantically-meaningful spans of the choice sequence of this test case.\n\nEach span has the format ``[label, start, end, discarded]``, where:\n\n* ``label`` is an opaque integer-value string shared by all spans drawn from a particular strategy.\n* ``start`` and ``end`` are indices into the choice sequence for this span, such that ``choices[start:end]`` are the corresponding choices.\n* ``discarded`` is a boolean indicating whether this span was discarded (see |PrimitiveProvider.span_end|).\n\n.. note::\n\n Only present if observability is configured to include choices (see |ObservabilityConfig|).\n\n.. note::\n\n Spans are a relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| (and particularly |PrimitiveProvider.span_start| and |PrimitiveProvider.span_end|) for more details about spans."
Copy file name to clipboardExpand all lines: hypothesis-python/docs/reference/schema_observations.json
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -36,7 +36,7 @@
36
36
},
37
37
"coverage": {
38
38
"type": ["object", "null"],
39
-
"description": "Mapping of filename to list of covered line numbers, if coverage information is available, or None if not. Hypothesis deliberately omits stdlib and site-packages code.",
39
+
"description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute might change format without warning, as we figure out the right format to support both line and branch coverage.\n\nMapping of filename to list of covered line numbers, if coverage information is available, or None if not. Hypothesis deliberately omits stdlib and site-packages code.\n\n.. note::\n\n Only present if observability is configured to include coverage (see |ObservabilityConfig|).",
0 commit comments