Skip to content

Stabilize observability #4563

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Liam-DeVoe wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Stabilize observability #4563

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
0d8896b
stabilize observability
Liam-DeVoe Dec 12, 2025
c91c141
reword release notes
Liam-DeVoe Dec 12, 2025
72077a4
Merge branch 'master' into stabilize-observability
Liam-DeVoe Jan 5, 2026
ea96f0b
Merge branch 'master' into stabilize-observability
Liam-DeVoe Jan 5, 2026
4fe2f91
format
Liam-DeVoe Jan 5, 2026
29175bc
update wording
Liam-DeVoe Jan 5, 2026
c4ee5d2
Merge branch 'master' into stabilize-observability
Liam-DeVoe Jan 12, 2026
ecaa6e3
add observability union
Liam-DeVoe Jan 12, 2026
1331778
Merge branch 'master' into stabilize-observability
Liam-DeVoe Jan 12, 2026
9f2845f
covering test case
Liam-DeVoe Jan 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions hypothesis-python/RELEASE.rst
View file
Open in desktop
Copy link
Member

@Zac-HD Zac-HD Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is probably big enough that we should pull out a short blog post on observability!

(which will also allow us to cut this changelog down to a more reasonable size)

Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
RELEASE_TYPE: minor

This release adds :ref:`observability <observability>`, a new feature in Hypothesis.

Observability is intended for those who want to dig into the raw data from tests or strategies—including those building tools or research on top of Hypothesis.

When observability is enabled, Hypothesis writes data about each test case to the ``.hypothesis/observed`` directory in an analysis-ready `jsonlines <https://jsonlines.org/>`_ format. This data includes a timing breakdown, coverage data, a representation of each argument, the result of any |assume| or |.filter| calls, and any other data we think is useful for analysis. See :ref:`observability <observability>` for the full data format.

Observability can be controlled in two ways:

* via the new |settings.observability| argument,
* or via the new ``HYPOTHESIS_OBSERVABILITY`` environment variable.

See :ref:`Configuring observability <observability-configuration>` for details.

If you use VSCode, we recommend the `Tyche <https://github.com/tyche-pbt/tyche-extension>`__ extension, a PBT-specific visualization tool designed for Hypothesis's observability interface.

(Note that Hypothesis has had an implementation of observability for a number of months, which was marked as experimental. This release brings observability into our public API, and removes that experimental label.)
8 changes: 4 additions & 4 deletions hypothesis-python/docs/changelog.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -641,7 +641,7 @@ Fixes a bug with solver-based :ref:`alternative backends <alternative-backends>`
6.137.0 - 2025-08-05
--------------------

Add the |add_observability_callback|, |remove_observability_callback|, |with_observability_callback|, and |observability_enabled| methods to the :ref:`observability <observability>` interface. The previous |TESTCASE_CALLBACKS| is deprecated.
Add the ``add_observability_callback``, ``remove_observability_callback``, ``with_observability_callback``, and ``observability_enabled`` methods to the :ref:`observability <observability>` interface. The previous ``TESTCASE_CALLBACKS`` is deprecated.

This release also adds better threading support to observability callbacks. An observability callback will now only be called for observations generated by the same thread.

Expand Down Expand Up @@ -976,11 +976,11 @@ Further improve the performance of the constants-collection feature introduced i
6.135.3 - 2025-06-08
--------------------

This release adds the experimental and unstable |OBSERVABILITY_CHOICES| option for :ref:`observability <observability>`. If set, the choice sequence is included in ``metadata.choice_nodes``, and choice sequence spans are included in ``metadata.choice_spans``.
This release adds the experimental and unstable ``OBSERVABILITY_CHOICES`` option for :ref:`observability <observability>`. If set, the choice sequence is included in ``metadata.choice_nodes``, and choice sequence spans are included in ``metadata.choice_spans``.

These are relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence and choice spans.

We are actively working towards a better interface for this. Feel free to use |OBSERVABILITY_CHOICES| to experiment, but don't rely on it yet!
We are actively working towards a better interface for this. Feel free to use ``OBSERVABILITY_CHOICES`` to experiment, but don't rely on it yet!

.. _v6.135.2:

Expand Down Expand Up @@ -1097,7 +1097,7 @@ This patch resolves a Pandas FutureWarning (:issue:`4400`) caused by indexing wi
6.131.29 - 2025-05-27
---------------------

The observations passed to |TESTCASE_CALLBACKS| are now dataclasses, rather than dictionaries. The content written to ``.hypothesis/observed`` under ``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY`` remains the same.
The observations passed to ``TESTCASE_CALLBACKS`` are now dataclasses, rather than dictionaries. The content written to ``.hypothesis/observed`` under ``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY`` remains the same.

.. _v6.131.28:

Expand Down
10 changes: 4 additions & 6 deletions hypothesis-python/docs/prolog.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
.. |settings.suppress_health_check| replace:: :obj:`settings.suppress_health_check <hypothesis.settings.suppress_health_check>`
.. |settings.stateful_step_count| replace:: :obj:`settings.stateful_step_count <hypothesis.settings.stateful_step_count>`
.. |settings.backend| replace:: :obj:`settings.backend <hypothesis.settings.backend>`
.. |settings.observability| replace:: :obj:`settings.observability <hypothesis.settings.observability>`

.. |~settings.max_examples| replace:: :obj:`~hypothesis.settings.max_examples`
.. |~settings.database| replace:: :obj:`~hypothesis.settings.database`
Expand All @@ -39,6 +40,7 @@
.. |~settings.suppress_health_check| replace:: :obj:`~hypothesis.settings.suppress_health_check`
.. |~settings.stateful_step_count| replace:: :obj:`~hypothesis.settings.stateful_step_count`
.. |~settings.backend| replace:: :obj:`~hypothesis.settings.backend`
.. |~settings.observability| replace:: :obj:`~hypothesis.settings.observability`

.. |settings.register_profile| replace:: :func:`~hypothesis.settings.register_profile`
.. |settings.get_profile| replace:: :func:`~hypothesis.settings.get_profile`
Expand Down Expand Up @@ -66,6 +68,8 @@
.. |Verbosity.normal| replace:: :obj:`Verbosity.normal <hypothesis.Verbosity.normal>`
.. |Verbosity.quiet| replace:: :obj:`Verbosity.quiet <hypothesis.Verbosity.quiet>`

.. |ObservabilityConfig| replace:: :obj:`ObservabilityConfig <hypothesis.ObservabilityConfig>`

.. |HypothesisException| replace:: :obj:`HypothesisException <hypothesis.errors.HypothesisException>`
.. |HypothesisDeprecationWarning| replace:: :obj:`HypothesisDeprecationWarning <hypothesis.errors.HypothesisDeprecationWarning>`
.. |NonInteractiveExampleWarning| replace:: :obj:`NonInteractiveExampleWarning <hypothesis.errors.NonInteractiveExampleWarning>`
Expand Down Expand Up @@ -145,12 +149,6 @@
.. |AVAILABLE_PROVIDERS| replace:: :data:`~hypothesis.internal.conjecture.providers.AVAILABLE_PROVIDERS`
.. |run_conformance_test| replace:: :func:`~hypothesis.internal.conjecture.provider_conformance.run_conformance_test`

.. |add_observability_callback| replace:: :data:`~hypothesis.internal.observability.add_observability_callback`
.. |remove_observability_callback| replace:: :data:`~hypothesis.internal.observability.remove_observability_callback`
.. |with_observability_callback| replace:: :data:`~hypothesis.internal.observability.with_observability_callback`
.. |observability_enabled| replace:: :data:`~hypothesis.internal.observability.observability_enabled`
.. |TESTCASE_CALLBACKS| replace:: :data:`~hypothesis.internal.observability.TESTCASE_CALLBACKS`
.. |OBSERVABILITY_CHOICES| replace:: :data:`~hypothesis.internal.observability.OBSERVABILITY_CHOICES`
.. |BUFFER_SIZE| replace:: :data:`~hypothesis.internal.conjecture.engine.BUFFER_SIZE`
.. |MAX_SHRINKS| replace:: :data:`~hypothesis.internal.conjecture.engine.MAX_SHRINKS`
.. |MAX_SHRINKING_SECONDS| replace:: :data:`~hypothesis.internal.conjecture.engine.MAX_SHRINKING_SECONDS`
Expand Down
2 changes: 2 additions & 0 deletions hypothesis-python/docs/reference/api.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,8 @@ Settings
.. autoclass:: hypothesis.Verbosity
:members:

.. autoclass:: hypothesis.ObservabilityConfig

.. autoclass:: hypothesis.HealthCheck
:undoc-members:
:inherited-members:
Expand Down
28 changes: 10 additions & 18 deletions hypothesis-python/docs/reference/integrations.rst
View file
Open in desktop
Copy link
Member

@Zac-HD Zac-HD Dec 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine to merge as-is, but I'd like to split out a how-to guide and trim down the reference considerably as a followup.

Original file line number Diff line number Diff line change
Expand Up @@ -75,17 +75,10 @@ If you're interested in similar questions, `drop me an email`_!
Observability
-------------

.. note::
.. tip::

The `Tyche <https://github.com/tyche-pbt/tyche-extension>`__ VSCode extension provides an in-editor UI for observability results generated by Hypothesis. If you want to *view* observability results, rather than programmatically consume or display them, we recommend using Tyche.

.. warning::

This feature is experimental, and could have breaking changes or even be removed
without notice. Try it out, let us know what you think, but don't rely on it
just yet!


Motivation
~~~~~~~~~~

Expand All @@ -108,24 +101,23 @@ debuggers such as `rr <https://rr-project.org/>`__ or `pytrace <https://pytrace.
because there's no good way to compare multiple traces from these tools and their
Python support is relatively immature.

.. _observability-configuration:

Configuration
~~~~~~~~~~~~~
Configuring observability
~~~~~~~~~~~~~~~~~~~~~~~~~

If you set the ``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY`` environment variable,
Hypothesis will log various observations to jsonlines files in the
The standard way to configure observability is with |settings.observability|.

Alternatively, observability can be configured by setting the ``HYPOTHESIS_OBSERVABILITY`` environment variable. If ``HYPOTHESIS_OBSERVABILITY`` is set to one of ``True``, ``true``, or ``1``, |settings.observability| defaults to ``True``. Note that unlike |settings.observability|, ``HYPOTHESIS_OBSERVABILITY`` only configures whether observability is enabled or disabled, not additional options like |ObservabilityConfig|.

When observability is enabled, and ``callbacks`` in |ObservabilityConfig| has not been configured to remove the default callback, Hypothesis will log various observations to jsonlines files in the
``.hypothesis/observed/`` directory. You can load and explore these with e.g.
:func:`pd.read_json(".hypothesis/observed/*_testcases.jsonl", lines=True) <pandas.read_json>`,
or by using the :pypi:`sqlite-utils` and :pypi:`datasette` libraries::

sqlite-utils insert testcases.db testcases .hypothesis/observed/*_testcases.jsonl --nl --flatten
datasette serve testcases.db

If you are experiencing a significant slow-down, you can try setting
``HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_NOCOVER`` instead; this will disable coverage information
collection. This should not be necessary on Python 3.12 or later, where coverage collection is very fast.


Collecting more information
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down Expand Up @@ -181,7 +173,7 @@ While the observability format is agnostic to the property-based testing library
Choices metadata
++++++++++++++++

These additional metadata elements are included in ``metadata`` (as e.g. ``metadata["choice_nodes"]`` or ``metadata["choice_spans"]``), if and only if |OBSERVABILITY_CHOICES| is set.
These additional metadata elements are included in ``metadata`` (as e.g. ``metadata["choice_nodes"]`` or ``metadata["choice_spans"]``), if and only if observability is configured to include choices (see |ObservabilityConfig|).

.. jsonschema:: ./schema_metadata_choices.json
:hide_key: /additionalProperties, /type
Expand Down
12 changes: 0 additions & 12 deletions hypothesis-python/docs/reference/internals.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,18 +27,6 @@ Alternative backends
.. autoclass:: hypothesis.errors.BackendCannotProceed
.. autoclass:: hypothesis.internal.intervalsets.IntervalSet

Observability
-------------

.. autofunction:: hypothesis.internal.observability.add_observability_callback
.. autofunction:: hypothesis.internal.observability.remove_observability_callback
.. autofunction:: hypothesis.internal.observability.with_observability_callback
.. autofunction:: hypothesis.internal.observability.observability_enabled

.. autodata:: hypothesis.internal.observability.TESTCASE_CALLBACKS
.. autodata:: hypothesis.internal.observability.OBSERVABILITY_COLLECT_COVERAGE
.. autodata:: hypothesis.internal.observability.OBSERVABILITY_CHOICES

Engine constants
----------------

Expand Down
4 changes: 2 additions & 2 deletions hypothesis-python/docs/reference/schema_metadata_choices.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"properties": {
"choice_nodes": {
"type": ["array", "null"],
"description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute may change format or disappear without warning.\n\nThe sequence of choices made during this test case. This includes the choice value, as well as its constraints and whether it was forced or not.\n\nOnly present if |OBSERVABILITY_CHOICES| is ``True``.\n\n.. note::\n\n The choice sequence is a relatively low-level implementation detail of Hypothesis, and is exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence.",
"description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute may change format without warning.\n\nThe sequence of choices made during this test case. This includes the choice value, as well as its constraints and whether it was forced or not.\n\n.. note::\n\n Only present if observability is configured to include choices (see |ObservabilityConfig|).\n\n.. note::\n\n The choice sequence is a relatively low-level implementation detail of Hypothesis, and is exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| for more details about the choice sequence.",
"items": {
"type": "object",
"properties": {
Expand Down Expand Up @@ -31,7 +31,7 @@
"choice_spans": {
"type": "array",
"items": {"type": "array"},
"description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute may change format or disappear without warning.\n\nThe semantically-meaningful spans of the choice sequence of this test case.\n\nEach span has the format ``[label, start, end, discarded]``, where:\n\n* ``label`` is an opaque integer-value string shared by all spans drawn from a particular strategy.\n* ``start`` and ``end`` are indices into the choice sequence for this span, such that ``choices[start:end]`` are the corresponding choices.\n* ``discarded`` is a boolean indicating whether this span was discarded (see |PrimitiveProvider.span_end|).\n\nOnly present if |OBSERVABILITY_CHOICES| is ``True``.\n\n.. note::\n\n Spans are a relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| (and particularly |PrimitiveProvider.span_start| and |PrimitiveProvider.span_end|) for more details about spans."
"description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute may change format without warning.\n\nThe semantically-meaningful spans of the choice sequence of this test case.\n\nEach span has the format ``[label, start, end, discarded]``, where:\n\n* ``label`` is an opaque integer-value string shared by all spans drawn from a particular strategy.\n* ``start`` and ``end`` are indices into the choice sequence for this span, such that ``choices[start:end]`` are the corresponding choices.\n* ``discarded`` is a boolean indicating whether this span was discarded (see |PrimitiveProvider.span_end|).\n\n.. note::\n\n Only present if observability is configured to include choices (see |ObservabilityConfig|).\n\n.. note::\n\n Spans are a relatively low-level implementation detail of Hypothesis, and are exposed in observability for users building tools or research on top of Hypothesis. See |PrimitiveProvider| (and particularly |PrimitiveProvider.span_start| and |PrimitiveProvider.span_end|) for more details about spans."
}
},
"required": ["traceback", "reproduction_decorator", "predicates", "backend", "sys.argv", "os.getpid()", "imported_at", "data_status", "interesting_origin", "choice_nodes", "choice_spans"],
Expand Down
2 changes: 1 addition & 1 deletion hypothesis-python/docs/reference/schema_observations.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
},
"coverage": {
"type": ["object", "null"],
"description": "Mapping of filename to list of covered line numbers, if coverage information is available, or None if not. Hypothesis deliberately omits stdlib and site-packages code.",
"description": ".. warning::\n\n EXPERIMENTAL AND UNSTABLE. This attribute might change format without warning, as we figure out the right format to support both line and branch coverage.\n\n Mapping of filename to list of covered line numbers, if coverage information is available, or None if not. Hypothesis deliberately omits stdlib and site-packages code.\n\n.. note::\n\n Only present if observability is configured to include coverage (see |ObservabilityConfig|).",
"additionalProperties": {
"type": "array",
"items": {"type": "integer", "minimum": 1},
Expand Down
9 changes: 8 additions & 1 deletion hypothesis-python/src/hypothesis/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,12 @@

import _hypothesis_globals

from hypothesis._settings import HealthCheck, Phase, Verbosity, settings
from hypothesis._settings import (
HealthCheck,
Phase,
Verbosity,
settings,
)
from hypothesis.control import (
assume,
currently_in_test_context,
Expand All @@ -30,11 +35,13 @@
from hypothesis.entry_points import run
from hypothesis.internal.detection import is_hypothesis_test
from hypothesis.internal.entropy import register_random
from hypothesis.internal.observability import ObservabilityConfig
from hypothesis.utils.conventions import infer
from hypothesis.version import __version__, __version_info__

__all__ = [
"HealthCheck",
"ObservabilityConfig",
"Phase",
"Verbosity",
"__version__",
Expand Down
Loading