Stabilize observability

[Liam-DeVoe]

Closes #4387.

• @settings(observability=...) takes True, False, or ObservabilitySettings. If not set, inherits from HYPOTHESIS_OBSERVABILITY, which is either true or false (no ObservabilitySettings parsing/control).
• HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_NOCOVER removed with no deprecation period
• HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY and HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_CHOICES kept for backwards-compat, but will be removed eventually
• observability.OBSERVABILITY_COLLECT_COVERAGE and observability.OBSERVABILITY_CHOICES replaced with observability.OBSERVABILITY_SETTINGS

For a future PR: with observability callbacks, there's no way to control ObservabilitySettings. You don't want to set @settings(observability=ObservabilitySettings(...)) on the test, because this would write to .hypothesis/observed, and you just want the observations in-memory.

You can do this by monkeypatching observability.OBSERVABILITY_SETTINGS. (indeed, hypofuzz does this right now). I'd like to get rid of that var, and add add_observability_callback(options: ObservabilitySettings). But semantic questions like "what if two callbacks where only one sets coverage=True" have to be answered. I do think this can be done, I just don't want to do it right now. So I've left the var in, but undocumented.

old description

• Disable coverage by default
  • IMO this is too much data (and too high risk for collision with sys.monitoring tools / slowdowns, but that's a weaker argument) to enable by default, but I'm open to discussion
• @settings(observability=...) takes three possible values:
  • observability=True enables observability
  • observability=False disables observability
  • observability=[option1, option2], like observability=["coverage"], enables observability, and also enables whatever options are specified
• if @settings(observability=...) is not set, inherit from HYPOTHESIS_OBSERVABILITY envvar
• HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_NOCOVER removed with no deprecation period
• HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY and HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_CHOICES kept for backwards-compat because it was easy, but are now-undocumented and will be removed at some point

I'd like to keep a very clear separation between @settings(observability=...), and add_observability_callback. IMO the former should always write to/be related to .hypothesis/observed, and the latter should never be (ie it's sent in-memory).

This brings up a non-trivial issue, probably for a future PR: when adding observability callbacks, there's no way to control the coverage and choices options. You don't want to set @settings(observability=["choices"]) on the test/profile, because this would write to .hypothesis/observed, and you just want the observations in-memory. Previously this was done with observability.OBSERVABILITY_COLLECT_COVERAGE and observability.OBSERVABILITY_CHOICES. I'd like to get rid of those vars eventually, and add add_observability_callback(options=["coverage"]). But semantic questions like "what if two callbacks where only one sets options=["coverage"] have to be answered. I do think this can be done, I just don't want to do it right now. So I've left those vars in, though now un-documented.

---

• update hypofuzz monkeypatching to use new OBSERVABILITY_SETTINGS

I'd also love to write a hypothesis.works blog post about this, at some point..

[cc @hgoldstein95]

[Zac-HD]

(incomplete thoughts)

[Zac-HD]

Some more notes after our in-person conversation; settings issue coming soon.

[Liam-DeVoe]

This is ready for review 👍 (it's a big one!)

[Zac-HD]

(oops, apparently my weeks-ago review didn't upload properly 😓(

[Zac-HD]

I think this is fine to merge as-is, but I'd like to split out a how-to guide and trim down the reference considerably as a followup.

[Zac-HD]

I think this is probably big enough that we should pull out a short blog post on observability!

(which will also allow us to cut this changelog down to a more reasonable size)

[Zac-HD]

this feels off somehow.

• wrong mechanism for setting a callback maybe
• also, I'd like to make backends give us an ObservabilityConfig rather than a callback alone

will think further on it though

[Liam-DeVoe]

also, I'd like to make backends give us an ObservabilityConfig rather than a callback alone

agreed, was punting this due to the size of this PR

[Zac-HD]

Given the size and age, pulling bits out for easier merging is a great strategy. I think we're fairly close here, but...

[Zac-HD]

I think this is in great shape, but removing the experimental/provisional status means that I want to tune a few last things before we merge:

• PrimitiveProvider should have an ObservabilityConfig as an attribute, not the current callback

and there are some followups for later

• blog post announcing it + listing some new research ideas etc
• think about better data formats for coverage info
• hypofuzz integration (simplifying coverage collection)
• better multi-thread _deliver_to_file performance via a cheap deque + lock

---

and then I dropped this for a few days, leaving just the following diff: Liam-DeVoe/hypothesis@stabilize-observability...Zac-HD:hypothesis:zac/update-observability

I'll probably get back to it next weekend, but it might be worth pulling out some smaller stuff that can be merged without taking the time to think carefully about the whole API change. (deliver-to-file, maybe the PrimitiveProvider.observability thing too as an internal/unstable interface?)