Load-sensitive tests

[sarahec]

Three tests failed on a heavily-loaded machine and often work under lighter loads:

1. test_bucket_waiting
2. test_limiter_async_factory_get
3. test_mp_bucket_failures

UPDATE: The list became so long, based on many runs on different systems, that I opted to block files instead. The list before:

  disabledTests = [
    # hangs intermittently
    # https://github.com/vutran1710/PyrateLimiter/issues/245
    "test_limiter_01"
    # pass when run standalone, fail at higher loads
    # https://github.com/vutran1710/PyrateLimiter/issues/247
    "test_bucket_waiting"
    "test_limiter_async_factory_get"
    "test_limiter_delay"
    "test_mp_bucket"
    "test_mp_bucket_async"
    "test_mp_bucket_failures"
    "test_sqlite_filelock_bucket"
    # Slow: > 1.5 seconds/test run standalone on a fast machine
    # (Apple M3 Max with highest performance settings and 36GB RAM)
    "test_bucket_01"
    "test_bucket_02"
    "test_bucket_full"
    "test_bucket_leak"
    "test_factory_leak"
    "test_bucket_performance"
  ];

The new list:

  disabledTestPaths = [
    # Slow: > 1.5 seconds/test run standalone on a fast machine
    # (Apple M3 Max with highest performance settings and 36GB RAM)
    # and/or hang under load
    # https://github.com/vutran1710/PyrateLimiter/issues/245
    # https://github.com/vutran1710/PyrateLimiter/issues/247
    "tests/test_bucket_all.py"
    "tests/test_bucket_factory.py"
    "tests/test_limiter.py"
    "tests/test_multiprocessing.py"
  ];

Many of the slow tests are slow with a particular fixture value (sqlite routinely slows things down to > 20s/test).

I wonder if it would be useful to mark a subset of tests as smoke tests (via @pytest.mark)?

[paultiq]

Yes, many of these tests have timing assumptions that are fragile.

I fought a bit when adding "multiprocessing", and modified noxfile.py to break the tests into single threaded and multi-threaded cases.

Are the tests running in an isolate VM? Does disabling xdist help, as a first step? -n 0

[sarahec]

Yes, many of these tests have timing assumptions that are fragile.

I fought a bit when adding "multiprocessing", and modified noxfile.py to break the tests into single threaded and multi-threaded cases.

Are the tests running in an isolate VM? Does disabling xdist help, as a first step? -n 0

Generally, nixpkgs builds in an isolated "sandbox" though it doesn't get dedicated CPUs or GPUs. I'll try disabling xdist and see if it helps.