IMP: add test-deployment command

[gregcaporaso]

This PR would be nice to have merged for 2025.7, since we'll be teaching using the Docker container at the Snowbird workshop in September. Since it hasn't had a lot of testing yet, I put a warning in the help text on the command that it's subject to change.

Relatively quick command that is useful for testing in a tiny environment:

qiime dev test-deployment --skip qiime2 --skip q2cli --skip q2-types

Output includes the typical unit test output (pages and pages and pages), and then the following summary:

============================================================
TEST RESULTS SUMMARY
============================================================
Package                                            Status
------------------------------------------------------------
q2_metadata                                        PASS
q2_types                                           PASS
q2templates                                        PASS
------------------------------------------------------------
Total: 3, Passed: 3, Failed: 0

[gregcaporaso]

Doing some testing in a clean amplicon environment i'm getting some test failures.

These seem to be test configuration issues, and are some are probably candidates for skipping if proper configuration isn't in place.

For example, I get some failures related to selenium being missing in my environment from q2-vizard, q2-longitudinal, and q2-feature-table (these can be tested individually with qiime dev test-deployment --target q2-vizard):

======================================================================
ERROR: q2_vizard.tests.test_scatterplot (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: q2_vizard.tests.test_scatterplot
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/q2_vizard/tests/test_scatterplot.py", line 13, in <module>
    from selenium import webdriver
ModuleNotFoundError: No module named 'selenium'

======================================================================
ERROR: q2_feature_table.tests.test_summarize (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: q2_feature_table.tests.test_summarize
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/q2_feature_table/tests/test_summarize.py", line 20, in <module>
    from selenium import webdriver
ModuleNotFoundError: No module named 'selenium'

Some test failures are coming from missing click.testing (qiime dev test-deployment --target q2cli):

======================================================================
ERROR: tests.test_tools (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_tools
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/Users/jgcap/4-git-repos/qiime2/q2cli/q2cli/tests/test_tools.py", line 19, in <module>
    from click.testing import CliRunner
ModuleNotFoundError: No module named 'click.testing'

I'm also getting some test failures that seem related to cache and the parallel framework - I suspect this has something to do with my configuration, but maybe also tests that should be skipped under some circumstances (qiime dev test-deployment --target qiime2):

======================================================================
ERROR: test_multi_user (qiime2.core.tests.test_cache.TestCache)
This test determines if we can have multiple users successfully
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/qiime2/core/tests/test_cache.py", line 612, in test_multi_user
    with _fake_user_for_cache(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/qiime2/core/tests/test_cache.py", line 87, in _fake_user_for_cache
    raise ValueError('This action requires super user permissions which '
ValueError: This action requires super user permissions which you do not have

======================================================================
ERROR: test_load_complex_config (qiime2.sdk.tests.test_config.TestConfig)
Test that all parsl modules we currently map are correct
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/parsl/executors/high_throughput/executor.py", line 555, in _start_local_interchange_process
    (self.worker_task_port, self.worker_result_port) = comm_q.get(block=True, timeout=120)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/multiprocessing/queues.py", line 114, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/qiime2/sdk/tests/test_config.py", line 225, in test_load_complex_config
    with ParallelConfig(config, mapping):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/qiime2/sdk/parallel_config.py", line 396, in __enter__
    PARALLEL_CONFIG.dfk = parsl.load(PARALLEL_CONFIG.parallel_config)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 1445, in load
    cls._dfk = DataFlowKernel(config)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 190, in __init__
    self.add_executors(config.executors)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 1158, in add_executors
    executor.start()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/parsl/executors/high_throughput/executor.py", line 428, in start
    self._start_local_interchange_process()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/parsl/executors/high_throughput/executor.py", line 558, in _start_local_interchange_process
    raise Exception("Interchange failed to start")
Exception: Interchange failed to start

[Oddant1]

This one should already be skipping:

======================================================================
ERROR: test_multi_user (qiime2.core.tests.test_cache.TestCache)
This test determines if we can have multiple users successfully
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/qiime2/core/tests/test_cache.py", line 612, in test_multi_user
    with _fake_user_for_cache(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/q2dev-amplicon-2025-06-28/lib/python3.10/site-packages/qiime2/core/tests/test_cache.py", line 87, in _fake_user_for_cache
    raise ValueError('This action requires super user permissions which '
ValueError: This action requires super user permissions which you do not have

It should skipif you don't have sudo or if you're on a Mac. Not sure why it's trying to run at all here.
https://github.com/qiime2/qiime2/blob/cefa0f7bc53a58f0301d6341330726c4b7b6261e/qiime2/core/tests/test_cache.py#L581

[Oddant1]

The other one is a bit stranger. I think it's the only place in the entire framework where we're still creating a HighThroghputExecutor for testing, but we never submit anything to it.

[gregcaporaso]

Closing in favor of a GitHub Action-based workflow.