-
Notifications
You must be signed in to change notification settings - Fork 30
Description
Describe the bug
We encountered an issue where an import statement tried to load a library that didn't exist in the runtime. As this occurred in every scenario run, all of these scenarios failed with the "untested" state. The exit code for behavex is 0.
In the HTML report, the feature file is flagged as green but when you drill down to the scenarios they are all grey (untested).
This led to us thinking the tests were passing but in reality were all failing.
To Reproduce
In our case we found this issue importing pyodbc but didn't have the relevant ODBC drivers in the runtime.
Steps to reproduce the behavior:
- Add an import statement that imports a dependency where an underlying library cannot be found (e.g.
pyodbc) - Run the test in an environment without ODBC drivers. For example, using a Python container.
- You'll see that tests passed
- The test output will say
1 untested(or more depending on how many tests were run) - In the test output folder, there's a
stdMainProcess2.txtthat will show a stack trace like this:
Error using Runner class, details:
Traceback (most recent call last):
File "/virtualenv-FFP0Gh2h-py3.11/lib/python3.11/site-packages/behavex/runner.py", line 1021, in _launch_behave
runner.run()
File "/virtualenv-FFP0Gh2h-py3.11/lib/python3.11/site-packages/behave/runner.py", line 1139, in run
return self.run_with_paths()
^^^^^^^^^^^^^^^^^^^^^
File "/virtualenv-FFP0Gh2h-py3.11/lib/python3.11/site-packages/behave/runner.py", line 1144, in run_with_paths
self.load_step_definitions()
File "/virtualenv-FFP0Gh2h-py3.11/lib/python3.11/site-packages/behave/runner.py", line 1131, in load_step_definitions
load_step_modules(step_paths)
File "/virtualenv-FFP0Gh2h-py3.11/lib/python3.11/site-packages/behave/runner_util.py", line 596, in load_step_modules
exec_file(os.path.join(path, name), step_module_globals)
File "/virtualenv-FFP0Gh2h-py3.11/lib/python3.11/site-packages/behave/runner_util.py", line 567, in exec_file
exec(code, globals_, locals_)
File "features/steps/__init__.py", line 1, in <module>
File "/my_glue.py", line 2, in <module>
import pyodbc
ImportError: libodbc.so.2: cannot open shared object file: No such file or directory
Expected behavior
The exit code should reflect that the tests failed to run. I'm not sure what the untested state is for, but in this case it was a failure so I wonder if the status should reflect that somehow and maybe even reference the relevant behavex log files for further debugging.