Make better configury check for all the python modules required for Sphinx

[rhc54]

Thank you for taking the time to submit an issue!

Background information

What version of Open MPI are you using? (e.g., v4.1.6, v5.0.1, git branch name and hash, etc.)

Head of OMPI main branch

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

Git clone

If you are building/installing from a git clone, please copy-n-paste the output from git submodule status.

Please describe the system on which you are running

• Operating system/version: Rocky 9
• Computer hardware: container
• Network type: TCP

---

Details of the problem

Trying to build OMPI main and hit the following fatal error:

Configuration error!

Versions
========

* Platform:         linux; (Linux-6.12.54-linuxkit-aarch64-with-glibc2.34)
* Python version:   3.11.7 (CPython)
* Sphinx version:   8.2.3
* Docutils version: 0.21.2
* Jinja2 version:   3.1.6
* Pygments version: 2.19.1

Last Messages
=============

None.

Loaded Extensions
=================

None.

Traceback
=========

      File "/usr/local/lib/python3.11/site-packages/sphinx/config.py", line 616, in eval_config_file
        raise ConfigError(msg % traceback.format_exc()) from exc
    sphinx.errors.ConfigError: There is a programmable error in your configuration file:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.11/site-packages/sphinx/config.py", line 601, in eval_config_file
        exec(code, namespace)  # NoQA: S102
        ^^^^^^^^^^^^^^^^^^^^^
      File "/opt/hpc/build/ompi/docs/conf.py", line 20, in <module>
        import sphobjinv as soi
    ModuleNotFoundError: No module named 'sphobjinv'


The full traceback has been saved in:
/tmp/sphinx-err-_1l67_q5.log

To report this error to the developers, please open an issue at <https://github.com/sphinx-doc/sphinx/issues/>. Thanks!
Please also report this if it was a user error, so that a better error message can be provided next time.
make[1]: *** [Makefile:2785: _build/man/ompi-wrapper-compiler.1] Error 2
make: *** [Makefile:1536: all-recursive] Error 1

It looks like this came in with the intersphinx changes by @jsquyres. Looks like you have to install some new module, but I can't find the magic to do so - and am therefore dead in the water. Sigh.

[bosilca]

pip install sphobjinv works for me.

[jsquyres]

docs/requirements.txt was updated.

You can just pip install -r docs/requirements.txt and it'll pick up the new requirement.

[rhc54]

Unfortunately, you have to first try to build and have it report failure before you realize that something out-of-the-ordinary is now required. I was able to finally get it past the problem (had to do some updates to my container), but it comes as a surprise to anyone who has been building OMPI in the past.

I wonder if it is possible to run the install on the requirements file as part of the make? I mean - what's the point of attempting a make if you know it is going to fail due to a missing requirement??

[rhc54]

Let me try to clarify my point. You check for autotools, python, sphinx, hwloc, libevent, pmix, and prrte - all because they are hard requirements. But you have added hard requirements for Python as well, and you do not check that those are met. So as a user, I'm forced to deal with hard errors because of missing Python modules.

Seems like you aren't treating things on an even basis, which makes it kinda annoying to those of us who are simply trying to build an update and find it broken. Don't know if it's possible or how hard it would be - just wondering why it isn't being done.

[jsquyres]

Ya, I hear what you're saying -- you have to fail and then you have to do the pip install. Sorry about that. ☹️ But at least you don't have to re-autogen or re-configure -- you can just pip install and re-make immediately. I guess my worldview is a little shifted these days -- I work with so many python packages that seeing this kind of "hey, I couldn't find a python package" message is just normal to me, and I just instinctively pip install -r ..../requirements.txt to get any new requirements.

Way back at the beginning of doing Sphinx here in OMPI, we talked about automatically making a virtual environment and using that (as part of make), but that was rejected because there are several different ways to make python virtual environments, and there are about 10,000 different opinions about conventions, etc. We decided it was best to say "you need all the python packages in docs/requirements.txt, however you want to get them): https://docs.open-mpi.org/en/v5.0.x/developers/sphinx.html#installing-sphinx. That being said, we could certainly extend the language in there to say "you need to install Sphinx and all the relevant Python packages" (vs. "you need to install Sphinx").

[bosilca]

We can run pip install --dry-run -r docs/requirements.txt and scan the output for any ^Would install. If not empty then some modules are missing.

[jsquyres]

You check for autotools, python, sphinx ...

Actually, that's a fair point. We do have oac_setup_sphinx.m4. But it doesn't check for this new python module. Indeed, we're not technically checking for all the python modules in docs/requirements.txt.

[jsquyres]

We can run pip install --dry-run -r docs/requirements.txt and scan the output for any ^Would install. If not empty then some modules are missing.

Yeah, that's a good idea. Maybe something like this. Well, maybe not "pip", but actually trying to python import each module or somesuch. .../me goes to think about this.

I'll re-open this, but remove "blocker".