Python 3.7 and 3.8 compatibility

[ndharms]

I'm mainly posting this issue for discussion and to highlight possible todos.

I was trying to install PyKED and PyTeCK in an environment that requires python 3.7+. I ended up spending a fair amount of time banging my head against the wall with Anaconda until I realized that PyKED (and PyTeCK subsequently) uses python 3.6. Given that Python 3.8 has been released, should there be an effort to convert/modify aspects of PyKED (and PyTeCK) to support 3.7 and 3.8? If so what changes need to be made?

[bryanwweber]

I think the only thing that needs to change is that new packages need to be built. Basically, if you bump the build number in the conda.recipe/meta.yaml that might work. It's also possible that something like limiting, e.g., Pandas is causing the reliance on 3.6, because there aren't built packages for that old of a dependency on 3.7 and 3.8.

You could probably install with Pip, provided that you installed all the dependencies of PyKED first and then just used pip to install PyKED. Or you could install from source.

[kyleniemeyer]

Yeah, I was about to say the same. I imagine/hope most dependencies support 3.7, but 3.8 is probably less likely at this point.

[kyleniemeyer]

Quick look at dependencies:

• Pandas supports 3.7
• Pyyaml 3.13 supports 3.7
• numpy v1.17.2 supports 3.7
• I think Pint v0.8.1 has a noarch package
• habanero v0.6.0 has noarch
• uncertanties v3.0.3 supports 3.7

The problem might be Cerberus. Right now we only support up versions below 1.2, and v1.1 only supports up to Python 3.6. (It looks like for v1.2 and v1.3.1 now have noarch packages...).

@bryanwweber do you recall why we limit Pint to <0.9?

[bryanwweber]

The upper limit on all the dependencies is basically best practice, to avoid breaking changes assuming they follow semver. Since Pint is still zero-ver, it seemed best to limit to the next point release.

[kyleniemeyer]

@nateharms if you have the bandwidth, it'd be great if you could check out PyKED's compatibility with newer versions of those dependencies—that will make it easier for us to support Python 3.7+.

But, if cerberus is really limiting us here, then I guess we'll need to revisit the issues limiting us to v1.1

[ndharms]

I don't mind looking into it, I'll let y'all know how things progress.

[ndharms]

Okay, from working with the build-environment.yaml file, I had to make the following changes to the dependency list get the environment to solve:

Added:

• python == 3.7

Modified:

• cerberus>=1.0.0,<1.2 -> cerberus>=1.2,<1.3
• pandas >=0.22.0,<0.23 -> pandas >=0.23.0,<0.24

And when running conda list I get the following

# packages in environment at /Users/nathan/anaconda/envs/build-env:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.12                   py37_0  
anaconda-client           1.7.2                    py37_0  
appnope                   0.1.0                    py37_0  
asn1crypto                1.2.0                    py37_0  
attrs                     19.3.0                     py_0  
babel                     2.7.0                      py_0  
backcall                  0.1.0                    py37_0  
beautifulsoup4            4.8.1                    py37_0  
blas                      1.0                         mkl  
bleach                    3.1.0                    py37_0  
bzip2                     1.0.8                h1de35cc_0  
ca-certificates           2019.9.11            hecc5488_0    conda-forge
cerberus                  1.2                        py_1    conda-forge
certifi                   2019.9.11                py37_0    conda-forge
cffi                      1.13.0           py37hb5b8e2f_0  
chardet                   3.0.4                 py37_1003  
clyent                    1.2.2                    py37_1  
conda                     4.7.12                   py37_0  
conda-build               3.18.10                  py37_3  
conda-package-handling    1.6.0            py37h1de35cc_0  
cryptography              2.3.1            py37hdbc3d79_0  
decorator                 4.4.0                    py37_1  
defusedxml                0.6.0                      py_0  
doctr                     1.8.0                      py_0    conda-forge
docutils                  0.15.2                   py37_0  
entrypoints               0.3                      py37_0  
filelock                  3.0.12                     py_0  
glob2                     0.7                        py_0  
habanero                  0.6.0                      py_0    conda-forge
idna                      2.8                      py37_0  
imagesize                 1.1.0                    py37_0  
importlib_metadata        0.23                     py37_0  
intel-openmp              2019.4                      233  
ipython                   7.8.0            py37h39e3cac_0  
ipython_genutils          0.2.0                    py37_0  
jedi                      0.15.1                   py37_0  
jinja2                    2.10.3                     py_0  
jsonschema                3.1.1                    py37_0  
jupyter_core              4.6.0                    py37_0  
libarchive                3.3.3                h786848e_5  
libcxx                    4.0.1                hcfea43d_1  
libcxxabi                 4.0.1                hcfea43d_1  
libedit                   3.1.20181209         hb402a30_0  
libffi                    3.2.1                h475c297_4  
libgfortran               3.0.1                h93005f0_2  
libiconv                  1.15                 hdd342a3_7  
liblief                   0.9.0                h2a1bed3_2  
libxml2                   2.9.9                hf6e021a_1  
lz4-c                     1.8.1.2              h1de35cc_0  
lzo                       2.10                 h362108e_2  
markupsafe                1.1.1            py37h1de35cc_0  
mistune                   0.8.4            py37h1de35cc_0  
mkl                       2019.4                      233  
mkl-service               2.3.0            py37hfbe908c_0  
mkl_fft                   1.0.14           py37h5e564d8_0  
mkl_random                1.1.0            py37ha771720_0  
more-itertools            7.2.0                    py37_0  
nbconvert                 5.6.0                    py37_1  
nbformat                  4.4.0                    py37_0  
nbsphinx                  0.4.3                      py_0    conda-forge
ncurses                   6.1                  h0a44026_1  
numpy                     1.17.2           py37h99e6662_0  
numpy-base                1.17.2           py37h6575580_0  
openssl                   1.0.2r               h1de35cc_0    conda-forge
packaging                 19.2                       py_0  
pandas                    0.23.4           py37h6440ff4_0  
pandoc                    2.2.3.2                       0  
pandocfilters             1.4.2                    py37_1  
parso                     0.5.1                      py_0  
pexpect                   4.7.0                    py37_0  
pickleshare               0.7.5                    py37_0  
pint                      0.8.1                      py_1    conda-forge
pip                       19.3.1                   py37_0  
pkginfo                   1.5.0.1                  py37_0  
prompt_toolkit            2.0.10                     py_0  
psutil                    5.6.3            py37h1de35cc_0  
ptyprocess                0.6.0                    py37_0  
py-lief                   0.9.0            py37h1413db1_2  
pycosat                   0.6.3            py37h1de35cc_0  
pycparser                 2.19                     py37_0  
pygments                  2.4.2                      py_0  
pyopenssl                 19.0.0                   py37_0  
pyparsing                 2.4.2                      py_0  
pyrsistent                0.15.4           py37h1de35cc_0  
pysocks                   1.7.1                    py37_0  
python                    3.7.0                hc167b69_0  
python-dateutil           2.8.0                    py37_0  
python-libarchive-c       2.8                     py37_13  
pytz                      2019.3                     py_0  
pyyaml                    3.13             py37h1de35cc_0  
readline                  7.0                  h1de35cc_5  
requests                  2.22.0                   py37_0  
ripgrep                   0.10.0               hc07d326_0  
ruamel_yaml               0.15.46          py37h1de35cc_0  
setuptools                41.4.0                   py37_0  
six                       1.12.0                   py37_0  
snowballstemmer           2.0.0                      py_0  
soupsieve                 1.9.3                    py37_0  
sphinx                    2.2.0                      py_0  
sphinxcontrib-applehelp   1.0.1                      py_0  
sphinxcontrib-devhelp     1.0.1                      py_0  
sphinxcontrib-htmlhelp    1.0.2                      py_0  
sphinxcontrib-jsmath      1.0.1                      py_0  
sphinxcontrib-qthelp      1.0.2                      py_0  
sphinxcontrib-serializinghtml 1.1.3                      py_0  
sqlite                    3.30.0               ha441bb4_0  
testpath                  0.4.2                    py37_0  
tk                        8.6.8                ha441bb4_0  
tqdm                      4.36.1                     py_0  
traitlets                 4.3.3                    py37_0  
uncertainties             3.0.3                 py37_1000    conda-forge
urllib3                   1.24.2                   py37_0  
wcwidth                   0.1.7                    py37_0  
webencodings              0.5.1                    py37_1  
wheel                     0.33.6                   py37_0  
xz                        5.2.4                h1de35cc_4  
yaml                      0.1.7                hc338f04_2  
zipp                      0.6.0                      py_0  
zlib                      1.2.11               h1de35cc_3  
zstd                      1.3.7                h5bba6e5_0 

And then the tests (python setup.py test) won't run when you have cerberus >= 1.2 because of import errors:

__________________________________ ERROR collecting pyked/tests/test_validation.py ___________________________________
pyked/tests/test_validation.py:35: in <module>
    v = OurValidator(schema)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:169: in __init__
    self.schema = kwargs.get('schema', None)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:509: in schema
    self._schema = DefinitionSchema(self, schema)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/schema.py:69: in __init__
    self.validate(schema)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/schema.py:197: in validate
    self._validate(schema)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/schema.py:218: in _validate
    if not self.schema_validator(schema, normalize=False):
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:879: in validate
    self.__validate_unknown_fields(field)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:912: in __validate_unknown_fields
    if not validator({field: value}, normalize=False):
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:877: in validate
    self.__validate_definitions(definitions, field)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:940: in __validate_definitions
    result = validate_rule(rule)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:922: in validate_rule
    return validator(definitions.get(rule, None), field, value)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:1236: in _validate_schema
    self.__validate_schema_mapping(field, schema, value)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:1247: in __validate_schema_mapping
    if not validator(value, update=self.update, normalize=False):
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:877: in validate
    self.__validate_definitions(definitions, field)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:940: in __validate_definitions
    result = validate_rule(rule)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:921: in validate_rule
    validator = self.__get_rule_handler('validate', rule)
../../anaconda/envs/build-env/lib/python3.7/site-packages/cerberus/validator.py:338: in __get_rule_handler
    "domain.".format(rule, domain))
E   RuntimeError: There's no handler for 'isvalid_history' in the 'validate' domain.

NOTE: I modified the setup.py file by commenting out line 55 install_requires=install_requires to get it to run with my build environment.

[bryanwweber]

So, at least hypothetically, @kyleniemeyer (as the maintainer of the Cerberus conda-forge feedstock) could push a commit there that bumps the build number for 1.1 and sets it to a noarch package, then another commit to come back to the current version. I'm not sure if conda-forge would approve of that, but what they don't know?

[bryanwweber]

But honestly, you're probably better off with pip here. Just create a new environment, conda install everything except PyKED and Cerberus, then pip install pyked should pull in the right version of Cerberus.

[ndharms]

Update: I set up my environment according to @bryanwweber's suggestion and everything is working in order. It's a good workaround for now, but maybe a change that could be addressed in the future?

Thanks again for y'all's help!

[bryanwweber]

Right, the key here is that we need to fix or remove the dependence on Cerberus. Feel free to jump in to that 😄