Safely using setup.cfg for metadata

7 min readFeb 20, 2020

I love the fact that setuptools now reads almost all of the information from setup.cfg. This in conjunction with the file: whatever syntax makes versioning and managing requires much easier. That’s great if you are using modern python, pip, and setuptools versions but what if you aren’t? This short article describes my journey down that road.

I’m going to use a packaging example on my github account as a working example. If you want to follow along, the repository is at:

dave-shawley/contacts

Simple web application for managing your contact information. The contacts service is simple to install: $ pip install…

github.com

When this article was written, the repository was at the pre-setupcfg tag so reset to commit f5027e6 if you want to follow along. My goal was to make setup.py go from:

to something like the following:

#!/usr/bin/env python
import setuptools
setuptools.setup()

The result of doing this without a setup.cfg file is an interesting case to consider. Especially since you can construct a python package from such a simple setup file 😮

$ python setup.py sdist
running sdist
running egg_info
creating UNKNOWN.egg-info
writing UNKNOWN.egg-info/PKG-INFO
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing top-level names to UNKNOWN.egg-info/top_level.txt
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txtrunning check
warning: Check: missing required meta-data: name, urlwarning: Check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be suppliedcreating UNKNOWN-0.0.0
creating UNKNOWN-0.0.0/UNKNOWN.egg-info
making hard links in UNKNOWN-0.0.0...
hard linking setup.py -> UNKNOWN-0.0.0
hard linking UNKNOWN.egg-info/PKG-INFO -> UNKNOWN-0.0.0/UNKNOWN.egg-info
hard linking UNKNOWN.egg-info/SOURCES.txt -> UNKNOWN-0.0.0/UNKNOWN.egg-info
hard linking UNKNOWN.egg-info/dependency_links.txt -> UNKNOWN-0.0.0/UNKNOWN.egg-info
hard linking UNKNOWN.egg-info/top_level.txt -> UNKNOWN-0.0.0/UNKNOWN.egg-info
Writing UNKNOWN-0.0.0/setup.cfg
creating dist
Creating tar archive
removing 'UNKNOWN-0.0.0' (and everything under it)$ ls dist
UNKNOWN-0.0.0.tar.gz

I mention this because this is exactly what happens if the version of setuptools does not support reading metadata from setup.cfg! We will come back to this later on. The result is ugly for end users and something that I want to avoid for my libraries. So how can we improve this situation for users while keeping the simplicity aspects of the Zen of Python in mind.

So what does a setup.cfg look like when using this method? It contains the metadata values and other keyword args that were previously passed to setuptools.setup function. Something like the following:

This file can get considerably more complex, but this short one matches the package that we started with.

So what’s the problem?

The problem that I ran into is that the metadata and options in setup.cfg are completely ignored by setuptools.setup before version 30.3 which is over two years old when I wrote this. It was released in December of 2016, so why do I care?

Because Linux distributions take some time to catch up. For example, Ubuntu bionic was the first release to install a setuptools that has minimal support for reading options from setup.cfg. This is the LTS release at the time of writing and is supported until 2023. It has setuptools version 39.0.1 installed. Unfortunately, version 39 of setuptools was released in the middle of adding this feature in earnest. The ability to read the version number from a file was added in version 39.2. If the current LTS doesn’t support the feature then there is a good chance that someone will stumble into issues and need help. That is not the experience that I want Python developers or users to have.

setup_requires kwarg might work

The setup_requires parameter to setuptools.setup seems to be made for this purpose. It lets you explicitly specify the packages that are required to exist before the setuptools.setup function is run.

The examples in this section are run against Python 3.5 and setuptools version 20.7.0 which is what is available in Ubuntu xenial. I think that this is a good candidate for a “broken” environment.

If we use the setup.cfg and setup.py described above, then building a source distribution gives us the following. If you are following along in my repository, then switch to commit f8798c4.

$ ./setup.py sdist
running sdist
running egg_info
creating UNKNOWN.egg-info
writing top-level names to UNKNOWN.egg-info/top_level.txt
writing UNKNOWN.egg-info/PKG-INFO
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
running check
warning: check: missing required meta-data: name, urlwarning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be suppliedcreating UNKNOWN-0.0.0
creating UNKNOWN-0.0.0/UNKNOWN.egg-info
creating UNKNOWN-0.0.0/tests
making hard links in UNKNOWN-0.0.0...
hard linking CONTRIBUTING.rst -> UNKNOWN-0.0.0
hard linking MANIFEST.in -> UNKNOWN-0.0.0
hard linking README.rst -> UNKNOWN-0.0.0
hard linking setup.cfg -> UNKNOWN-0.0.0
hard linking setup.py -> UNKNOWN-0.0.0
hard linking UNKNOWN.egg-info/PKG-INFO -> UNKNOWN-0.0.0/UNKNOWN.egg-info
hard linking UNKNOWN.egg-info/SOURCES.txt -> UNKNOWN-0.0.0/UNKNOWN.egg-info
hard linking UNKNOWN.egg-info/dependency_links.txt -> UNKNOWN-0.0.0/UNKNOWN.egg-info
hard linking UNKNOWN.egg-info/top_level.txt -> UNKNOWN-0.0.0/UNKNOWN.egg-info
hard linking tests/__init__.py -> UNKNOWN-0.0.0/tests
hard linking tests/test_status_endpoint.py -> UNKNOWN-0.0.0/tests
copying setup.cfg -> UNKNOWN-0.0.0
Writing UNKNOWN-0.0.0/setup.cfg
creating dist
Creating tar archive
removing 'UNKNOWN-0.0.0' (and everything under it)

Needless to say, this isn’t a good. I changed the version number to 0.0.1 in the commit so nothing is correct here. Let’s add setuptools==39.2 as a setup_requires kwarg and try that again.

$ ./setup.py sdist
/usr/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type'
  warnings.warn(msg)
/usr/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'project_urls'
  warnings.warn(msg)
/usr/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'python_requires'
  warnings.warn(msg)Installed /tmp/contacts/.eggs/setuptools-39.2.0-py3.5.egg
running sdist
running egg_info
creating UNKNOWN.egg-info
writing UNKNOWN.egg-info/PKG-INFO
writing top-level names to UNKNOWN.egg-info/top_level.txt
writing dependency_links to UNKNOWN.egg-info/dependency_links.txt
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest file 'UNKNOWN.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
writing manifest file 'UNKNOWN.egg-info/SOURCES.txt'
running check
warning: check: missing required meta-data: name, url
...

That didn’t work out so well. Why not?

I emphasized a line in the output that shows that the setuptools package was indeed installed into the .eggs directory as described in the setuptools developers guide. In full transparency, I was not expecting this to work; however, I was expecting a different type of failure. I dug through the setuptools source code and nothing caught my eye. It should have raised a pkg_resources.VersionConflict as far as I could tell. I decided to add another package to the setup_requires clause just to see if the behavior changed. I added ietfparse==1.0.0 which is a package that I wrote and it has no additional runtime requirements. Surprisingly enough, I got the expected behavior.

$ ./setup.py sdist
Traceback (most recent call last):
  File "./setup.py", line 6, in <module>
    setuptools.setup(setup_requires=['setuptools==39.2', 'ietfparse==1.0.0'])
  File "/usr/lib/python3.5/distutils/core.py", line 108, in setup
    _setup_distribution = dist = klass(attrs)
  File "/tmp/contacts/env/lib/python3.5/site-packages/setuptools/dist.py", line 269, in __init__
    self.fetch_build_eggs(attrs['setup_requires'])
  File "/tmp/contacts/env/lib/python3.5/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
    replace_conflicting=True,
  File "/tmp/contacts/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 826, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/tmp/contacts/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1085, in best_match
    dist = working_set.find(req)
  File "/tmp/contacts/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 695, in find
    raise VersionConflict(dist, req)
pkg_resources.VersionConflict: (setuptools 20.7.0 (/tmp/contacts/env/lib/python3.5/site-packages), Requirement.parse('setuptools==39.2'))

I can honestly say that I do not understand why this only occurs when you have more than one requirement in setup_requires. It is likely a defect somewhere in setuptools or pkg_resources version installed.

The underlying problem is that setuptools.setup does not reload the setuptools module while it is executing. If you think about how this would be implemented for a second, it is pretty simple. The setuptools module is loaded by the python interpreter, it runs the setup function in the module. The setup function processes the args and kwargs and installs the new setuptools package (which it does). Then it would have to reload the active module and re-execute the setup function with the new module in the system path. I’m not sure that is possible today. It certainly would be a lot of special case code to handle including setuptools in setup_requires.

So setup_requires is not the magic bullet that I hoped for.

Explicitly fail in setup.py

This is less than ideal, but we can check the setuptools version in setup.py and fail if it is too old. Here’s what I started out with.

It is nice and explicit. The result is a failing status code and a plain error message. It turns out that we can do better. The pkg_resources library includes a function named require that is easier to use and provides very nice output.

The output is a little noisier than a single line but it includes useful information and makes it clear that something is broken at the interpreter level. I find that developers are particularly bad at reading human-friendly error messages. They do respond rather quickly to stack traces though. Here’s what the output looks like now:

$ ./setup.py sdist
Traceback (most recent call last):
  File "./setup.py", line 6, in <module>
    pkg_resources.require('setuptools>=39.2')
  File "/tmp/contacts/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/tmp/contacts/env/lib/python3.5/site-packages/pkg_resources/__init__.py", line 834, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (setuptools 20.7.0 (/tmp/contacts/env/lib/python3.5/site-packages), Requirement.parse('setuptools>=39.2'))

Nice and explicit: you have the wrong version of setuptools installed! I consider that a win. If you want to use configuration in setup.cfg, then you should make life a little better on your users and include the pkg_resources.require call in your setup.py. It means fewer confused users creating issues for you and makes even educate your users about one of the dirtier corners of python packaging.