Issue 12944: Accept arbitrary files for distutils' upload command

We need to specify files to upload to pypi.  Otherwise we have to use the web interface.

'Regarding the "setup.py upload" command, this isn't very helpful because it will not upload a package that was already built - for example, a Windows package built on another host while I am trying to release from a Linux desktop. Since automatic package building across a build farm is the only way I can actually build packages for most platforms, this rules out use of the "upload" command.'

See this post for more details:
http://as.ynchrono.us/2011/09/releasing-python-software-is-tedious.html

This has been a serious annoyance in the Twisted and pyOpenSSL release processes; it effectively prevents us from timely uploads of win32 binaries of any kind.

This was discussed some time ago in a thread starting here: http://mail.python.org/pipermail/catalog-sig/2011-May/003736.html  I’ll summarize the important parts below.

Currently, upload will only push files that are products of a command run from the same command line, e.g. “sdist upload”.
If you run “sdist”, check the result, and then do “sdist upload”, the second sdist should be exactly the same, unless your setup.py is buggy, you made changes to your files or you have a computer problem.  (I agree that’s a lot of unless).  So for distutils, having to run “sdist upload” is not inefficient, and makes you be explicit about the files you want to push, which is IMO good.

A trick can be used to avoid the second sdist to redo all its work: Fist you run “python setup.py sdist --keep-temp”, then you check the sdist, and to upload you call “python setup.py sdist --dry-run upload”.  I’m not in favor of adding that trick to the doc, as for normal usage, running sdist twice is okay.

As you may not be aware, distutils is under a feature freeze.  To cut a long story short, let’s just say that it’s impossible to gradually improve it without breaking a lot of third-party code, so we only fix bugs and we add features in distutils2, the next gen (included in Python 3.3 and available on PyPI for 2.4-3.2 soon).

I think there are good arguments to improve upload for distutils2, mainly:
- you never know what can go wrong, so you want to upload files you have tested
- PyPI will reject bad files anyway, no sense in being more restrictive on the client side

I propose a new option, say -f/--file, that takes a filename as argument:

  pysetup run upload -f dist/spam-0.2.tar.gz -f dist/spam-0.2.exe

I’d prefer to avoid globs (upload -i 'dist.spam-0.1.*' e.g.) because I want to retain the original design choice of the upload command: the person doing the upload needs to be explicit about what kind of file for what versions and with what Python version to upload.

Does that sounds good?

> pysetup run upload -f dist/spam-0.2.tar.gz -f dist/spam-0.2.exe

I'm not sure why it's "run upload" instead of just "upload", but maybe that's the convention in pysetup.  Apart from that, this looks like a vast improvement to me.  As far as not supporting globs goes, I guess I don't really care either way.  If globs aren't supported, then I'll probably end up globbing somewhere else (or constructing the full upload list in some other automated way).  The motivation for the complaint is that too many things require manual interaction; I'm not going to use a new, easily automated utility manually. :)

> I'm not sure why it's "run upload" instead of just "upload"
pysetup has actions and commands.  Actions do stuff on the Python installation: “pysetup metadata foobar” prints metadata, “pysetup remove foobar” uninstalls, “pysetup install foobar” fetches from PyPI and installs, etc.  Commands are like distutils commands, they work on a project in a source directory; for namespacing reasons, they are subitems of the run command.  (I’m working on documenting that.)

> Apart from that, this looks like a vast improvement to me.
Good!  Adding the easy keyword, as it’s not hard to implement if one understands how upload works, or someone could add tests without much difficulty.  I may get to it myself shortly, it will make a nice change from bug fixes and improve my karma :)

Someone from the mentoring project has volunteered to work on a patch.

Close as out of date since packaging has been removed from stdlib.

distutils2, while not actively developed any more for inclusion in the stdlib in 3.4, will mutate into one or more projects where these bug reports will be useful.  Bugs will certainly be migrated (and closed in this tracker), but please keep them open for easy query.

With the lift of the distutils feature freeze decided at PyCon 2013, this feature request could be done for 3.5.

I had a patch for this, but now think it would be an attractive nuisance, given that distutils does not always ensure proper HTTPS verification.

Not adding the ability to build sdists/wheels, check them and then upload them will help push people to twine, which doesn’t have all the .pypirc parsing bugs and is a good HTTPS citizen.

I will however commit a patch to clarify the message «no dist file created in earlier command».

A doc ticket already exists, moving this one back to a feature request and closing it.

type: behavior -> enhancement
title: Better error message for "setup.py upload" missing sdist/bdist -> Accept arbitrary files for distutils' upload command
messages: + msg311030
superseder: Better error message for setup.py upload command without sdist/bdist
resolution: wont fix
stage: needs patch -> resolved

versions: + Python 2.7, Python 3.6, Python 3.7, - Python 3.5
type: enhancement -> behavior

versions: - Python 3.3

nosy: + asvetlov
messages: + msg178249

resolution: out of date

messages: + msg144516
stage: needs patch

components: + Distutils2, - Distutils
title: setup.py upload to pypi needs to work with specified files -> Accept arbitrary files for packaging's upload command
nosy: + alexis
versions: + 3rd party, Python 3.3
messages: + msg144382