◐ Shell
reader mode source ↗
Skip to content

bpo-35346, platform: replace os.popen() with subprocess#10786

Merged
vstinner merged 1 commit into
python:masterfrom
vstinner:platform_popen
Dec 7, 2018
Merged

bpo-35346, platform: replace os.popen() with subprocess#10786
vstinner merged 1 commit into
python:masterfrom
vstinner:platform_popen

Conversation

@vstinner

@vstinner vstinner commented Nov 29, 2018

Copy link
Copy Markdown
Member

Replace os.popen() with subprocess.Popen in the platform module:

  • Pass the command as a list of strings, not as as a string;
  • Don't use a shell;
  • _syscmd_ver() now redirects errors to DEVNULL.

https://bugs.python.org/issue35346

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Can subprocess.check_output() be used here?

@vstinner

Copy link
Copy Markdown
Member Author

Can subprocess.check_output() be used here?

This function is documented in the "Older high-level API" section of the subprocess documentation. It seems like subprocess.run() is now preferred. subprocess.run() kills the process on exception, IMHO it's a nice enhancement (to not leak a process on error). I modified my PR to use subprocess.run().

@serhiy-storchaka

Copy link
Copy Markdown
Member

Should not check=True be passed? And check_output() looks as a more convenient specialized function for this case. You can omit stdout=PIPE and check=True.

@malemburg

Copy link
Copy Markdown
Member

+1 on using .check_output() instead. It's a special purpose API for exactly this use case and also available in Python 2.7.

@vstinner

Copy link
Copy Markdown
Member Author

+1 on using .check_output() instead. It's a special purpose API for exactly this use case and also available in Python 2.7.

I don't plan to backport this change to other branches.

check_output() / check=True.

I chose to keep the code checking manually the returncode. I don't see the point of a raising an exception, since the exception is ignored anyway.

@serhiy-storchaka

Copy link
Copy Markdown
Member

I think the code would be simpler, and therefore more readable, if use check_output().

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Visible effect: suppressed the output to stderr. I think it is worth a news entry. And it can be worth backporting.

@vstinner

Copy link
Copy Markdown
Member Author

I think the code would be simpler, and therefore more readable, if use check_output().

done

Visible effect: suppressed the output to stderr. I think it is worth a news entry.

done

And it can be worth backporting.

My PR only changes _syscmd_ver() when os.uname() doesn't exist, so only impacts Windows. I don't recall that I ever saw any spurious error when using platform on Windows.

I would prefer to minimize changes in stable branches. So if someone considers that it's worth it (I don't think so), I suggest to add " 2> %s" % DEV_NULL to _syscmd_ver() command, rather than using check_output().

@vstinner

Copy link
Copy Markdown
Member Author

Hum, my PR didn't work as expected on Windows. Currently, _syscmd_ver() tries to run "ver" (as a string) with os.open() which uses a shell. My PR avoids shell=True. _syscmd_ver() still worked thanks to "cmd /c ver"... but that's basically like calling "ver" with shell=True, since subprocess uses cmd.exe as the shell:

            if shell:
                startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
                startupinfo.wShowWindow = _winapi.SW_HIDE
                comspec = os.environ.get("COMSPEC", "cmd.exe")
                args = '{} /c "{}"'.format (comspec, args)

subprocess may not use cmd.exe if COMSPEC is set, but cmd.exe should always be available on Windows, no?

@vstinner

Copy link
Copy Markdown
Member Author

@malemburg, @serhiy-storchaka: Does my PR look better with check_output()?

@serhiy-storchaka

Copy link
Copy Markdown
Member

Thank you, it looks better. But please defer merging it until issues with _syscmd_file() be resolved.

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

LGTM if it LGT @malemburg.

@vstinner

Copy link
Copy Markdown
Member Author

Marc Andre Lemburg:

+1 on using .check_output() instead. It's a special purpose API for exactly this use case and also available in Python 2.7.

@malemburg: My PR now uses check_output() as you requested. Would you mind to review it and make sure that you agree with the change?


Note: It hasn't been mentioned explicitely previously, but my change removes platform.DEV_NULL. This constant is not documented. Since Python 3 has os.devnull and subprocess.DEVNULL, I don't think that it's worth it to keep platform.DEV_NULL for backward compatibility.

@vstinner

vstinner commented Dec 3, 2018

Copy link
Copy Markdown
Member Author

@malemburg: I plan to merge this PR at the end of the week.

Replace os.popen() with subprocess.check_output() in the platform module:

* platform.uname() (_syscmd_ver() helper function) now redirects
  stderr to DEVNULL;
* Remove platform.DEV_NULL;
* _syscmd_uname() and _syscmd_file() no longer catch AttributeError.
  The "except AttributeError:" was only needed in Python 2, when
  os.popen() was not always available. In Python 3,
  subprocess.check_output() is always available.
@vstinner

vstinner commented Dec 5, 2018

Copy link
Copy Markdown
Member Author

I squashed the 7 commits into a single commit to update the commit message, I rebased my PR and I also replaced os.open() with subprocess.check_output() in test_platform.

@vstinner vstinner merged commit 3a521f0 into python:master Dec 7, 2018
@vstinner vstinner deleted the platform_popen branch December 7, 2018 10:10
@python python deleted a comment from bedevere-bot Dec 7, 2018
@python python deleted a comment from bedevere-bot Dec 7, 2018
@python python deleted a comment from bedevere-bot Dec 7, 2018
@python python deleted a comment from bedevere-bot Dec 7, 2018
vstinner added a commit that referenced this pull request Dec 7, 2018
@pablogsal

Copy link
Copy Markdown
Member

I removed the comments of the buildbot failures because they were false positives due to bad git checkouts. I am investigating why this PR that we made to fix that problem did not work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants