◐ Shell
reader mode source ↗
Skip to content

bpo-27413: add --no-ensure-ascii argument to json.tool#201

Closed
dhimmel wants to merge 13 commits into
python:masterfrom
dhimmel:json_tool
Closed

bpo-27413: add --no-ensure-ascii argument to json.tool#201
dhimmel wants to merge 13 commits into
python:masterfrom
dhimmel:json_tool

Conversation

@dhimmel

@dhimmel dhimmel commented Feb 20, 2017

Copy link
Copy Markdown
Contributor

Work in progress, pending consensus on whether feature should be added

This pull request increases the utility of python -m json.tool by adding support for setting ensure_ascii and indent in the json.dump call.

Happy to also address any other arguments that need updating. Or other issues with json.tool.

@dhimmel

dhimmel commented Feb 20, 2017

Copy link
Copy Markdown
Contributor Author

Here a related issue from the issue tracker: Add an option to json.tool to bypass non-ASCII characters. This is my first time contributing to Python, so let me know if there's anything I need to do related to the issue tracker.

@methane

methane commented Feb 21, 2017

Copy link
Copy Markdown
Member

Sorry, please file an issue to bugs.python.org.
We use github only for code review, not for design discussion.

@dhimmel

dhimmel commented Feb 21, 2017

Copy link
Copy Markdown
Contributor Author

We use github only for code review, not for design discussion.

@methane no problem. I'll open an issue on bugs.python.org. Give me a couple days... currently traveling.

@dhimmel

dhimmel commented Feb 23, 2017

Copy link
Copy Markdown
Contributor Author

please file an issue to bugs.python.org.

@methane I opened bpo-29636 for design discussion of --indent. In addition, this PR would close the existing issue bpo-27413 regarding the --no-ensure-ascii argument.

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

The main problem -- output characters on the locale that doesn't support them.

@dhimmel dhimmel force-pushed the json_tool branch 2 times, most recently from e8a7cee to da24e13 Compare February 26, 2017 23:16
@dhimmel

dhimmel commented Feb 27, 2017

Copy link
Copy Markdown
Contributor Author

@serhiy-storchaka and @methane, thanks for the review thus far. It's been very helpful.

Builds are now passing.

The main problem -- output characters on the locale that doesn't support them.

I'm not sure whether I've addressed this. Could you be a little more specific on the locale issues? Is this just something that affects the tests?

31 hidden items Load more…
@serhiy-storchaka

serhiy-storchaka commented Feb 27, 2017

Copy link
Copy Markdown
Member

test_ensure_ascii is failed with non-UTF-8 encoding.

$ LC_ALL=en_US.iso88591 ./python -m test.regrtest -v -m test_ensure_ascii test_json
...
======================================================================
FAIL: test_ensure_ascii (test.test_json.test_tool.TestTool)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/test/test_json/test_tool.py", line 137, in test_ensure_ascii
    self.assertEqual(expect.splitlines(), json_stdout.splitlines())
AssertionError: Lists differ: [b'"\\u00a7 \\ud83d\\udc0d \\u03b4 \\ud834\\udc37"'] != [b'"\\u00c2\\u00a7 \\u00f0\\u009f\\u0090\\u008d \\[39 chars]b7"']

First differing element 0:
b'"\\u00a7 \\ud83d\\udc0d \\u03b4 \\ud834\\udc37"'
b'"\\u00c2\\u00a7 \\u00f0\\u009f\\u0090\\u008d \\[38 chars]0b7"'

- [b'"\\u00a7 \\ud83d\\udc0d \\u03b4 \\ud834\\udc37"']
+ [b'"\\u00c2\\u00a7 \\u00f0\\u009f\\u0090\\u008d \\u00ce\\u00b4 \\u00f0\\u009d'
+  b'\\u0080\\u00b7"']

----------------------------------------------------------------------

Other tests are passed by accident. Input data is encoded with UTF-8 and decoded with locale encoding in json.tool. Output data is encoded with locale encoding in json.tool and you get back the initial UTF-8 representation.

Try to pass escaped data not encodable with locale encoding.

$ echo '["\u20ac"]' | LC_ALL=en_US.iso88591 ./python -m json.tool --no-ensure-ascii
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/Lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/serhiy/py/cpython/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpython/Lib/json/tool.py", line 64, in <module>
    main()
  File "/home/serhiy/py/cpython/Lib/json/tool.py", line 58, in main
    sort_keys=options.sort_keys,
  File "/home/serhiy/py/cpython/Lib/json/__init__.py", line 180, in dump
    fp.write(chunk)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u20ac' in position 7: ordinal not in range(256)

@berkerpeksag

Copy link
Copy Markdown
Member

By the way, --no-ensure-ascii and --indent/--no-indent are two separate enhancements so there should be two pull requests.

@dhimmel dhimmel changed the title Increase the utility of json.tool Feb 27, 2017
dhimmel added a commit to dhimmel/cpython that referenced this pull request Feb 27, 2017
@dhimmel

dhimmel commented Mar 10, 2017

Copy link
Copy Markdown
Contributor Author

@berkerpeksag @serhiy-storchaka @methane: I'm excited to continue work on this pull request and now have a better grasp of the locale-dependent encoding issues that I still have to deal with. However, I was hoping that we could get the following PRs merged (in the following order since they're dependent):

  1. Fix stderr bug in json.tool test #346: fixing stderr bug in existing json.tool test
  2. bpo-29636: Add --indent / --no-indent arguments to json.tool #345: addition of indentation options to json.tool which affects the structure of the json.tool code.
  3. then it will be easier to work address the outstanding issues with this PR.

Of course review thoroughly... just that it seems like review momentum has stalled.

berkerpeksag pushed a commit that referenced this pull request Mar 15, 2017
berkerpeksag pushed a commit to berkerpeksag/cpython that referenced this pull request Mar 15, 2017
berkerpeksag added a commit that referenced this pull request Mar 16, 2017
dhimmel added a commit to dhimmel/cpython that referenced this pull request Jul 20, 2017
@brettcannon

Copy link
Copy Markdown
Member

To try and help move older pull requests forward, we are going through and backfilling 'awaiting' labels on pull requests that are lacking the label. Based on the current reviews, the best we can tell in an automated fashion is that a core developer requested changes to be made to this pull request.

If/when the requested changes have been made, please leave a comment that says, I have made the requested changes; please review again. That will trigger a bot to flag this pull request as ready for a follow-up review.

akruis added a commit to akruis/cpython that referenced this pull request Jan 6, 2019
…ptor"

objects, if soft-switching is enabled. The code has been in Stackless
forever, but was disabled for some unknown reason.
akruis pushed a commit to akruis/cpython that referenced this pull request Jan 12, 2019
…ptor"

objects, if soft-switching is enabled. The code has been in Stackless
forever, but was disabled for some unknown reason.
@brettcannon brettcannon added the type-feature A feature request or enhancement label Aug 22, 2019
@txtsd

txtsd commented Dec 1, 2019

Copy link
Copy Markdown

Any chance of this getting merged?

@dhimmel

dhimmel commented Dec 5, 2019

Copy link
Copy Markdown
Contributor Author

Closing in favor of #17472.

@dhimmel dhimmel closed this Dec 5, 2019
jaraco pushed a commit that referenced this pull request Dec 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting changes type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants