This is similar to #13745 but the raised the changes to idlelib folder as a separate PR to assist backporting workflow for idlelib.

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

I am not sure why code comments changes cause test failure and especially it happens in test_asyncio and not idlelib related test failure. I have rebased and retriggered the run.

Thanks @tirkarthi for the PR, and @terryjreedy for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

GH-13770 is a backport of this pull request to the 3.7 branch.

Since new docs are added everyday I just try to run the spell check every 2-3 months. I use aspell and it reports lot of false positives like variable names, function names, tests etc. For Docs folder it reports 10k words and Lib folder 30k words as typos. Hence manual skimming is required and sometimes I overlook the typo to find it next time.

Are you reducing .py files to strings and comments first?

aspell takes the input and splits it into words based on word boundary I think to get all words from the source. Below is the command I use

aspell list < Lib/idlelib/**/*py | tr '[:upper:]' '[:lower:]' | sort -u | less

The following, using the regexes in idlelib.colorizer, strips a file down to comments and strings. If you could feed the output for all (idlelib or other stdlib) py files to aspell, there would be less to wade through, as the result have only a small fraction of the variable names. (Since I am on Windows and don't know the aspell API, I cannot be more specific.)

import re

comment = r"#[^\n]*"
stringprefix = r"(?i:r|u|f|fr|rf|b|br|rb)?"
sqstring = stringprefix + r"'[^'\\\n]*(\\.[^'\\\n]*)*'?"
dqstring = stringprefix + r'"[^"\\\n]*(\\.[^"\\\n]*)*"?'
sq3string = stringprefix + r"'''[^'\\]*((\\.|'(?!''))[^'\\]*)*(''')?"
dq3string = stringprefix + r'"""[^"\\]*((\\.|"(?!""))[^"\\]*)*(""")?'
comstring = "|".join([comment, sq3string, dq3string, sqstring, dqstring])

pattern = re.compile(comstring, re.S)
with open("c:/Programs/Python38/Lib/idlelib/autocomplete.py") as f:
    for m in pattern.finditer(f.read()):
        print(m.group(0))

Hi @tirkarthi

Just came up with the idea.
Is it possible that we can create a typo checker (bot) ?

Like the-knights-who-say-ni or bedevere-bot,
when receive a PR, the bot automatically checks the typo.

More details.

Every time when the typo checker finds a typo, it warns the PR submitter.
PR submitter can fix the typo based on the report.

It is okay, if the typos were found and submitter doesn't fix it.
Since it is probably intended to spell the word that way.

I had that in the back of my mind when I posted the code above. However, the problem is that even after reducing the code checked, there will still be mostly false positives, and for many modules, there will always be false positives. @tirkarthi could get some numbers after interfacing the filter with aspell. And of course, the bot would have to have aspell available. The report could be made optional with a spellcheck label.

More details.

Every time when the typo checker finds a typo, it warns the PR submitter.
PR submitter can fix the typo based on the report.

It is okay, if the typos were found and submitter doesn't fix it.
Since it is probably intended to spell the word that way.

I also think it would be great to add this feature to the CI test. For example, trailing whitespace testing.

I share the same concern about high number of false positives. I mostly took the code from the rust-lang book where there is a separate dictionary.txt file with list of false positives and this shell script to check spelling. It can work at the scale of book but not sure about Python docs, code level docstrings and comments that can have lot of variables, function names and so on that there is a higher rate of changes to keep updating dictionary.txt.

Hi @terryjreedy and @tirkarthi

I summarized the discussion above.

Consensus:

• Reducing typo in a constructive way.

To be Solved:

• Even after reducing the code checked, there will still be mostly false positives.
• High frequency of updating dictionary.txt (false positive typo).

Next Step:

• Interfacing the filter with aspell to get some numbers.

MISC:

• The report could be made optional with a spellcheck label.
• Code for reducing .py files to strings and comments.
• Command line for spellcheck.

Is @tirkarthi currently working on interfacing the filter?
I can help otherwise :D

No, I am not working on the filters data or improving aspell side and manual check for every 2-3 months seems good. Feel free to continue with the approach and I may not have time to continue with this.

Thanks

Further general discussion should be on the core-workflow list. A specific proposal for git should be an issue for the core-workflow repository, which I believe is where bots originate.

Thanks @terryjreedy , I'll move the discussion to core-workflow repository. :D