◐ Shell
reader mode source ↗
Skip to content

bpo-45434: Mark the PyTokenizer C API as private#28924

Merged
vstinner merged 1 commit into
python:mainfrom
vstinner:private_tokenize
Oct 13, 2021
Merged

bpo-45434: Mark the PyTokenizer C API as private#28924
vstinner merged 1 commit into
python:mainfrom
vstinner:private_tokenize

Conversation

@vstinner

@vstinner vstinner commented Oct 13, 2021 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Member

Rename PyTokenize functions to mark them as private:

  • PyTokenizer_FindEncodingFilename() => _PyTokenizer_FindEncodingFilename()
  • PyTokenizer_FromString() => _PyTokenizer_FromString()
  • PyTokenizer_FromFile() => _PyTokenizer_FromFile()
  • PyTokenizer_FromUTF8() => _PyTokenizer_FromUTF8()
  • PyTokenizer_Free() => _PyTokenizer_Free()
  • PyTokenizer_Get() => _PyTokenizer_Get()

Remove the unused PyTokenizer_FindEncoding() function.

import.c: remove unused #include "errcode.h".

https://bugs.python.org/issue45434

@vstinner
bpo-45434: Mark the PyTokenizer C API as private …
9d1485e 
Rename PyTokenize functions to mark them as private:

* PyTokenizer_FindEncodingFilename() => _PyTokenizer_FindEncodingFilename()
* PyTokenizer_FromString() => _PyTokenizer_FromString()
* PyTokenizer_FromFile() => _PyTokenizer_FromFile()
* PyTokenizer_FromUTF8() => _PyTokenizer_FromUTF8()
* PyTokenizer_Free() => _PyTokenizer_Free()
* PyTokenizer_Get() => _PyTokenizer_Get()

Remove the unused PyTokenizer_FindEncoding() function.

import.c: remove unused #include "errcode.h".
@vstinner

vstinner commented Oct 13, 2021

Copy link
Copy Markdown
Member Author

@pablogsal @isidentical @lysnikolaou: Do you see any reason to keep the PyTokenizer as public?

My plan is to move E_xxx constants from Include/errcode.h to Include/internal/pycore_parser.h, and then remove the Include/errcode.h header filer.

@isidentical

isidentical commented Oct 13, 2021

Copy link
Copy Markdown
Member

Do you see any reason to keep the PyTokenizer as public?

I don't see any reason for external apps to depend on these methods, especially when we already have a pure Python version.

@vstinner vstinner merged commit 713bb19 into python:main Oct 13, 2021
@vstinner vstinner deleted the private_tokenize branch October 13, 2021 15:22
@vstinner

vstinner commented Oct 13, 2021

Copy link
Copy Markdown
Member Author

My plan is to move E_xxx constants from Include/errcode.h to Include/internal/pycore_parser.h, and then remove the Include/errcode.h header filer.

In fact, it's not possible without breaking the backward compatibility: https://bugs.python.org/issue45434#msg403841

@vstinner

vstinner commented Oct 13, 2021 •
edited
Loading

Copy link
Copy Markdown
Member Author

But marking the tokenizer C API as private is still worth it ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal pablogsal approved these changes

@isidentical isidentical isidentical approved these changes

@brettcannon brettcannon Awaiting requested review from brettcannon brettcannon is a code owner

@encukou encukou Awaiting requested review from encukou

@ericsnowcurrently ericsnowcurrently Awaiting requested review from ericsnowcurrently ericsnowcurrently is a code owner

@lysnikolaou lysnikolaou Awaiting requested review from lysnikolaou

@ncoghlan ncoghlan Awaiting requested review from ncoghlan ncoghlan is a code owner

@warsaw warsaw Awaiting requested review from warsaw

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@vstinner @isidentical @pablogsal @the-knights-who-say-ni @bedevere-bot