◐ Shell
reader mode source ↗
Skip to content

bpo-28053: Allow custom reducer when using multiprocessing#15058

Open
pierreglaser wants to merge 32 commits into
python:mainfrom
pierreglaser:inject-custom-context-into-process-cls
Open

bpo-28053: Allow custom reducer when using multiprocessing#15058
pierreglaser wants to merge 32 commits into
python:mainfrom
pierreglaser:inject-custom-context-into-process-cls

Conversation

@pierreglaser

@pierreglaser pierreglaser commented Jul 31, 2019

Copy link
Copy Markdown
Contributor

This PR is a follow up on #9959. It implements the suggestions I made here #9959 (comment).

I'm making it a new PR to get the CI working now and gather separate feedback.

  • pass the current context to Process objects
  • split AbstractPickler into AbstractPickler and AbstractUnpickler
  • clean everything up (still WIP)

https://bugs.python.org/issue28053

@pierreglaser

Copy link
Copy Markdown
Contributor Author

You will notice that for the Process class, I chose to pass the reducer (which is the only bit of information we need from the context for now) to the Process class at construction instead of the Context

    def process(self, *args, **kwargs):
        return self._Process(*args, reducer=self.get_reducer(), **kwargs)

This comes from the fact that theProcess is actually relying on the default context to be properly defined:

class Process(process.BaseProcess):
_start_method = None
@staticmethod
def _Popen(process_obj):
return _default_context.get_context().Process._Popen(process_obj)

For the process class, passing the context object a Process bind to would require changing it to something like:

class Process(process.BaseProcess):
    _start_method = None
    @staticmethod
    def _Popen(process_obj):
          if self._ctx is not None:
              return self._ctx.get_context()._Process._Popen(process_obj)
          return _default_context.get_context()._Process._Popen(process_obj)

with _Process being a Process class specified (or not, in this case the default is Process) by the user. You can see the infinite recursion here as if _Process is not specified or set to Process, self._ctx.get_context()._Process._Popen is Process._Popen... To prevent this situation we would need to add a bunch of guards, and this ends up being not so trivial.

So implementation-wise, it is easier to simply pass the reducer in BaseContext.process, and not the whole context. Also, one noticeable consequence is that the reducer in Process is static, as it is not called using something like self._ctx.get_reducer() as it is done elsewhere.

But I am opened to suggestions.

@pierreglaser

Copy link
Copy Markdown
Contributor Author

@pitrou what could I do to ease the review process of this PR?

@pitrou

pitrou commented Aug 7, 2019

Copy link
Copy Markdown
Member

@pierreglaser The first necessary condition is for me to find some time... sorry :-/

@pierreglaser

Copy link
Copy Markdown
Contributor Author

@pitrou no worries :)

68 hidden items Load more…
@pierreglaser

pierreglaser commented Feb 2, 2020

Copy link
Copy Markdown
Contributor Author

@pablogsal friendly ping: I am curious to know what are your thoughts on this.

@terryjreedy

Copy link
Copy Markdown
Member

(sorry, failed rebasing attempt)

Rebasing is, as evidenced by repeated failures like this, error prone. Besides the obvious noise, removing the spurious review requests does not unsubscribe people.

If you were just trying to update your branch to your current repository/fork master (once synchronized), rebasing is also not needed. After checking out the branch, git pull origin/master will, if there are no conflicts, pull, merge, and commit updates to the branch with a standard commit message.

@tiran tiran removed their request for review April 17, 2021 21:05
@rhettinger rhettinger removed their request for review May 3, 2022 06:31

@MaxwellDupre MaxwellDupre left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Run multiprocessing tests:
Ran 369 tests in 215.720s
OK (skipped=35)
Ran 39 tests in 22.709s
OK
Ran 369 tests in 85.400s
OK (skipped=39)
Ran 369 tests in 99.032s
OK (skipped=32)
Look ok to me.

@github-actions

github-actions Bot commented Apr 7, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 30 days with no activity.

@github-actions github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting core review stale Stale PR or inactive for long period of time. topic-multiprocessing

Projects

None yet

Development

Successfully merging this pull request may close these issues.