Issue 23757: tuple function gives wrong answer when called on list subclass with custom __iter__

Converting a list to a tuple appears to have an optimisation that is wrong in the presence of subclassing to override __iter__. It ignores the user defined iter and uses the normal list one. I've attached a file with a test case to demonstrate this.

I've verified this on both python 2.7 and python 3.4. It's presumably also the case on everything in between.

This was found because it caused a bug with a type that pytz uses, which lazily populates the list on iteration: https://bugs.launchpad.net/pytz/+bug/1435617

This is a specific instance of the general problem covered by issue 10977.

Ah, I hadn't seen that. Thanks for the link.

But... is it really? They have basically the same root cause, but the general problem seems to be hard to fix, while the specific problem here seems to be basically "don't use the concrete API here because it breaks things".

Right, but unless I miss my guess (I haven't looked at the code) it isn't that the concrete api is being used by the constructor, it's that the concrete API is being used by the iterator protocol called by the constructor.  If I'm wrong it would be sensible to fix the constructor and we can reopen this.  Even if I'm right perhaps there's a way to fix the constructor, but in that case it would be best addressed in the context of that issue.

So as a data point, this problem seems to be unique to tuple. set(x), list(x), tuple(iter(x)) all seem to work as expected and respect the overridden __iter__ (set and list were both included in the test case I attached to demonstrated this, iter I just checked right now). This suggests that the problem is in what tuple is doing, not in some general iterator protocol,

Can we get this reopened? As David MacIver points out, this seems entirely a wart in tuple's constructor (compared to all the other builtin types), whereas 10977 is worrying about how 3rd party code using the C API can corrupt subclasses of builtin types (a much larger scope, and much less likely to be resolved in a good way).

Does it make sense to require python code wishing to case a tuple or tuple subclass do so using tuple(list(o)), or should tuple(o) work as expected? The primary use is of course converting a mutable sequence to an immutable representation to use as a dict key.

Added a patch.  Needs to have the OP's test case added.

And it would be nice to add the same test for list, set, etc (if they don't exist).

Raymond's patch LGTM. Here is updated patch with tests.

New changeset b6121a4afad7 by Raymond Hettinger in branch '2.7':
Issue #23757:  Only call the concrete list API for exact lists.
https://hg.python.org/cpython/rev/b6121a4afad7

New changeset c79530e08985 by Raymond Hettinger in branch '3.4':
Issue #23757:  Only call the concrete list API for exact lists.
https://hg.python.org/cpython/rev/c79530e08985

assignee: rhettinger
keywords: + patch
stage: needs patch -> commit review

superseder: Concrete object C API considered harmful to subclasses of builtin types

nosy: + r.david.murray
messages: + msg239138
resolution: duplicate
stage: resolved