Issue 26292: Raw I/O writelines() broken for non-blocking I/O

Copy of Antoine Pitrou's email (sent in 2012 ;-):
https://mail.python.org/pipermail/python-dev/2012-August/121396.html

Hello,

I was considering a FileIO.writelines() implementation based on
writev() and I noticed that the current RawIO.writelines()
implementation is broken: RawIO.write() can return a partial write but
writelines() ignores the result and happily proceeds to the next
iterator item (and None is returned at the end).

(it's probably broken with non-blocking streams too, for the same
reason)

In the spirit of RawIO.write(), I think RawIO.writelines() could return
the number of bytes written (allowing for partial writes).

Regards

Antoine.

-- 
Software development and contracting: http://pro.pitrou.net

"(it's probably broken with non-blocking streams too, for the same
reason)"

Yes it is :-)

We hitted this issue on eventlet when I changed the socket.send() method in eventlet 0.18 to stop sending data on partial write, whereas eventlet < 0.18 used a loop to ensure that all bytes are written.

The side effect of my change is that (indirectly) socket.makefile().writelines() may also use partial write if the underlying send() only writes partially data.

The problem is that writelines() has *no* result, so the caller cannot be aware of the partial write and data is lost...

eventlet:

* send() change: https://github.com/eventlet/eventlet/issues/274
* writelines() bug: https://github.com/eventlet/eventlet/issues/295


"In the spirit of RawIO.write(), I think RawIO.writelines() could return
the number of bytes written (allowing for partial writes)."

I don't know yet what is the best option for eventlet, but we should probaly enhance the Python stdlib io module to return something on writelines() and *document* the behaviour on partial write.

The problem writelines() API is that writelines() not only takes a single text/bytes string, but a *list* of text/bytes strings.

Another option is to modify to retry on partial write to ensure that all data is written. If the underlying write() method returns None (blocking I/O error), we must raise an error. It doesn't make sense to retry writing on a non-blocking I/O error.

In this case, we must document that writelines() *must not* be used on non-blocking I/O (ex: non-blocking socket, pipe, whatever).

Is deprecating RawIOBase.writelines() an option, and only recommending BufferedIOBase.writelines() and TextIOBase.writelines()?

Otherwise, I think it would make most sense to keep retrying until all the data is written. This mirrors how I understand readline() and readall() work (keeps reading until it gets as much as necessary).

For non-blocking mode, readline() does not support that (see Issue 13858). It does not make much sense to me to have writelines() support non-blocking mode either.

> Is deprecating RawIOBase.writelines() an option, and only recommending BufferedIOBase.writelines() and TextIOBase.writelines()?

IMHO the problem is the same for other classes.

For BufferedIOBase, the documentation says write() will not return a short number of bytes, so I don’t see how writelines() is a problem there. For TextIOBase, the documentation is not so clear <https://docs.python.org/release/3.4.3/library/io.html#io.TextIOBase.write>, but I understand the standard library implementations do not do short writes.

Victor, why did you change the title to specify non-blocking mode? I think blocking mode can also be handled at the same time.

I propose:

1. In existing versions (2.7, 3.5): Document that it is undefined what IOBase.writelines() will do if a write() call does a partial write, returns None, or encounters a blocking error. Explicitly document that BufferedIOBase.writelines() and TextIOBase.writelines() will completely write all the data passed, or raise an exception. Document that BlockingIOError.characters_written is undefined even for BufferedIOBase.writelines().

2. Commit my Issue 26721 change to socketserver, so that StreamRequestHandler.wfile implements BufferedIOBase instead of RawIOBase.

3. In a new version (3.6): Deprecate IOBase.writelines() and thus RawIOBase.writelines(), in favour of either using BufferedIOBase, or manually calling write(). Emit a DeprecationWarning, but add BufferedIOBase and TextIOBase implementations that do not emit the warning.

BufferedIOBase.writelines() could be fixed to report the correct BlockingIOError.characters_written value, but that could be handled as a separate bug if anyone cares.

Here is a patch documenting that RawIOBase.writelines() is undefined for partial writes and blocking errors, as mentioned in (1) above.

The alternative of requiring write() to be retried would IMO be unfair to existing writelines() implementations. It also seems out of place considering we don’t even have a RawIOBase.writeexactly() type of method.

I close this issue as duplicate of bpo-13322 which is older and has a longer history.

resolution: duplicate
stage: resolved

versions: + Python 3.5