◐ Shell
clean mode source ↗

Issue 37709: CSVReader ignores dialect.lineterminator

This is tested under python 3...

filename = "csv_Sample.csv"
from csv import DictReader
datafile = open(filename, 'r')
data = csv.DictReader(datafile, lineterminator = '\x07', delimiter='\x06')
print(next(data))
    OrderedDict([('Field1', 'A'), ('Field2', 'B'), ('Field3', 'C'), ('Field4', 'D'), ('Field5', 'E'), ('Field6', 'F'), ('Field7', 'G'), ('Field8', 'H'), ('Field9', 'I'), ('Field10\x07', 'J\x07')])
print(ord(data.reader.dialect.lineterminator))

So it's untested under python 2, since I've stopped developing under Py2.  

I noticed the note in the CSV reader documentation, *AFTER* I diagnosed the issue with the CSV reader...  Which is why I opened the bug / feature enhancement request, since this is an very odd edge case.

I agree 90+% of all CSVs are going to be \n line terminated, but if we offer it for writing, we should offer it for reading.

The main emphasis here is this code will not working in the real world, eg.

filename = "csvFile.csv"
from csv import DictReader, DictWriter
import csv
with open(filename, mode='w') as output_file:
    outcsv = csv.writer(output_file, delimiter=',', lineterminator=";")
    outcsv.writerow(['John Cleese', 'CEO', 'March'])
    outcsv.writerow(['Graham Chapman', 'CFO', 'November'])
    outcsv.writerow(['Terry Jones', 'Animation', 'March'])
    outcsv.writerow(['Eric Idle', 'Laugh Track', 'November'])
    outcsv.writerow(['Michael Palin', 'Snake Wrangler', 'March'])

with open(filename, mode='r') as input_file:
    csv_reader = csv.reader(input_file, delimiter=',', lineterminator=";")
    for row in csv_reader:
        print(row)
Looking at your sample file, it seems stranger than you first indicated. Your line terminator actually appears to be '\x07\r\n', not just '\x07'.  Opening your file in text mode will leave you with '\x07' as the last character of the last cell in each row. I've attached two files, bell.csv, which has just '\x07' as the line terminator, and lfmapper.py, which provides a class (suboptimally named LFMapper) which takes a file object opened in binary mode and optional line_terminator and encoding args, and performs the necessary slicing of the input bytes, decoding them and returning strings.

Unless Python grows a way for you to tell the open() function what string to use as the line terminator in text mode, I don't think your example is ever going to work without some sort of shim class.