To be a little clearer: Since tok->filename is used as a flag in other
places, I'm not sure where to set it without breaking other things.
This is the location of the segfault and the following would "fix" it
(using a placeholder for the name):
Index: Parser/tokenizer.c
===================================================================
--- Parser/tokenizer.c (revision 83019)
+++ Parser/tokenizer.c (working copy)
@@ -582,6 +582,8 @@
if (badchar) {
/* Need to add 1 to the line number, since this line
has not been counted, yet. */
+ if (tok->filename == NULL)
+ tok->filename = "<file>";
PyErr_Format(PyExc_SyntaxError,
"Non-UTF-8 code starting with '\\x%.2x' "
"in file %.200s on line %i, "