The solution of adding caching, If-Modified-Since, etc. is a good one,
but I quail in fear at the prospect of expanding the saxutils resolver
into a fully caching HTML agent that uses a cache across processes. We
should really be encouraging people to use more capable libraries such
as httplib2 (http://code.google.com/p/httplib2/), but this is slightly
at war
with the batteries-included philosophy.
So, I propose we:
* add warnings to the urllib, urllib2, saxutil module docs that parsing
can retrieve arbitrary resources over the network, and encourage the
user to use a smarter library such as httplib2.
* update the urllib2 HOWTO to mention this.
I'm willing to do the necessary writing.