FIX: guard against integer overflow in FT2Image dimensions by arshsmith · Pull Request #31860 · matplotlib/matplotlib

FT2Image(width, height) sized its buffer with calloc(width * height, 1),
where width and height are unsigned long. The multiplication could
overflow (e.g. 2**16 * 2**16 on a 32-bit long, or 2**32 * 2**32 on a
64-bit long), making calloc allocate a buffer far smaller than
m_width * m_height while the stored dimensions stayed huge.
draw_rect_filled then clamps to those stored dimensions and writes past
the end of the under-sized buffer -- a heap buffer overflow. The calloc
return value was also never checked for NULL.

Reject dimensions whose product overflows unsigned long (raising
OverflowError) and raise on allocation failure (MemoryError). Normal
dimensions are unaffected. FT2Image is no longer used on any internal
rendering path, so this only hardens the public (deprecated) class.