I did some digging on my side, the fact you see ncurses referenced from readline is due to the build linking readline to libtermcap:
cc -fstack-protector -shared -Wl,-x -o libreadline.so.8 -Wl,-soname,libreadline.so.8 `lorder readline.So vi_mode.So funmap.So keymaps.So parens.So search.So rltty.So complete.So bind.So isearch.So display.So signals.So util.So kill.So undo.So macro.So input.So callback.So terminal.So text.So nls.So misc.So compat.So xmalloc.So history.So histexpand.So histfile.So histsearch.So shell.So mbutil.So tilde.So | tsort -q` -ltermcap
And libtermcap is:
% ll /usr/lib/libtermcap.so*
0 lrwxr-xr-x 1 root wheel - 13B 18 apr 08:29 /usr/lib/libtermcap.so@ -> libncurses.so
That configuration option you referenced, Stefan, is that --with-termlib (generate separate terminfo library)?