... see no reason to have to download and clutter their installation with modules they'll never use.

That's another matter. There are already the --without-foo flags, but if someone put some research into it there could be a --small-footprint-build flag that is shorhand for several --without-foo.