Is there a way to avoid byte-compiling at compile time that I've missed or do you just discard the .o files?
AFAIK there is no byte-compiling to avoid when you compile normally.
Only "make verify" byte-compiles everything because it performs an install operation. This happens in a special directory test-install though, so you never get byte-compiled files in any of the dirs used by "make export" or "make bin_export".