Avoiding power-of-two is something wikipedia ("Hash table") comments on:
Some older hashes are even worse, requiring table sizes to be a prime number rather than a power of two, again computing the bucket index as the hash value modulo the table size. In general, such a requirement is a sign of a fundamentally weak function; using a prime table size is a poor substitute for using a stronger function.
Add to that that prime numbers mean modulo functions which are very expensive on modern CPUs. If I recall correctly the cycle count on x86_64 is in the hundreds which obviously cannot compare to masking and shifting.