Both can be true. "Fit" certainly is used to exclude all kinds of groups. That's not good & hurts the teams that do it.

On the other hand, a bad fit on a team can lead to poor collaboration and morale. If that's possible to avoid, teams will do it.