Because before you can think about a union, there has to be a bargaining unit. And members of the bargaining unit have to have a community of interests.
In a widely anticipated decision, the National Labor Relations Board has reversed its 2004 decision in Oakwood Care Center,1 and determined that a union seeking to represent employees in a bargaining unit composed of employees solely employed by a “user employer” (a company that hires temporary...
www.littler.com
"Those “community of interest” factors include criteria such as: common functions and duties, shared skills, functional integration, interchange, frequency of contact with other employees, commonality of wages, hours, and other working conditions, permanent transfers, shared supervision, common work location, and bargaining history."
I would argue that swimmers, football players, baseball players and gymnasts have almost no commonality of interests as so defined.