Bees are a society, and when they swarm, they make a decision that is for the benefit of the collective – and as always happens with collective decision making, there are always some people that get left out.
The bees will leave, hang on a tree or in a bush for an hour or so and then dejectedly return to the hive where the queen is. This is not a given, but it can happen.