I dug around on the internet for a while and skimmed Mark Winston's chapter on drones and mating in "The Biology of the Honey Bee", and I found no references to this concept of unsuccessful drones not being admitted solely on the grounds that they have failed to mate. In fact I found several references on the internet to the contrary, and this sentence from Winston's book also indicates the opposite: "During sunny weather drones take several flights a day, averaging three to five flights an afternoon and returning to the nest for about 15 min between flights to consume additional honey for energy."
Also, not to say that nature always makes 100% logical sense (crawlybacks for example), but I don't know why the workers should care that the drones didn't mate. Like, they have the queen make hundreds of them for the express reason that most will not be successful. And on top of that, by not allowing the drones to reenter the hive for shelter and food, they actually REDUCE their chances of success. Wally's point is also really good.