back to your original question: is letting a swarm go bad? It will happen one day no matter what you do. I use pretty much every trick in the book and I've had one hive swarm in five years. That one swarmed even though I took the old queen away and left them only 3 or 4 Q. cells. They just really felt like it
However, to make no effort at swarm prevention is IMO irresponsible. Bees are "livestock" introduced from europe and don't belong in the wild. Also one of your neighbors could face a pretty hefty bill for having bees removed from their wall.