We have masonry bees and carpenter bees. They are not generally aggressive and we have learned to live with them rather than kill them. They can be great pollinators and with the decline in honeybee populations, we need more pollinators!
We did get rid of a wasp nest last year as soon as we saw it under construction. They are more aggressive. So I think it depends on (1) the type of bees you have - or if they are wasps, hornets, etc., - or if you have found a large nest. Also (2) whether anyone in your family is very allergic to stings. If so, then you may have to take more aggressive action. My sister is very allergic and so my parents were pretty vigilant about getting rid of bees, etc.