When it's a big server, I'd understand your point but I only run a small RP server with my friends so it's not so much of an issue.
In fact your server being small is why you should ban only for good reasons, whereas big servers are the ones that can afford banning players less carefully because they still have lots of them.