I wouldn't be too worried about it as long as you keep your packages constantly up to date to protect against security flaws. This webserver has been graciously spared due to this strategy. Use unguessable passwords and also backup frequently so that you can recover data if you ever get hacked.
I use IP address blocking sparingly, and only for IP addresses that are causing the site to slow down dramatically. Many ISPs hide many users behind single IP addresses, so you always run the risk of blocking legitimate users of your site.