
I get this question a lot of times. Here is just some off the top of my head:
1. Stick with the STABLE version which at the time of my post is 1.18
2. Do not over install mods, especially unsupported mods that may be a security risk later
3. Ensure your permissions to Guests are VERY limited and your Members not beyond what would give them a chance to cause havoc.
4. Choose your Moderators wisely
5. Clear logs, ensure you only have the necessary files writeable
6. Change your password regularly.
I'll defer from JB on this one: Personally, I think that unstable is okay, but don't get any of the early betas. For example, if I were to be setting up a forum right now, I'd be using SMF 2.0 RC 1.2 as opposed to 1.1.10. I've spent some time on the beta team, and those guys work hard to clear the bugs out... By some standards, 2.0 was stable when it first hit public beta.
I also would like to warn against using too many mods. They can present security holes (because they are not checked nearly as well as the forum software itself), and if you have too many of them, can cause issues when installing security patches.