If a guy asks you if you want to hang out, you probably understand that he's not looking for friends. Odds are he has enough of these. Rather, he wants to date you. If you're not attracted to him, you should probably avoid hanging out with him because you will give him the wrong idea. However, there's really no need to crush his self-esteem by telling him you don't like the cut of his jib. Rather, you should make an excuse that sounds reasonable, but also makes it clear that you are not available.
Tell a guy you can't make it because you're hanging out with your boyfriend that day/night. This is helpful whether you have a boyfriend or not, because it is immediately taking you "off the market." It makes it clear that you are not interested in hanging out with this guy romantically because you are currently dating someone. This is a good way to tell him you want to be friends, too -- if you say, "We can still hang out, though," you are making clear what kind of hangout the guy can expect. If he calls you after this, he'll know he's making a new friend, not dating a new girlfriend.
Tell a guy that you're sick. This is good because it lets him down without taking you "off the market." If, for example, you like the guy but don't want to date anyone right now, telling him you're sick is just putting him on hold -- you're not telling him you're not attracted to him, but you are telling him you can't see him right now.
Tell a guy that you have to work late or work on an assignment if you are a student. This is good for the same reason as the sickness excuse. It lets you put off hanging out with him right now with a believable excuse, but it also lets you see him in the future. It's a little more definite than sickness, though. Most guys know that if you really like them you'll put an enormous effort into seeing them, even if this means clocking off work at the usual time and coming in on the weekend. So, by using this excuse you're showing your priorities and giving a clear message.
The key to excuses is that you use them often. If you tell a guy that you're sick, then have to work late, then have to wash your hair, he will get the message. After all, you may be sick once, but the odds of you being sick and then having other things pop up are extremely low. So, the real key to not hanging out with a guy is to make a number of seemingly benign excuses. The aggregate will give him the message without hurting his feelings.