It's contrast that's the key.
It always is, but with scaring the hell out of people it's the contrast that gets people.
I'm a sucker for horror, I love it, but I really am a sucker for horror. I can't let it go. It creeps inside my spine and it stays there. I have to limit my dosages for this genre because even poorly done horror freaks me the hell out.
One of the more interesting things I've learned about this recently was from Bioshock. Originally, the enemies in Bioshock were going to be these sort of alien lizard things. I've seen the concept art, and they're definitely monstrous.
They're not scary in the least.
In contrast, the enemies that they actually did implement were decidedly unsettling. The "splicers" are normal people who are suffering from what amounts to horrific and artificial cancer; cancer that turns you into a monster while giving you super powers.
But they're not weird bug creatures, or lizards, they're sick people. They don't attack you with a mindless growl, they attack you while shouting "I just want someone to talk to!"
And they're motivations are believable. How many of us (and I'm talking specifically about the dorkosphere here) could resist telekinesis? How many of us could not take that offer, even if it came with a huge price-tag?
That's scary. The monster is us.
Stephen King wrote a good book on this subject called Danse Macabre, where he correctly identifies that horror is the intrusion of the abnormal into the normal. Weird things, in weird places, doing unusual things... are incomprehensible. They're not scary, they're just bizarre.
Change one or two of those things: place, people, or action into something grotesque and you get horror.
Normal Place, Normal People, Abnormal Action:
This option is arguably the scariest. The 'horror' is entirely hidden. This is the invasion of the body snatchers. You look outside and you see people who are just doing something -wrong-.
Zombies and other 'madness' things could be considered to fall into this category... at least at the beginning.
Normal Place, Abnormal People, Normal Action:
This is very subtle and not normally explored, but it does show up typically as a -break- from other forms of horror. The scene that pops into my head is the scene in the new version of the war of the worlds where a hidden Tom Cruise and company watch as a trio of aliens fiddle around with some garage junk in their bombed out basement refuge.
The aliens are just sort of playing around, but they're so strange looking, and they represent such a great threat, that their normal actions, and normal scenery make them chilling.
Abnormal Place, Normal People, Normal Action:
This type of horror is all atmospheric, and is typically used to build tension. This is where your heroes descend into the monsters lair and it just keeps going. They keep climbing through the strange and bizarre environment, but they keep not finding the monster. The bizarre nature of the environment keeps building.
The best use of this is when you throw the monster at them JUST when they finally decide that they're safe. This is easy to do in film, hard to do in print, and punishingly difficult to pull off at a table. If you've pulled this one off tell me about it. I want to hear how.
It can also work like this:
If you've successfully had your character infiltrate the fortress built within the corpse of the necromantically animated dead god, fought your way through a thousand putrescent horrors... and then, suddenly, the smell of roses.
You turn the corner and you find yourself facing a tiny elegantly carved door, through which is a small, perfect, room, the room of a child, decorated in bright pastels.
The child's name is Sally. She's not a monster. She's a normal kid. She's six. She -likes- it here.
Ta da... horror.
Normal Place, Abnormal People, Abnormal Action:
This is the 'monster invades suburbia' scenario. A monster is scary when it's in its lair. It's way MORE scary when it's standing in your living room.
Abnormal Place, Abnormal People, Normal Action:
You sneak into the secret den of the devouring rat people and you shimmy through their fithils and finally you find yourself gazing in horror into the pit where they're.... doing laundry.
This one can easily devolve into comedy (even faster then other forms of horror), but it can be useful to inject some humanity into a previously completely incomprehensible foe.
This makes it all the worse when the washer-lady see's you and tries to eat your face.
Abnormal Place, Normal People, Abnormal Action
I can't think up a good example for this one. This is starting to run into the "too many weird’s" problem.
I might be inclined to include the new (and awesome) game Dead Space in this category. You're running around a decaying spaceship, dismembering zombie-aliens... but you're effectively a repair-man.
I originally discounted this example because you're character is basically a non-entity. He's just the vessel for the player to experience this horrific situation through... which is perfect: the player the normal person who is forced into an abnormal environment, and asked to do unfamiliar (and disgusting) tasks.
The entire 'survival horror' genre could be put in this category.
Pacing is also incredibly important. It's extremely difficult to keep up the horror ALL the time. You're players will get fatigued. They're get silly. You need to give them a rest. Unremitting horror sounds fun, but doesn't work.
For extra points you provide the players with a safe area, a place where the horror doesn't happen. A place they can blow off steam a little, and then you do nothing with it. You let them play.
And let them play.
And let them play.
And you reinforce that this place is safe. Every time they come back there it's safe. Nothing lurking in the dark.
You keep doing this until they -players- really believe it's safe.
Then you screw with them. Just once, just a -little-, and let them fix the problem almost immediately.
Then let it be safe again.
And then when they've almost forgotten the previous incident... THEN you pull out your big guns.
... and don't be afraid of playing dirty. Horror is hard to pull off. It's SO easy for horror to become comedy. Don't feel bad for cheating. Use the music, lower the lights, and use static on radios. Use the tricks because you're trying to do something hard.
And when you DO get your groove on, and you really have your players invested in the scare... please remember people like me, people who scare easy, and understand when they say "That's just about enough for me tonight guys. Can we play Scrabble now?"