People enjoy the adrenaline rush of being scared. Similar to rollercoasters
People may also not enjoy actually being scared, but enjoy the feeling of relief when the threat is removed.
That may explain the 'bump in the night' kind of film but I've no idea why people would want to watch the new breed of torture movie - Saw, Hostel, Wolf Creek etc.
I guess it's just morbid curiosity, seeing what sick ideas some people can come up with.