Well, it is clear that neither are to be recommended if you had a choice! Obviously, the response to this question is going to be very subjective, as both are clearly presented in terms that demonstrate their terror and horror, but for me, the pendulum is a far worse method of death than the pit. The pit, of course, was a hidden danger - it was only by accident that the narrator tripped and discovered it. Yet the pendulum is a torture-device beyond compare because the victim is aware of its every movement as it descends bit by bit with the aim of slicing its victim into two. Consider how the narrator describes the terror he feels when he has worked out the situation that he is in:
What boots it to tell of the long, long hours of horror more than mortal, during which I counted the rushing vibrations of the steel! Inch by inch - line by line - with a descent only appreciable at intervals that seemed ages - down and still down it came! Days passed - it might have been that many days passed - ere it swept so closely over me as to fan me with its acrid breath. The odour of the sharp steel forced itself into my nostrils. I prayed - I wearied heaven with my prayer for its more speedy descent.
Clearly, this method of death is so much more horrendous than the pit because of the madness that it causes in its victim as they are forced to watch the blade descend ever lower, every moment thinking that they are closer to being cut in twain.