In Macbeth, did Macbeth ever feel guilty about the atrocities he committed, or did he just regret that his quest for power was unsuccessful?
I think that if we look carefully at the text, Macbeth, even when it is clear that the game is up and he is going to be defeated, never displays repentance or regret for his actions. The closest perhaps he gets to it comes when he faces Macduff and says to him:
But get thee back, my soul is too much charg'd
With blood of thine already.
Here, we see that Macbeth obviously doesn't want to face Macduff because he has killed so many of his family already and does not want to add another to the list. However, this is the closest comment we see indicating any form of regret in what he says.
As regarding your second possibility, that Macbeth is only sad that his quest for power was unsuccessful, I am not necessarily sure that we can find evidence to support this view either. To be honest, until his last appearance in the play, Macbeth does not show much regret about anything. Even when it is clear that all of the prophecies heralding his death have come to pass, he still chooses to fight his way to death, unyielding to the end. Note what he says:
I will not yield,
To kiss the ground before young Malcolm's feet,
And to be baited with the rabble's curse.
Though Birnam wood be come to Dunsinane,
And thou oppos'd, being of no woman born,
Yet I will try the last: before my body
I throw my warlike shield: lay on, Macduff;
And damn'd be him that first cries, "Hold, enough!"
Whatever we think of Macbeth, we certainly have to appreciate his defiance and bravery up until the very end in the face of certain defeat. He will "try the last," never letting himself give in or become weak in the face of the witch's prophecies. He displays no regret that his bid for power was ultimately unsuccessful, only a defiant spirit that carries him through to his death.