Since the Speaker of the House of Representatives sets the agenda, is there any way to unseat the Speaker ?
Curiously, there's nothing in the Constitution that requires the House to elect one of its own members as a Speaker (Article I, Section 26), so defeating a reigning Speaker in his/her district isn't necessarily a way to get him/her out.
According to the Rules of the House (which are based on Jefferson's Manual), a Speaker may be removed at the will of the House, and a Speaker pro tempore appointed. (Rules of the House, Sec. 315.) The annotations to the Manual provided by the Office of the Clerk of the House note that the House has never removed a Speaker (although they have removed other sitting officers). A resolution to remove an officer is a matter of privilege, meaning it has precedence over all other questions except a motion to adjourn. (Rules, Sec. 664 et seq.)
The "question of privilege" applies to your question, in that the Speaker cannot elect not to address a question of privilege, even if it involves him/herself being unseated. The Clerk's commentary on the handling of questions of privilege notes that, "while under Rule IX a question of privileges of the House takes precedence over all other questions," the Speaker "may entertain unanimous consent requests for 'one-minute speeches' pending recognition for a question of privilege, since such requests . . . temporarily waive the standing rules of the House relating to the order of business" (Rules of the House, sec. 665). The practical outcome of such a move would be a House-equivalent of a Senate filibuster -- if the Speaker can keep those one-minutes coming, he/she can postpone impending doom while his/her allies are out rounding up allies.
Informally, my colleagues are right -- unseating a sitting Speaker by defeating him/her in a district election is the most common way to achieve "involunary retirement." You might find it interesting to read up on the life and times of Czar (or Uncle) Joe Cannon, who was a master at Speakership (1903-1911) and dominated the House, but was finally ousted by a change in the make-up of the Congress from Republican to Democrat. By the way, that's another way to get a new Speaker -- change which party constitutes the majority in the House.
Other less direct means of getting him/her out is to promote the incumbent to Vice President or some other government position (which requires the Speaker to resign his/her position) or remove both the President and Vice President from office -- which would elevate the Speaker to the Presidency, again requiring resignation from other government positions.
So, the short answer to your question is, yes, there are ways to unseat the Speaker.
Certainly, it stands to reason that if the Speaker of the House was doing such an awful job, members of their party would approach them in confidence and ask them to step down. I cannot imagine that in a political climate that is tenuous at best for incumbents there would be a passivity involved if the speaker was doing a horrific job that would impact their own chances of reelection. The previous thoughts were really accurate in being able to identify how the Speaker of the House is an elected official and being up for reelection could be one of the best ways to unseat the speaker. Dennis Hastert, former Republican Speaker of the House, was a Representative from Illinois who experienced this himself. Despite being one of the most powerful people in Washington, he was unable to save himself from challenges in his own district.
The most common way to unseat a Speaker of the House would be for the Speaker to be defeated in an election and thereby lose her seat in Congress. But that is not the only way. The Speaker is chosen, essentially, by a vote of the party that has the most members in the House of Representatives. Therefore, the Speaker can be removed from the post by a vote of the majority party. So, if the majority party were to disapprove of the way that the Speaker was handling the job, they could remove her.