"Shut up your mouth" (or better, "shut your mouth," since the compound verb "shut up" is intransitive when used in the sense "cease speaking") is in the imperative mood.
The imperative mood in English is characterized by the implicit but unexpressed subject "you." Thus, in "Shut your mouth," the deep structure is "You shut your mouth," with the "you" omitted in the final structure by rule.
This causes obvious problems if you want to put the sentence into the passive voice. An ordinary sentence is taken from active into passive by the following transformation, which removes the original subject (optionally inserting it via a prepositional phrase) and promotes the original object to subject position, changing the verb into the appropriate form of "to be" plus the past participle of the original verb:
You broke the window.
The window was broken (by you).
However we cannot perform this transformation with an imperative sentence:
(You) shut your mouth!
Mouth be shut (by you).
The second sentence is absurd, since the rules for the imperative mood demand a second-person subject that is grammatically present but unexpressed.The passive transformation, in removing the original subject and moving the original object into its place, necessarily destroys the imperative effect.
There is, however, a way around this. You can express the imperative by using a helper verb such as "let," which will allow you to retain the imperative while still permitting the main verb to be transformed into a passive form:
Drink the medicine! (= You drink the medicine!)
Let the medicine be drunk! (= You let the medicine be drunk by you!)
This retains the imperative force because the insertion of the helper verb "let" (or other verbs, such as "may") opens up a position for the unexpressed second-person subject and so preserves the imperative mood of the whole.
It should be noted that these formations seem somewhat dated in contemporary speech, and should be used sparingly.