I think the engine here definitely refers to the goldfinch and her family. Before she arrives, the laburnum tree is quiet, "quite still," with nothing to indicate that there is anything alive inside it. However, when the goldfinch arrives, the tree seems to surge to life, but the "machine" which springs into action here is not the tree itself but simply the creatures living inside the tree, who have now been stimulated to chitter and "tremor" at the arrival of their mother. Hughes describes this as "the engine of her family," a gathering of small birds who the goldfinch is able to "stoke" as she would an engine by feeding the baby birds with the food she has gathered for them.

To a certain extent, the "engine" is confined within the body of the tree—the laburnum is the casing of the engine, as it were. However, the imagery of the poem paints the mother goldfinch as the stoker, feeding the engine of her chicks in the same way that an engineer might feed a fire running an engine. It is not the silent tree which needs her attentions, but only the living machine she has stowed inside it.

It seems to me the engine must be first the goldfinch and then her family.  The laburnam was still, until the goldfinch entered and became the action or catalyst for the machine.  "She enters the thickness and a machine starts up."  From the quiet and unmoving, we now see, feel, and hear many things:  "a twitching chirrup," "a startlement," "chitterings, and a tremor of wings, and trillings."  In fact, "The whole tree trembles and thrills" as she enters it.  "It is the engine of her family."  Mama bird brings "fuel" to her babies; and when she leaves, "the laburnam subsides to empty."

The laburnam made no movement or sound on its own; once the "engine" came there was life and sound and motion.  When the engine leaves (when the goldfinch finds a far branch on which to perch), the machine is once again silent.