The speaker's thoughts, in this complex poem, on the flowers given her by her beloved are clouded by the Shakespearean style play on words that follows after the lines about the flowers: "Take back these thoughts ...." What is important here is that "take back" represents the double meaning words can have; she is wittily and charmingly toying with her beloved, who has apparently been away as the flowers continued unrenewed "all summer through and winter."
"Take back" isn't used in it's sense of to return a former gift now spurned and rejected due some breach of faith on the givers part. If "take back" were used in this way, she would not have said the flowers continued to grow in the room they grace through both summer and winter without seeming to miss the "sun and showers." She would have said something like his inconstancy or his betrayal had made them fade and wither--so take them back--a blighted reminder of crushed hopes.
"Take back" is used to mean allow me to return in kind--as you gave to me, so do I give to you. This is set up and confirmed by the line that precedes "Take back." It reads: "So, in the like name of that love of ours, ...." A paraphrase of this line might be: "Therefore, according to our love that has likewise grown without the renewal of your presence, ...." What precisely is he to take back?
She is asking him to take back (as you gave me, so I give you) the words that "here [by the flowers] unfolded too." These words were so deeply welcome and embedded in her heart that on all the "warm and cold days" of the lapsed time through which they were seemingly parted, she "withdrew them from her heart's ground," or lifted them from their abiding place in her heart to view as she also viewed the flowers. Browning is of course continuing the garden metaphor with "unfolded" and "ground" and later with “weeding.”
What were those words that he spoke and she is now returning? Hints are contained in the last five lines. Eglantine is a simple English rose that symbolically represents purity. Ivy is a seemingly endless climbing vine that symbolizes fidelity. It is the eglantine and the ivy, which together symbolize the speaker, that she asks him to (1) to take back; (2) to accept as she accepted his flowers; (3) to store where they shall not "pine," which means to wither away with grief or mourning; and to (3) conduct himself so as not to betray her purity and fidelity: "to keep their colours true." She concludes by saying the gifts she gives him, in kind for the words he first "unfolded" to her, have "their roots" grown down in her soul: "their roots are left in mine."
The words he "unfolded" to which she now says to "Take back" are words declaring his love and devotion, which in Browning's day was accompanied by a proposal of marriage. One last point to clear up: It is now easier to see that "Indeed, those beds and bowers / Be overgrown with bitter weeds and rue" refers to the weeds of loneliness from separation that have grown on her heart during his absence, which only his loving hand can pluck from that ground: "And wait thy weeding.”
One thing to consider about this poem is that the narrator is looking back to the past. In the past, the flowers have been pleasing and emotionally fulfilling, as was the love affair. The flowers, even though taken out of their natural environment and brought into a contrived setting, were gladly embraced by the narrator. However, something has happened and the relationship has gone beyond the point where flowers can work their magic.
"And which on warm and cold days I withdrew
From my heart's ground. Indeed, those beds and bowers
Be overgrown with bitter weeds and rue,"
Withdrew is key here as it indicates the time for flowers is in the past. Hence the narrator is looking back to something. We cannot help but notice either, the reference to the weeds (possibly evil or untrue deeds done to the love) or the reference to the word rue which means to regret or be sorry for.
Clearly the narrator once loved the flowers as the represented what seemed to be real love, but that love has turned out to be nothing more than weeds.