Thoreau was not the only famous writer who felt that we tend to fritter away our lives. Here is what Chekhov says in his best story, "The Lady with the Pet Dog":

Useless pursuits and conversations always about the same things absorb the better part of one’s time, the better part of one’s strength, and in the end there is left a life grovelling and curtailed, worthless and trivial, and there is no escaping or getting away from it—just as though one were in a madhouse or a prison.

Thoreau had one great advantage--if it isn't considered a great misfortune or a great handicap. He was never married. A single man is able to do pretty much as he pleases. If he is frugal and spartan in his tastes, he can live very cheaply and devote much of his time to reading, writing, and communing with nature. But if he is married he is likely to have children, which leads to wanting a good home and seemingly endless expenses, worries, and details.

When William Wordsworth wrote

The world is too much with us, late and soon.
Getting and spending we lay waste our powers...

he was married and had three children. He was also the sole support of his sister Dorothy. So, unlike Thoreau, Wordsworth had six people to support, including himself. And he was trying to do that by writing poetry. Eventually he had five children altogether. He seems to have been a nature lover by necessity. There was no place for him to write in his little cottage, so he took long walks and composed his poems in his head. 

Another interpretation of Thoreau's words is one that pertains to many in contemporary times who fritter their lives away in details and lose sight of the larger and more important issues and relationships in life. For, at the beginning of the paragraph before this quoted line of Thoreau's, he writes,

Still we live meanly, like ants; though the fable tells us that we were long ago changed into men; like pygmies we fight with cranes; it is error upon error, and clout upon clout, and our best virtue has for its occasion a superfluous and evitable wretchedness.

Thoreau observes that men are like ants as they rush from spot to spot, unaware of the world around them, fixed merely on  small, meaningless details.  They are caught up with the trite and quotidian happenings and lose sight of the more universal which is meaningful in life. By simplifying their lives and eliminating superfluous details, people will grasp what is truly meaningful: family, friends, love, respect, integrity. 

Shams and delusions are esteemed for soundest truths, while reality is fabulous. If men would steadily observe realities only, and not allow themselves to be deluded,... life would be like a fairy tale....Children who play life, discern its true law and relations more clearly than men....

Thoreau points to the destructive power of petty fears and petty pleasures; they cloud men's thinking and true enjoyment of the life of the soul. He urges people to look more deeply at life and recognize and nurture what is truly meaningful. Men should cultivate such virtues as the love of Nature and truth--what is "sublime and noble." Further, Thoreau urges men to spend their days as deliberately as Nature and deal in realities, not trivialities. Arguably, in Walden, Thoreau satirizes the institutions of man and his materialism which lead him to "fritter" his life away like the ants who lay "clout upon clout."

When Thoreau speaks of the detail that fritters away one's life, he speaks of the emotional and physical clutter that surrounds individuals.  Thoreau is demanding that individuals find a more simplistic form of being in the world:  "I went to the woods because I wished to live deliberately, to front only the essential facts of life, and see if I could not learn what it had to teach, and not, when I came to die, discover that I had not lived."  Thoreau makes the argument that individuals who believe in the power of these "details" really don't live their lives.  Only when individuals recognize that these details are not meaningful and to get away from them constitute real and true meaning can there be something substantive to being in the world.

The quote of "Our life is frittered away by detail. An honest man has hardly need to count more than his ten fingers, or in extreme cases he may add his ten toes, and lump the rest. Simplicity, simplicity, simplicity!" demands a reduction in what constitutes "meaningful" attachments in the world.  A paraphrase of this can be that individuals need to prioritize what is important and discard the rest.  Another paraphrase of the quote could be that individuals should emphasize that which is meaningful and important and place the other elements in their proper context.  When Thoreau speaks of "Let your affairs be as two or three, and not a hundred or a thousand," it speaks to this prioritizing as essential.  Thoreau is striving to make meaning out of being in the world.  This can only happen when individuals do not allow their lives to be "frittered away."  In this, another paraphrase can be that individuals should live a life that is not cluttered with extraneous details.