I would argue that Freneau is referring here to the customs of Native Americans. These customs are "fancies" in that they appear somewhat strange and exotic to the white man. And the Native Americans are "ruder" in that they are more primitive, less advanced in their ways, than white American settlers. Although the tone of the poem is sympathetic towards Native Americans, it still betrays the Enlightenment myth of the noble savage, with its condescension towards indigenous culture.

It says a lot about the systematic destruction of Native American life and culture that the "ruder race" are only able to indulge their "fancies" in the after-life, once they've been "released" from the trials and tribulations of the mortal world. For it is only in the spirit world that America's indigenous people can be truly free of the white man and his pernicious influence.

Philip Freneau's poem "The Indian Burying Ground" tells of a place where Indians, long after their death, still sit "seated with their friends." The poem, therefore, speaks to the Native American thought regarding life after death.

Later in the poem, one line states "the fancies of a ruder race." Given that a poem's interpretation depends upon how a reader interprets the poem, a single line in a poem is no different. For me, the line refers to the White Man who infringed upon the Indians' lands and lives. The stanza which the line is found in speaks to a grave stone which has been placed on the Indian burial ground. The grave stone was not a typical marker for the Indians and its presence depicts the White Man's overtaking of their lands.

While the tombstone has been partially eroded by the rain, it still represents White Man's infringement on the Indians' lands and traditions. Therefore, the ruder race is the white man.