Is this sentence correct "They returned back to the house because they forgot the dog"?.
Actually, a part of it is a bit repetitive. According to a great source I often use, Purdue OWL, you need to "eliminate words that explain the obvious or provide excessive detail." On their site, they say this:
"Always consider readers while drafting and revising writing. If passages explain or describe details that would already be obvious to readers, delete or reword them.
In your sentence, using "returned" and "back" makes the sentence somewhat repetitive. The correct form of the sentence would become:
"They returned to the house because they forgot the dog."