Colonel Brandon's secret has to with his lost love Eliza. When he was younger he fell in love with Eliza, but she married his brother instead. After returning from the army, Colonel Brandon found out Eliza was now divorced and dying. Before she died, he promised to take care of her daught, also named Eliza. He took care of her daughter and eventually found out that Willoughby seduced and abandoned the young Eliza. Brandon then challenged Willoughby to a duel. He tells Elinor his story after Willoughby abandons Marianne.
Check out the eNotes section on Sense and Sensibility for more detailed information about the book.