As I understand it, formal groups within a firm are groups that are created officially by the management. You might have a task force that is formed to work on a given problem or you might have a permanent department with a set hierarchy.
By contrast, informal groups are not officially created. These are ad hoc groups that spring up based on who gets along with who, who has what kinds of shared interests, etc. These groups are harder for firms to control because they do not fit in the corporate hierarchy. But they are often a "place" where a lot of valuable work and networking goes on.