The images used in this poem are not what might first come to mind upon reading it!
The "yellow man" in the east is the Sun rising in the morning on the eastern horizon. During the day, the Sun moves in the sky from east to west; as the sun sets in the west, colors change when the "purple man" appears and turns the clouds and sky shades of purple, red, pink, and orange.
The word "purple" most directly symbolizes the sunset. It may further be interpreted to reflect the power of God, the "He" who controls the daily movement of the sun from east to west; or "He" may refer to the Sun carrying the colors within itself for release at the proper time every day.