The pastor, known as Father or Mr. Robinson in the novel, is not described in any detail at all. He is the narrator, through whose eyes we see the rest of the characters, but he does not look into any of the mirrors salvaged from the ship and tell us about his own appearance. He is always gazing outward, at what is all around him on the island. We are more likely to get a description of the animals he sees, the wild terrain, or the homes he and his sons build than of himself. He is more interested in the states of people's souls than their appearances.
However, we do learn the following from him:
Excellent health had been enjoyed by us all during these ten years, though my wife occasionally suffered from slight attacks of fever, and the boys sometimes met with little accidents.
The fact that he does not mention having himself been ill or accident prone suggests that he has been robust, agile, and fit during his time on the island.
The pastor also describes his sons as handsome and strong, and we have to assume they take after their father. Further, his energy and ability to perform physical feats would indicate that he is strong and healthy. However, we never learn the color of his eyes or hair, leaving it to our own imaginations to construct a picture of him that suits us. It is not unusual in the period in which the novel was written, the early 1800s, for physical descriptions to be left out of a text: Jane Austen also leaves the reader to fill in the details of her characters' appearances.