Captain Ahab offers the princely reward of a gold doubloon to any crew member who spots Moby Dick. Doubloons were Spanish gold coins and were considered quite valuable. They were widely used in North America right up until the mid-nineteenth century, so it wouldn't have been very difficult for Ahab to get his hands on one of them.
Ahab nails the doubloon to the mast as a way of preventing any of the ship's crew from getting cute and trying to steal it for themselves. By putting up the reward where everyone can see it, Ahab is also giving his men an excellent incentive to spot the great white whale. Anytime their spirits start to flag during the long, hard voyage, the men can look at the doubloon and remind themselves that there's a big reward at stake should they be fortunate enough to eye Moby Dick on the horizon.