The difference is that velocity has a direction to it and speed does not. If you are going to be technically correct, you have to talk about the velocity of an object in a specific direction. Speed can be given regardless of direction, but velocity must have a direction -- it is a vector quantity while speed is a scalar quantity.
Speed is a scalar quantity and velocity is a vector quantity. As we know, a vector is described by magnitude and direction, meanwhile the scalar quantity is described only by the magnitude.
So, when we want to determine the rate of changing a position of an object, we'll have to calculate the velocity. If we want just to know how fast is moving that object, we'll have to determine only the speed.