The milk is warm because it just recently came out of the cow. Like humans, cows are warm-blooded animals, so their milk comes out warm to the touch.

This probably happens fairly often in Jess's life, since he lives on the farm and milks the cow on a routine basis, but it happens specifically at the beginning of Chapter 13: "Building the Bridge." Let's take a look:

"He woke up Saturday morning with a dull headache. It was still early, but he got up. He wanted to do the milking. His father had done it ever since Thursday night, but he wanted to go back to it, to somehow make things normal again. He shut P. T. in the shed, and the dog's whimpering reminded him of May Belle and made his headache worse. But he couldn't have P. T. yapping at Miss Bessie while he tried to milk.

No one was awake when he brought the milk in to put it away, so he poured a warm glass for himself and got a couple of pieces of light bread."

Generally when we drink milk, it's nice and cold because it's been refrigerated. But Jess's milk hasn't gone through any refrigeration; it went straight from Miss Bessie into Jess's container, and from there into the glass.

You might think it sounds gross to drink warm milk, and if you haven't had it before, it probably would be gross to you! But Jess is used to it; it probably tastes fresh to him. And, babies drink naturally warm milk from their mothers, so it must not be that bad.