In Fahrenheit 451, Bradbury does not reveal where and when Montag stole his first book. Early in the book, he hints at Montag's stolen book collection at several points, like when the Mechanical Hound growls at him in the firehouse:
"If someone here in the firehouse knew about the ventilator then mightn't they "tell" the Hound . . . ?"
After Captain Beatty visits his house, Montag finally unveils his collection to his very shocked wife, Mildred. In total, Montag takes "some twenty books" from behind the grille before apologising to Mildred and then asking her to read them with him.
So, when Montag steals a book from the lady whose house he burns, it is not the first time. By not revealing when and where he first stole a book, Bradbury implies that Montag has been dissatisfied with the status quo for some time but needs a little encouragement. This is why his meeting with Clarisse is so significant.