The reason that Mr. Otis is annoyed could be referencing two different early locations in the story. The blood stain in the floor is annoying, especially after he tried three times to get rid of it. However, I think that you are most likely referring to the ghost's first appearance before Mr. Otis.
After Mr. Otis failed to get rid of the blood stain, he is willing to admit that there might be some truth to the ghost stories.
"Mr. Otis began to suspect that he had been too dogmatic in his denial of the existence of ghosts."
That very night, though, all doubt disappeared from Mr. Otis's mind about the presence of the ghost. Mr. Otis and the rest of the family went to bed at 11 p.m. Then two hours later Mr. Otis was woken up by strange noises. He left his room to investigate and came face to face with the ghost. But instead of being frightened, Mr. Otis told the ghost to use some special lubricating oil on the ghostly chains. That way the ghost won't wake anybody up.
"My dear sir," said Mr. Otis, "I really must insist on your oiling those chains, and have brought you for that purpose a small bottle of the Tammany Rising Sun Lubricator. It is said to be completely efficacious upon one application, and there are several testimonials to that effect on the wrapper from some of our most eminent native divines. I shall leave it here for you by the bedroom candles, and will be happy to supply you with more, should you require it."
Mr. Otis is rightly annoyed at having been awoken in the middle of the night, but it is the following morning that he expresses even more annoyance. The Canterville Ghost did not take the oil, and that really annoyed Mr. Otis, because now the family will keep being woken up.
The United States Minister was naturally a little annoyed to find that his present had not been accepted. "I have no wish," he said, "to do the ghost any personal injury, and I must say that, considering the length of time he has been in the house, I don't think it is at all polite to throw pillows at him,"—a very just remark, at which, I am sorry to say, the twins burst into shouts of laughter. "Upon the other hand," he continued, "if he really declines to use the Rising Sun Lubricator, we shall have to take his chains from him. It would be quite impossible to sleep, with such a noise going on outside the bedrooms."