In "The Canterville Ghost", what worries Mr. Otis about the ghost?

Mr. Otis is not very worried about the ghost, though he does come to believe it exists. At first, in practical American fashion, he pooh-poohs the idea that ghosts could be real. After he realizes the ghost is real, his chief worry is about how to remove the blood stain that it leaves on the library floor. Despite the family's repeated applications of Paragon detergent to scrub it out, the stain keeps reappearing. Another worry Mr. Otis has is that the ghost's  chains make so much noise. For that reason, Mr. Otis gives the ghost a bottle of Tammany Rising Sun Lubricator. Mr. Otis, along with the rest of the Otis family, creates a problem for the ghost: Mr. Otis is not afraid of him at all and focuses on scrubbing out his stains and asking him to oil his chains rather than reacting to him with fear.  

Why was Mr. Otis annoyed in The Canterville Ghost?

The reason that Mr. Otis is annoyed could be referencing two different early locations in the story.  The blood stain in the floor is annoying, especially after he tried three times to get rid of it.  However, I think that you are most likely referring to the ghost's first appearance before Mr. Otis.  

After Mr. Otis failed to get rid of the blood stain, he is willing to admit that there might be some truth to the ghost stories.  

"Mr. Otis began to suspect that he had been too dogmatic in his denial of the existence of ghosts."

That very night, though, all doubt disappeared from Mr. Otis's mind about the presence of the ghost. Mr. Otis and the rest of the family went to bed at 11 p.m.  Then two hours later Mr. Otis was woken up by strange noises.  He left his room to investigate and came face to face with the ghost.  But instead of being frightened, Mr. Otis told the ghost to use some special lubricating oil on the ghostly chains.  That way the ghost won't wake anybody up. 

"My dear sir," said Mr. Otis, "I really must insist on your oiling those chains, and have brought you for that purpose a small bottle of the Tammany Rising Sun Lubricator. It is said to be completely efficacious upon one application, and there are several testimonials to that effect on the wrapper from some of our most eminent native divines. I shall leave it here for you by the bedroom candles, and will be happy to supply you with more, should you require it."

Mr. Otis is rightly annoyed at having been awoken in the middle of the night, but it is the following morning that he expresses even more annoyance.  The Canterville Ghost did not take the oil, and that really annoyed Mr. Otis, because now the family will keep being woken up.  

The United States Minister was naturally a little annoyed to find that his present had not been accepted. "I have no wish," he said, "to do the ghost any personal injury, and I must say that, considering the length of time he has been in the house, I don't think it is at all polite to throw pillows at him,"—a very just remark, at which, I am sorry to say, the twins burst into shouts of laughter. "Upon the other hand," he continued, "if he really declines to use the Rising Sun Lubricator, we shall have to take his chains from him. It would be quite impossible to sleep, with such a noise going on outside the bedrooms."