When Bartleby begins his automatic response "I would prefer not to," the lawyer can not believe it. What makes the lawyer bewildered and unable to discipline or fire him is how calm and confidently Bartleby utters this phrase:
Had there been the least uneasiness, anger, impatience or impertinence in his manner; in other words, had there been any thing ordinarily human about him, doubtless I should have violently dismissed him from the premises.
Bartleby refuses again and again. The lawyer asks Nippers what he thinks and Nippers says Bartleby should be kicked out of the office. Nippers is typically more irritable in the morning, less so in the afternoon. Turkey has the opposite tendencies. However, both of them are affected by Bartleby's behavior and annoyed at having to do his work:
. . . at every page or two, Turkey deferentially dropped his opinion that this proceeding was quite out of the common; while Nippers, twitching in his chair with a dyspeptic nervousness, ground out between his set teeth occasional hissing maledictions against the stubborn oaf behind the screen.
The lawyer is frustrated with Bartleby but is also sympathetic to him, thinking that his odd behavior is involuntary. Days later, in the afternoon (when Turkey is more irritable and Nippers is more agreeable), the lawyer calls on them again to do Bartleby's work. Turkey wants to beat Barletby up and Nippers, calm in the afternoon, replies with calm professionalism:
I think his conduct quite unusual, and indeed unjust, as regards Turkey and myself. But it may only be a passing whim.
Bartleby's behavior only upsets the worker who's in a foul mood. If it is morning, Nippers is upset; if it is afternoon, it is Turkey who gets angry.
The lawyer decides to live with Bartleby's eccentricities. When he finds that Bartleby has been living in the office, he is filled with melancholy and sympathy, but then annoyance once again. During another morning, Nippers loses his temper again. Turkey calmly suggests that ale might help Bartleby snap out of his odd behavior. The lawyer notices that he and the other workers have begun to use the word "prefer" without realizing it.
Eventually, Bartleby refuses to do anything. The lawyer tries to fire him but he won't leave. The lawyer is so confounded and unsure of what to do that he decides to move his office closer to City Hall. Bartleby has disrupted their routine so much that the lawyer sees no other humane alternative than to move the entire business somewhere else.