O. Henry describes the weather conditions for two specific reasons, neither of which has to do with setting a mood of mystery.
The time was barely 10 o'clock at night, but chilly gusts of wind with a taste of rain in them had well nigh depeopled the streets.
- Because of the chilly gusts of wind and the threat of rain, Bob is standing inside the unlighted doorway rather than out in front. When the policeman approaches him, Bob assumes it is because he appears to be hiding, so he hurriedly explains what he is doing there and ends up telling the officer, whom he doesn't recognize, all about his appointment with his old friend Jimmy Wells after twenty years.
- It is because of the increasingly bad weather conditions that the plainclothesman is able to approach with his overcoat collar turned up to hide his face from Bob. This is why O. Henry adds this description of the weather towards the end.
There was now a fine, cold drizzle falling, and the wind had risen from its uncertain puffs into a steady blow. The few foot passengers astir in that quarter hurried dismally and silently along with coat collars turned high and pocketed hands.
Note that all the men on the sidewalks had their coat collars turned up high because of the cold and drizzle.
Jimmy had to send a plainclothes policeman to arrest Bob because Bob would be alarmed if he saw another uniformed cop approaching him after he had just been talking to one. Bob might not try to flee, but he would certainly deny his true identity, and the uniformed cop could probably not arrest him without a warrant and without probable cause. But the plainclothesman doesn't look like a cop, and he hides his face as much as possible. This does not look odd because of the weather. He never says he is Jimmy Wells but gets Bob talking to make sure of his identity before he arrests him.
I would say that it plays only an indirect part in heightening the mystery.
All we know about the weather in this story is that it is raining by the middle of the story (it felt like it was going to rain at the start) and that the wind is blowing (gusty at first, steadily blowing later). This is not particularly mysterious weather. If O. Henry had wanted the weather to truly heighten the mystery, he might have made it be foggy, since fog lends a mysterious mood.
The weather does help create a sense of mystery because it makes sure that there are few people out and about. The empty streets make the setting more mysterious than crowded streets would have been.