O. Henry does not specify how Jimmy Wells and "Silky" Bob first got to know each other. But Bob makes it clear to the police officer (whom he doesn't recognize as Jimmy) that the two of them grew up together.
"He and I were raised here in New York, just like two brothers, together. I was eighteen and Jimmy was twenty."
They were eighteen and twenty, respectively, when they had their farewell dinner at "Big Joe" Brady's restaurant twenty years earlier. "Silky" Bob seems older than Jimmy in the story, somehow. Perhaps that is because he has had so much more worldly experience than his friend. But Bob would be thirty-eight and Jimmy would be forty. A man can change a lot between twenty and forty--more than he will ever change for the rest of his life. That is the point O. Henry is making in this story. Jimmy's character is fully formed. He couldn't make himself let Bob go free, even if he wanted to. He has become a career police officer with a sworn allegience to duty.
New York in O. Henry's time was a city of neighborhoods. Although it was a big city, most people tended to live in their individual neighborhoods, where they knew everybody and where they did all their shopping, theater-going, hanging out at saloons, visiting friends, going to church, and where their kids all went to the same public schools. The neighborhoods tended to be segregated along ethnic lines. Jimmy Wells is obviously Irish, and Bob is probably Irish too.
The kids all played together on the streets and sidewalks. No doubt, Jimmy and Bob played stickball together and turned the fire hydrants into cold, gushing fountains when the weather was hot.
When Bob refers to "New York" he is probably talking about Manhattan, which used to have a larger proportion of permanent residents than it does today. Greater New York is a huge megalopolis consisting of five boroughs, but O. Henry's New York was smaller and the description of the setting in "After Twenty Years" sounds as if it must be Manhattan.
One of the best things about O. Henry's stories, and the reason for their longevity, is the way they capture the flavor of the past.