If this was indeed implemented, it would have to have been around two centuries ago, when horse and gun were the most effective methods of transportation and defense, respectively. Looks like this was a purely provincial law, not necessarily a country-wide thing going on in Canada.
As with other provincial laws, they are unique to a smaller population with perhaps a social circumstance so diverse and separate from the rest of the country that special and somewhat odd allocations and concessions might be permissible. In the case of the gun and the horse, it is basically (like the previous post inferred) that the person is given the chance to go far away, and the gun would protect them in the wilderness.
I am almost positive that if this is an anachronistic rule that is still in place for being overlooked, it still does not mean that it is being practiced today.
Personally, I highly doubt that this was ever true. I did a quick search and found it on places like a Mental Floss blog, things like that. It seems to be the kind of thing you just see everyone repeating on the internet regardless of if it's true.
If it ever was actually done, it's clearly not done now given that these sources all say you got a horse too. Supposedly that was to let you leave town after being released. However, I really doubt this ever happened since giving someone a horse in the old days would have been terribly expensive...