How did the Soviet Union maintain effective border security over such vast distances?
The simple answer is through harsh policies, a massive border guard force and Mother Nature.
While the USSR had a very lengthy border with neighboring countries and a massive coastline, much of that territory and coast was completely forboding to humans attempting to escape. Vast stretches of tundra, forest and ice cover the Asian portion of the Soviet Union, while the Arctic makes up most of their coastline. Suppose someone did "escape", where would they go? How would they get there.
Still the USSR had a massive border security force first under the NKVD and later under the KGB. There were nearly a quarter million of them in 1989 right before the revolution. Compare that to less than 20,000 in the US Border Patrol.
Soviet Border Guards also had authority to shoot on sight anyone attempting to leave or enter the country without permission, and they did by the hundreds. It didn't take long before very few even attempted it.