What an interesting question.
Some people will tell you that there are telltale signs when someone is lying. Allegedly, people shift their eyes in one direction or another, rather that using eye contact. I think this may be the case in some situations, but really good liars know all the tricks.
Sometimes instinct plays a role in measuring someone's honesty, but for people like me, I don't know that my "gut" feelings are accurate because "I want to believe" the best of everyone.
I don't believe there are ways to be certain all the time, but I think there are some good practices we can adopt to try to protect ourselves from those who are less than forthcoming.
First, I always try to exercise caution. If you become someone's friend, be careful what you share. The idea of "best friends" is nice, until there is a falling out: if you have spilled your guts to that person, he/she knows all that is necessary to hurt you if that is his/her intent.
My mother always says that if you are lucky, you can could your true and close friends in your life, on one hand. This won't apply to everyone, but I keep this in mind before I open my mouth. I have had someone lie to my face about something extremely important; the intent seemed genuine, but I also wanted to believe: biggest betrayal I ever experienced, but a great life lesson too.
Lastly, look at a person's track record. Look at how he/she treats other friends, former friend, parents, siblings and spouse. You cannot, certainly, believe everything you hear, especially from someone with a grudge from the past, or someone eaten away by jealousy who won't give you accurate feedback. However, I think that people tend to act the same way with everyone, and sometimes watching is a good idea. "What you do speaks so loud I can't hear what you're saying." And if people, several, are warning you, at least listen.
And if you're looking for a casual friendship and don't need to share your secrets, being friends with someone not completely honest may not hurt you (though it will get complicated), but I would never enter into a romantic or business relationship with someone you cannot fully trust; trust is the most important element in a positive, functional relationship of any kind.
This is one of life's toughest challenges. A lot of us do what to believe what others tell us, especially the good stuff. I cannot always tell if someone is being honest with me, so I tend to keep my mouth closed, don't share other people's secrets, and wait and see.