In fact, we do not always need to give evidence to our beliefs when talking about knowledge. We have two different parts of knowledge. One is about what is abstract like talking about beauty, for example. The second is about what we really see or notice. An example of this is when we talk about the solar system.
As for what is abstract, it is no use talking about evidence. You talk about something being beautiful, for example, and you show the things that make it beautiful. You talk about what is hidden or you can't talk about because it differes from one person to another. This is because you are talking about your emotions or feelings, so how can you convince others?
On the contrary, talking about the things that we see or deal with is something more or less easy. Let's talk about planes beying flying in the sky. Do we need to give evidences about the process of flying? In this case, the fact that we need evidences when talking about our beliefs is something controversial. In short, it depends on the thing that we are talking about