A medieval castle would clearly be a primary source if we are studying life in the Middle Ages. The guidelines in the link below specifically state that buildings are a type of primary source. They are classified as “relics or artifacts” of the time when they were built.
In your question, you ask about castles “apart from it was built at the time.” I assume that you mean something like “why, other than the fact that it was built at the time, is a castle a primary (or secondary) source?” There are two main reasons for this.
First, the castle cannot be a secondary source. Secondary sources take primary sources and interpret and/or analyze them. Clearly, a castle cannot interpret or analyze a primary source. If, for example, there were a tapestry in the castle that was made in 1600 and which depicts events from 1200, the tapestry would be a secondary source about the events of 1200, but the castle itself cannot act as a secondary source.
Most importantly, a castle is a primary source because it can give us direct evidence of how people lived in medieval times. As the link below says, a castle can “offer an inside view” of life in those times. From the castle we can see how large of rooms people lived in. We can see what sort of sanitary facilities they had. We can see whether they cared enough about religion to build elaborate chapels within their castles. We can see how they built defenses, which can tell us something about what sorts of threats they faced. A castle can give us an “inside view” of all of these things (and more). This is the main reason (other than when it was built) that it should be seen as a primary source on the Middle Ages.