If we are speaking of the rights of the accused versus the rights of victims in the United States, we must remember that there is well-settled legal doctrine that establishes the rights of the accused, rooted in the United States Constitution and its amendments. It is also important to remember the central idea from which all these protections flow, that the accused is an innocent person until proven otherwise.
The Constitution is silent on the rights of the victim, so, in fact, there are no established Constitutional rights. In more recent times, there have been some "rights" accorded victims who are alive, for example, protections for rape victims, who historically have been cross-examined as though they were at fault for being raped, or for juvenile victims of molestation, who are not always required to testify directly before the accused, sometimes testifying by video or in the judge's chambers. For the families of deceased victims, quite often, their testimony is part of sentencing proceedings or parole proceedings, not dispositive, but certainly admissible. In some jurisdictions, families may be notified when someone is released from prison.
One thing I like to consider when faced with seemingly important and pivotal questions like this, is the fact that we don't have a fixed, ready-made answer for it. Even though it's somewhat of an appeal to authority, I like to think that, after several thousand years of civilization, we would have found a concise answer if one existed. The same sort of reasoning goes for the "magic bullet" approach to education, welfare, and so on.
One of the problems preventing us from reaching a single common answer to this question is the fact that every case is unique. We have no means of comparing two cases in an impartial, empirical manner that clearly correlates causes and effects, i.e. in case A, the victim was prioritized, and in case B, the accused was prioritized, with effects X, Y and Z that can be extrapolated and applied to all similar cases.
Another important aspect of this question is to consider what we mean by "rights". Typically this means human rights, i.e. expectations that every human obtains at the moment of their birth. Exactly what is considered a human right is, itself, debatable; for example, we could argue that an imprisoned criminal has had their human rights violated because they are not free to pursue their own happiness (even if that happiness is found in crime).
However, I think ultimately we should put some emphasis behind the position of the accused. Our justice system (ostensibly) works on the principle of innocence until guilt is proven; thus, to give favor to the victim is to implicitly find guilt, or at least to punish, the accused. We must also consider the psychology of justice, and how societal standards can frame our opinions of guilt or innocence. A need for vengeance is often a problem in obtaining an impartial evaluation of facts; consider the Trayvon Martin case, and the number of public calls for George Zimmerman's murder to "atone" for Martin's death. This is an extreme example, but it clearly illustrates that injustice can follow from a victim-prioritized perspective.