First, the crew believes that Charlotte killed Mr. Hollybrass because the man was stabbed to death with a weapon that belonged to Charlotte: the dirk that was given to her by Zachariah at the beginning of their voyage. Dillingham had seen Charlotte holding the dirk in her hand while she considered returning it to Zachariah; Mr. Ewing, Foley, and Mr. Johnson also testify to having seen this during Charlotte's trial. Mr. Grimes admits that he taught her to use the knife before the storm, and that she was "uncommon good" at using it. Logically speaking, it would be a natural assumption that the owner of a murder weapon was the perpetrator of that murder.
Additionally, although Charlotte is "not given to lies," her surprising and relatively reckless behavior might aid in convincing the crew of her guilt. While discussing this with Zachariah in the brig, Charlotte asserts that the crew never would have believed that she could go before the mast or climb into the rigging while a storm raged on. Despite their doubts, she did do those things. By that same line of thinking, she knows that the crew might see this unpredictable display of skill as a sign of her ability to have murdered Mr. Hollybrass.