General Longstreet, along with General Lee, assumed that they would have known of the Army of the Potomac's movements because the main Confederate cavalry under J. E. B. ("Jeb") Stuart was supposed to be shadowing the Federal army.
During the Civil War, the only reliable way to gain knowledge about the motions of the enemy's troops was to have cavalry who could move around the enemy army and then report on several important things: the number of troops overall, the units making the army, the support troops and wagon train supporting the army and, most important, where they were heading.
In the case of the movement toward Gettysburg, Jeb Stuart's cavalry was up in Maryland and nowhere near the Army of the Potomac, but Longstreet didn't know that. The reason he doubted his own spy is that he couldn't believe the "eyes" of the Army, Stuart's cavalry, would have left the Army of Northern Virginia blind in such a vulnerable position. Stuart was arguably the finest cavalry commander in either army, and he knew his duties well, so it was an incredible lapse of good sense and duty that allowed him to abandon his real job to go up to Maryland and leave Lee's army without knowledge of the enemy's numbers and location.