Both of these bombers were so highly effective that it is very difficult to say that one or the other was more effective. This is particularly the case because they were used in such different ways during the war.
The B-29, you could say, was more effective because of its much greater range and its greater payload. This is, of course, what allowed it to bomb Japan from as far away as the Mariana Islands. This is not to say, however, that the B-17 was not effective. The B-17 was quite effective for use in the much smaller distances found in the European Theater.
One of the things that made the B-17 so effective was its toughness. This goes to the degree of protection that it offerred. The defensive gunnery on the B-17 was formidable, as was its general ability to continue to fly after having been hit rather severely. The B-29 was also very effective in protecting its crews. This was particularly true because of the fact that it flew so high and so fast. These capabilities made it very difficult for Japanese defenses to successfully attack it.
Overall, both of these bombers were hugely effective. The B-17 was responsible for the majority of the weight of bombs dropped by the US in WWII. The B-29 was the bomber that helped to end the war with Japan through the "carpet bombing" of Japanese cities. They were both highly effective at both attacking the enemy and protecting their own crews.