Actually, it is practically impossible to know who, if any one person, was responsible for the actual writing of the Constitution. At its opening session, the Convention first chose George Washington as its President and James Madison as its recording secretary. The second motion was to keep the proceedings secret. The room in which the convention met in Philadelphia is on the second floor of independence hall so that the windows could be opened and eavesdroppers could not listen in. It is widely reported that Benjamin Franklin, well into his eighties at the time, often left the proceedings early to refresh himself at a nearby tavern. When he became somewhat loose tongued about events in the meeting, young men who were sent to accompany him often gently steered him away.
Madison is commonly called the Father of the Constitution because the only record we have of its proceedings are his Notes on the Constitutional Convention. One can only surmise as to his actual participation in the meeting, as it is not mentioned. Although he authored the Virginia Plan, it was modified by the Connecticut Compromise, one of several compromises which comprise the Constitution.
So the safest answer is no one really knows if one individual more than others was directly responsible for the finished product.