I think that the main difference has to do with the extent to which religion is seen as a major motivating factor in the terrorism. But I don't think this is due to the religions involved. Instead, I think it's due to the broader circumstances.
With Islamist terrorists like Al Qaeda, there is a strong appeal to religion. By contrast, terrorists in Northern Ireland identified themselves as Catholics and hated Protestants, but the hatred was as much about politics as about religion.
I think, though, that that was because of the circumstances, not anything inherent to the religions. The Irish republicans had a clear political grievance. The Islamists have a political side to their anger, but it's harder to express because it's much less clear-cut than having Ulster break away from the UK.