In accord with the above post, if a civilized language differs greatly from one's native language in alphabet and sounds, it is more difficult to learn, certainly.  Chinese and Japanese do, also, have the tonal sounds that present difficulties, if not compromising situations as different tones have negative or positive connotations.

I think the hardest language to learn depends on what is your primary language.  For an English speaker, I would suggest the hardest languages are those that do not use the Roman alphabet and/or are not based on latin.  An English speaker can pick up other latin based languages like French or Spanish fairly easily.  The language even sound similar.  Other languages that use the same or a similar alphabet, like German, are also easier than some.  For me, learning Japanese was excruciating.  Not only was there vocabulary to learn but I had to learn three entirely new alphabets.  There was nothing similar to English so it was truly learning a whole new language from scratch.  I would imagines some middle eastern languages like Arabic present the same type of difficulties.  I did have a friend who spoke several languages that told me that learning sand script was the hardest thing he had ever done.

I have heard that Mandarin Chinese is the hardest language to learn. That said, I would think that tribal languages would be very hard given the limited nature of the language (limited to very small populations/villages).

I would suggest that most basic languages are relatively easy to learn. It is the use of slang and dialects within the language which can cause problems.

I have a friend that was in the Peace Corps and traveled extensively; he said that Afrikaans and Tsonga were extremely difficult to learn for an English speaker.  Tsonga apparently uses sounds or phonemes not used in the English language (kind of like a back of the throat click).  Just learning how to make the clicks was extremely difficult, not to mention incorporating them into a spoken language.

I think Chinese is really hard because of all the tones that must be used.  These are differences in sounds that are hard to even notice if you're not used to them.

One thing that is very interesting linguistically is that the hardest languages are the ones that very few people speak.  If the only people who speak some language are the ones who learn it as their first language, the language can be really hard and have all sorts of weird rules.  But if it's a major language and many people have to learn it, it comes to be easier over the years.  So the most difficult language is probably some language you've never heard of.

I've always heard that English is actually the hardest language to learn. As a native English speaker that seems hard to believe, but apparently we have so many irregularities in our verb forms and exceptions to rules that it's difficult to learn for non-English speakers.