You're right that Mandarin Chinese is a difficult language to learn. I think you're a little bit off as to why. In addition, you are wrong about how quickly one can type in Chinese, for the same reason.There's much more, including an example of the tremendous difference in "meaning density," in the original comment.
The greatest difficulty I saw when watching Americans learn Chinese is the tones. I watched a small number master the writing system, but I never saw a person who could differentiate the tones like a native speaker. And so once you get into an intermediate level, it becomes far easier to read and write Chinese than it does to understand and speak it.
Because the tones carry meaning, Chinese has actually developed into a language where the "meaning density" (I'm just making up a term here) is very high with regard to the syllables...
This becomes a huge problem when trying to watch the news, when they use Chinese acronyms (think the way the military abbreviates "Central Command" to "CENTCOM" or how the "Ministry of Peace" was abbreviated to
"minipax" in Orwell's 1984). The anchors simply say so much in such a short amount of time that non-native speakers (or even less-educated native speakers) are quickly overwhelmed.
...the Mandarin Chinese that we learn as students of a second language is completely different from what is spoken on the ground. Like your examples of French and Latin, it's only spoken and understood by the elites in China. I can't understand someone from Sichuan or Shandong speaking "Mandarin", the same way I can't understand some English accents from the British Isles. I would even venture to say that Standard Mandarin Chinese has fewer native speakers than English. For these reasons I think Mandarin Chinese has almost no chance of ever overtaking English in importance.
My roommate tells me that tones were also his greatest barrier in trying to learn Mandarin—as Shane says, this seems to be a problem for the vast majority of learners. It's yet another way in which Chinese departs from most other languages. This emphasizes, I think, how "English versus Chinese" alone isn't really the right way to look at the competition. At the very least, it's also "Spanish and Portuguese and French and Italian and German and Dutch and more against Chinese," as all the former languages are far more similar to English, with the same basic alphabet, related syntax, some shared vocabulary, and no tones. It will be extremely difficult to convince speakers of these languages to adopt Chinese as their second language of choice—the learning curve is just too steep. (To be fair, there are also additional languages more similar to Chinese, but they're spoken by a much smaller population.)
Shane's point that Standard Mandarin is to a large extent an artificial construction, spoken by a much smaller number of people on the ground, is also very good. One of the most remarkable aspects of American English is how it remains almost constant across large swathes of the country: when I was eleven, I moved from Phoenix, Arizona to Portland, Oregon (1000 miles away!) and barely noticed any difference in speech. In more established regions of the country, there are more distinct dialects, but even they're starting to disappear as General American continues its relentless push.
Admittedly, at one point this was an artificial "standard" dialect as well, elevated by television's somewhat arbitrary selection of Midwestern speech as its style of choice. It's possible that China will someday experience the same transformation. But it's starting with greater fragmentation of dialects than ever existed in America (arguably even in the English-speaking world at large), and in the meantime English can only solidify its dominance.