Nibblez, I too am a non-native Mandarin Chinese speaker.

I agree with you, Jerry, and everyone else here who says that immersion is absolutely key, no matter what language you're trying to learn. The fact is, people generally only master conversation in a second language when they have no choice in the matter, because they need the second language in order to survive living where they are and doing what they're doing.
It's nigh on impossible to master a language when all the native speakers around you refuse to use it with you, usually because they reckon (correctly, in many cases) that their English is better than your version of their language. This is the dark side to being a native English speaker in today's world.
A friend of mine moved to Israel. She has tried to learn Hebrew, but her accent and mannerisms give her away as American right off the bat, and since 2/3 of Israelis are conversant in English, they just switch right over. It's not that she can't survive there. But it's tough when all the jokes and little side comments are in Hebrew. She's also shit outta luck if she wants to to take a local licensing exam for her job. You don't really get employed somewhere, definitely not in the service sector, without speaking the local language, no matter how good the locals' English is.
With this in mind, I say that a major and as-yet underappreciated factor in the difficulty of learning a language, is access to places where you will have no choice but to use it. Easiest in this category are therefore languages spoken by countries with a 'big country' outlook: China, Russia, Brazil, Indonesia, and Mexico are good examples. Go to any of these, and it's very easy to find yourself in situations where you're toast if you can't communicate in the one language most people there know,
and expect everyone coming to their country to know!
By contrast, little languages spoken in small countries that are forced by circumstance to be very outward-looking and international, are very hard to learn. Locals in such places are likelier to use their local language as an 'insiders' language', and be more than ready to speak English and/or some other major lingua franca with everyone else. Learning these almost necessitates a preexisting closeness with a community of speakers, who like you and are patient with you.
Another big thing no one has mentioned: the first tough milestones to learning any second language is should be to acquire a familiarity with the sound set that that language uses. My Chinese professor spent the first two weeks of class just leading us in a chorus of all the possible sound combinations that could make up a Chinese word. He was brutal about correcting us, especially with the use of tones. We made fools of ourselves, but gained a lot of confidence together in our ability to make our mouths and brains move in a Chinese way. It's very easy to get discouraged learning any language, when the very sounds that make it up daunt you, and your accent is so thick that no native speakers understand much of what you say. I noticed a strong correlation between students who were willing and able to make their tongues form the basic sounds of Chinese, and those who were later willing to make the jump into trying out expressing ideas and saying things to native speakers spontaneously.
I don't mean to belittle the importance of how close your native language is to the one you're trying to learn. That provides a big leg up in terms of getting familiar with grammatical constructions and learning new vocab. It also likely provides you a cultural advantage -- I'm willing to bet native French and Spanish speakers think and organize the world in their heads much more similarly to me than the average Mandarin or Arabic speaker. But still, without the pressure immersion provides, even close languages won't be learned. I reject the notion of 'picking up' a language, which implies passivity. It's always part of a concerted effort for practical ends, when it works.