Japanese is a notoriously difficult language.
Everyone says so.
If I had a nickel for every time someone here told me that "Japanese is difficult even for us Japanese", I would have a lot of nickels. Unfortunately my corner grocery doesn't take nickels.
Although Japanese is an orphan on the linguistic tree, huddled along with Korean as a grammatical anomaly, this is not what people mean when they say Japanese is difficult. One of the many complaints lodged at today's Japanese youth is that they can't speak proper keigo, the high wire act of formal speaking that gilds everyday spoken Japanese with a glut of formal terms. In it's roughest and most simple form, "What will you eat?" can be grunted out as "Nani ku?" but in keigo it could come out as "Go-shokuji wa nani o meshiagaru desho ka?" Keigo is not something spoken naturally, but requires study and effort. It's not an uncommon sight to see people studying books like "How to Use Business Keigo" on the train. Lacarated keigo is a television staple, with whole shows based around famous comedians put in delicate situations, forced to use proper language and failing miserably.
No matter how far along I progress in my own Japanese study, proper keigo remains a mystery, the very peak of Japanese study. Conversational Japanese is clipped and quick, whole sentences sometimes pared down to a single key word understood by context, but keigo gilds those phrases with loads of florid embellishments. This in and of it itself is not impossible, but most difficult for non-native Japanese speakers is not just learning formal language but using it appropriately. I was once out drinking with a friend who had studied quite a bit of Japanese in the States and could speak fairly well, but had only been in Japan for a few weeks. When we started talking to the group next to us, he completely broke the tone by standing up, bowing, and reciting verbatim the standard formal greeting that everyone learns on the first day of Japanese class. That's great when new business partners and textbook mainstays Tanaka-san and Smith-san meet for the first time, but not so applicable in a rowdy Japanese bar.
The danger for most of us who have been here too long is speaking too informally. As a token white guy you tend to get introduced to a lot of important people in rural areas; I've let some pretty rough sentences slip while making small talk with the local mayor, who, politician that he is, registers a mild shock, then proceeds to hug the foreigners and kiss the babies.
Keigo may just sound like a useless formality, but on another level it reflects a universal psychological reality. Although it's not laid out formally we have a host of these in English. My favorite is calling a company and starting off with "I was just wondering if..." Really? You were just sitting around idly wondering if there were any brake pads in stock, and once you find out, your curiosity can be put to rest? Japanese keigo is infamous for this kind of indirect language, but the only real distinction between it and formal English is that in Japanese they are laid out in neat little tables.