What Mathematics to study for quantum mechanics?

Calculus, probability/statistics, differential equations and linear algebra. Then group theory for the more advanced stuff.

That being said, there's a lot to be said for just buying an introductory textbook and reading it. You should be able to see what you don't know, and you should be able to find references to teach you what you need.

A good rule of thumb is this: you should be able to pick up a textbook, thumb through it, and see what you know an what you don't know. Then you should be able to decide the best way to learn the stuff you don't, and you should find another book and repeat the procedure. If you get more than one book away from your previous book, you're not ready to read it yet. This is a small but essential piece of the process of getting a PhD---if you can't go through the process as I've described, you may find self study hard. And you should really _understand_ what you're reading---I spent 4 hours on Sunday reading six pages of a finance book that I have. Not because I didn't understand what I was reading, but because everytime I thought about it more, I understood more. The process of digesting a text so thoroughly that you're picking apart individual words is really what it takes (for me at least) to really _understand_ things. You have to wring every bit of information out of a book like a wet towel---the good books are the ones that are never dry.

Vector calculus, linear algebra, group theory (basic), Banach/Hilbert spaces, probability (basic, I'm terrible at it), Fourier analysis, PDEs.

If you want to do quantum field theory then you'll need the above as well as electromagnetism, electrodynamics, special relativity, functional analysis, complex methods.

I did a maths degree and initial quantum mechanics was a 2nd year course, basically differential equations and linear algebra. Dirac notation stuff came up in the third year course, the applications of quantum mechanics is another 3rd year course and then quantum field theory in the 4th year. As Ben says, don't expect to jump into the 'cool' stuff, there's a reason every university in the world requires various courses to be covered before getting to the cool stuff, they are required to understand the cool stuff. It's a long slog but well worth it. It's just a shame the number of people willing to put in the effort to gain understanding of physics is so small compared to the cranks who want to talk the talk but not walk the walk (I'm looking, metaphorically, at you Farsight).

Multi-variable calculus and linear algebra are what you want to have done before the 2nd year uni level quantum mechanics AN mentioned. You won't really need the rest for quite some time after that. QM would be a piece of cake if you did know all that other stuff in advance, but nobody does when they first start.
Oh, yes and the high-school level probability is pretty necessary so go back and actually learn it :). In fact, everybody should go back and learn that properly; in the last few years I have come to understand that quite a lot of the world's problems arise because people don't understand probability theory and statistics.

Today's Abstruse Goose (http://abstrusegoose.com/342):
http://abstrusegoose.com/strips/spin_is_just_a_myth_like_evolution_and_the_female_ orgasm.png
It's a sequel to a much older one (http://abstrusegoose.com/93):
http://abstrusegoose.com/strips/moment_of_clarity.JPG

A good rule of thumb is this: you should be able to pick up a textbook, thumb through it, and see what you know an what you don't know. Then you should be able to decide the best way to learn the stuff you don't, and you should find another book and repeat the procedure. If you get more than one book away from your previous book, you're not ready to read it yet.

How deep does the rabbit hole go? (a short thread, but a good one (says the thread starter :)))

Edit - fixed link. Thanks Kenny!

Bad link...