It is indeed because we have ten fingers. This is more obvious in some languages, in which the word for ten is clearly related to a word for fingers, hands, etc. I remember recently reading (perhaps in The Meaning of Tingo) about one language in which the word for eleven is something like "got to go down to the feet now."
AFAIK the word for five in all Indo-European languages (e.g., L. quinque, Gr. pentos, R. pyat) is derived from the root word p'enkwe meaning "finger." This word is still preserved in the Germanic languages; the relationship is most obvious in German fünf and finger.
The Basques--the only surviving pre-Indo-European people in Europe--may have vestiges of a base-five system. Their word for six, sei, is borrowed from Spanish seis.
French has vestiges of base-twenty. Ninety-two is quatre-vingts douze, "four twenties twelve," and ninety-nine is quatre-vingts dix-neuf, "four twenties ten-nine."
But everyone seems to use some multiple of five.