Plus over a long time span Earth's density and structure does change. Internally things flow around, continents move, and the water of course always sloshes around in the ocean basins, although this compared to the others is a brief consistant action and may not factor in. So an irregular shape will find its center of mass, but Earth is not a stable irregular shape. How much affects what is the question, and does it matter that much to us? Could the Earth's axis violently change, or is it by necessity a slow gradual thing? I think the possible polar switch will be a lot more impactful that the actual polar axis drift.
