This will become a genuine question, but in order to locate the casual reader in context (and because I am a long winded bastard), it may take a while to get to it. I hope someone can help, nonetheless.

Suppose that G is a generic group, and where gh denotes an unspecified but legit. group operation. One may say that g acts on h from the left. One may also have that g acts on h from the right i.e. hg. We may not assume that gh = hg, i.e. abelianism for our group.

Now let's call the left action of g as L_g. Then by the above, L_g(h) = gh. And let's call the right action R_g(h) = hg.

Now note the following startling fact: one of the group axioms stipulates associativity, i.e. g(hk) = (gh)k. This implies commutativity of the actions of the form L_g \cdot R_k = R_k \cdot L_g for any group, abelian or not. How weird is that?

No matter, it simply says that the algebra of elements of a group and that of their actions are quite different beasts. This motivates the following.

A representation of a group G is its realization as the group of all linear transformations on a vector space V. This is called the general linear group GL(V), so we may have a mapping \rho: G \to GL(V). Now since g can be viewed as an action on the group G, I will find that \rho(g): V \to V,\; \rho(g)(v) \in V.

OK, so here's the question: one may also find a representation of G on the dual to V, that is \rho': G \to GL(V^*). Then obviously, for some \varphi \in V^*, I will have that \rho'(g)(\varphi) \in V^*.

So (dropping the argument on g), I am told that \rho'(g) = \rho^t(g^{-1}). Now I understand why the transpose is required, as it is a pullback on V^*.

But why do I need to take the inverse element of my group as an argument on the rep. map? Or is this only true for matrix groups?

Probably because the elements in the dual act on the elements in the primal, so sticking the inverse in somehow annihilate the multiplication by g in the primal side?

For me it seems more like convention, how is rho' defined exactly?

Well, I thought I had arrived at your "annihilation" line of argument first thing this a.m. (using the obvious involutions), but I was wrong. The modified version I don't like at all, it goes like this:

There is a natural isomorphism \rho \to (\rho^*)^*. So set (\rho^*)^* = \rho'.

Also, obviously, if g \in G, then (g^{-1})^{-1}. So that (\rho^*)^*((g^{-1})^{-1} = \rho'(g). And if you hand me \rho'(g), by symmetry I will offer you \rho^*(g^{-1}). And conceding SouthStar's point in another thread, I will have have that \rho^*(g^{-1}) = \rho^t(g^{-1}). But I don't like my first premise.

The other trouble with this is that, since \rho(g): V \to V \in GL(V), then the linear transformation V^* \to V^* \in GL(V^*) should be the pullback (\rho(g))^* = (\rho(g))^t.

Grrr...

Let v\in V, \varphi\in V^*, and g\in G. Denote by \langle\varphi,v\rangle the duality pairing. Then

\langle\rho'(g)\varphi,\rho(g) v\rangle=\langle\rho^t(g^{-1})\varphi,\rho(g) v\rangle=\langle\rho^t(g)\rho^t(g^{-1})\varphi,v\rangle=\langle\varphi, v\rangle.

In other words, the duality pairing is preserved.

On the other hand, demanding that

\langle\rho'(g)\varphi,\rho(g) v\rangle=\langle\varphi, v\rangle

for any v\in V, \varphi\in V^*, we arrive at the definition

\rho'(g)=\rho^{-t}(g)=\rho^t(g^{-1}).

One heuristic reason: If you define \rho'(g)=\rho^t(g), then it would violate the group multiplication operation, since transposing inverts the order of multipliers. So one needs one more operation of this sort: transpose or inverse of g. Transpose is already used, so use inverse.

temur I had forgotten about the natural pairing. So, though I liked your "heuristic" argument, I didn't get the one preceding it. Here is my lumbering version of the former.

Let \rho:G \to GL(V). Then evidently, \rho is a group homomorphism, and for h,\;k \in G one has that \rho(hk) = \rho(h)\rho(k) \in GL(V). Suppose that g = hk \in G.

Now suppose that \rho(g) = \rho(hk): V \to V maps u to v in V (Why doesn't "\mapsto" render correctly here?).
Now whatever I can say about the map \rho'(hk) = \rho'(g):V^* \to V^*, we agreed in another thread that it's a pullback, whereby \rho'(g) maps, say \varphi_u to \varphi_v.

In other words, under this map, the natural pairing \langle \rho'(g)(\varphi_u), \;\rho(g)(u)\rangle \ne \langle \varphi_v,v \rangle is FUBAR.

Recall I set g = hk \in G. To recover the natural pairing, I use \rho^t(hk) = \rho(k)\rho(h) \in V^*. But \rho(k)\rho(h) \in V^* has nothing to do with \rho(g)!

So, by noting that k^{-1}h^{-1} = (hk)^{-1} = g^{-1} \in G, I may finally write \rho^t(g^{-1}): V^* \to V^*,\; \rho^t(g^{-1})(\varphi_u) = \varphi_v and so I have \langle \rho^t(g^{-1})(\varphi_u),\; \rho(g)(u) \rangle = \langle \varphi_v, v
\rangle .

So, by that inelegant grinding of gears, the pullback is thrown into reverse, and sanity is restored!