F=ma

The result equals Newtons, 1 newton is 1 joule?

If an object has a potential energy contained that equals x amount of energy.

The gravity applies a force on this object, a force that in the end is equal to joules of energy, are you not already measuring the magic?

example - a 1kg mass that contains x amount of energy potential, in a stationary reference frame on Earth , is subject to 9.81 newtons/9.81 Joules of energy loss compared to the original energy contained within the object.