Right now im running stock gears 38/50 on a hyper with mods.

Since I didn't measure the diameter of the gear ill just use the tooth count...

say my engine turn 5000 rpms without exploding

speed 1 X diameter 1 = speed 2 X diameter 2

so 5000 rpms x 38 (using this as a ratio) = speed 2 x 50

so speed 2 is 3800 rpms say I measured my speed that day at 26mph

mudbuddy offers 46,47,48,49,50, 52 lowers and 38, 40 uppers

if I go to a 46 lower my speed goes to 4120 rpms at prop which should equal 28.2mph
if I go to a 40 upper AND 46 lower my speed goes to 4347rpms at prop and 29.7 mph

and of course I loose my low end torque the smaller my lower gear gets.

So has anyone actually played around with gears and ratios compared to speeds to see what happens?

Yes. Db8 has pages of info on his testing. You can also add diff props to the mix to really fine tune it.

Really the only thing you're trying to do is get the motor to run in its optimum rpm range with your normal load and prop. Whatever gearing you need to achieve this is the "right" setup for your rig.

What you have to decide is how do you want to set it up.

You did not account for prop slip