I am currently running a 9x6 on a 20hp, I see where I'm really supposed to have like a 9x4. Is it really that big of a difference?

The extra pitch will cause the engine to work harder to turn the prop. Especially at wot and when loaded and when in a lot of mud or vegetation. The engine won't build as many rpm which will cost you speed. Can lead to early engine wear.

If you have a tach check your rpms with your average load and underway. That will determine if you are running the correct prop. If they are too low, pitch down, and vise versa