Hi I am new to the mud motor world. I would like to build a long tail setup. I have a rebuilt rotax 503 rated at 50hp @ 6500rpm and 41ftlb torque @ 6000rpm.
Using direct drive what prop would I need? It will be used on a 17' drift boat.

Any sugestions or insight would be great.

You are better off running a reduction gear. 4 cycle's work better in this application because of better torque at lower rpm. I had a 503 on a Kolb and it is a bad ass engine for that application.

I agree, if you can work a out a reduction of somewhere between 1.5:1 to 2:1 that would be best. The props at very high RPM tend to loose efficiency. If you aren't running a reduction and the prop is turning 6K I'd say a 9X6. With the reduction maybe a 9X10 or 9X11. Please understand these are just estimates based on boat size and the torque of you engine. I've never see a boat with a similar set or even a two stroke at that. If you decide to do this build please post pictures or videos. We would be curious to see how it comes out. Best of luck

If you have a target speed use the equation (pitch in inches)/12 X (estimated max loaded RPM)X60/5280 to get MPH. This would be a condition with no prop slip. Slip is normally around 20% so multiple that the MPH by .8 to get a closer approximation to top speed.
Here is the example using a direct drive and a 9X6 prop for your rotax spinning 6000 rpm.
6/12 *6000RPM*60/5280=34.1 MPH then account for slip 34.1*.8=27.3mph
After seeing these numbers and considering the torque and HP of your motor maybe you could go up on the pitch a little. However I will say with a mud motor its much better to have a slightly under pitched prop compared to a slightly over pitched prop. Best of luck