I had a rack style system that I just took out, because my steering felt like it was out of power steering fluid.  I'm looking around for a new one, but wasn't sure if I should go with another rack or go for a rotary style system.  Which is better and why, thanks!

I know that you can upgrade to a hydraulic steering system. That is what my boat has and it is smooth as can be.

I've used both.  The rotary NFB seems a little more solid and takes up less room behind the dash than a R & P NFB system.  It's also a tad more pricier.  At least it was the last time I priced steering systems 4 years ago.  

As stated, if you've got $1000 to spare, a hydraulic system is the best way to go.  

Complete Sea Star system: $759 - probably could be found for less.

rivermarinesupply.com/xcart/catalog/product_10779_Teleflex_Seastar_HK6400_Hydraulic_Steering_Kit