I spooled my baitcaster with quite a bit of fishing line and noticed when I was casting and reeling in it feel rough not smooth like it usually is.  So I took off some line and fixed it.  Did I over spool it?

What kind of line was it?

Any time I've overspooled a reel it will tend to over run and backlash.  If taking some line off fixed the issue, it sounds like you solved the problem.  

 

Unless your needing to make long casts, IMO a spool that is a bit low is easier to manage than one that is over filled.  All within reason, of course.  

Like the gentle above me said, if you overspooled it, it would backlash and jump out the spool on your first cast and every cast. Also it may be sticky.

38 minutes ago, TheReelDeal said:

What kind of line was it?

12lb Mono

Yep, the others are right you just overspooled it and got a slight backlash/overrun.

The general rule is fill to 1/8" of the spool lip, bait caster or spinning reel.