Bigger can be a big drawback sometimes. Bigger:
-costs more up front
-costs more to maintain
-uses more fuel
-weighs more and therefore compacts soil more (if mowing or tilling)
-can't fit in smaller spaces, like inside a moderately sized barn, or a garage
-is clumsy to use for tight turns, for instance mowing inside a 1-acre fenced pasture (or a 10-acre pasture if you're talking JD's 9xxx series and similar)
-doesn't necessarily take less time if the size relative to the task makes it clumsy
The drawbacks of smaller are pretty obvious, I think.
So, yes, it depends! But picking the right size for the job, or for the most common job, is the way to go.
I have a small utility (bigger than a CUT). It's the smallest I'd want to use for moving round bales, but the biggest thing that will fit inside the barn where we store those bales. It's also somewhat bigger than I need for mowing, way bigger than I need for snowplowing, pretty much just right for the tilling I do, and about right for the FEL work I do (including bales). Something smaller would do most of my tasks, but would be scary or inadequate for round bales. Larger would be too big for the barn, and useable but silly for my mowing and snow tasks. So, I feel good that I got about the right size the first time.