No, if you are using concrete or gravel a larger hole is necessary because the concrete/gravel can then distribute the forces over a larger soil surface area. The mass of the concrete or gravel also acts to hold it all together, of course, so the larger the hole the more benefit you get from that. The only time you really want a hole slightly bigger is if you are replacing the soil, since it is difficult to meet the original compaction and you would have a lot more to do. I have a 5' wire fence around about an acre that I drilled, and even compacting the holes again in layers they move around much more than the posts that I pound with the post pounder.