One thing to mention up front is that since you're disturbing more than 2500 sq.ft, you will need to get a land disturbing permit (VA state rule) and that means running a site plan by your local environmental or building department and dealing with erosion/sedimentation countermeasures ($$$). Since you're potentially altering drainage to neighboring properties, I suspect the permit won't come easy. There may be an easement for that drainage area passing through the neighbors property, but that just means they can't block it. It doesn't mean you can do stuff upstream that would alter drainage to downstream properties.
I think it would be a mistake to fill in the swale. It's not just a drainage point for the tree area, it's a drainage point for all the land around it (whether there is visible surface water or not -- that's a common misconception). Filling it and putting a pipe in will only allow drainage from the tree area, and all other existing flows will just collect and turn your fill into mud and create problems later on. To really do it right, you'd need slotted pipe, stone, and drainage fabric, and that is a big project.
If i was me, I would leave that swale just the way it is and simply drop a short section (15-20') of pipe where the driveway is to cross over, creating a culvert there. You can get a 20' section of 12-15" black poly pipe for cheap, and it will support a cement truck with 12" fill cover. That way you are not screwing up the swale or altering any current drainage.
That size building will require a permit in VA, which means you will potentially be having inspectors on the property who can see other violations regarding land disturbance, drainage, erosion/sedimentation, etc. I had to get all my ducks in a row with that stuff just to build a 20x20 barn.
I don't want to sound like the sheriff, but I have become very familiar with all these rules after having built a home and barn, working with a developer, and becoming the land agent for our subdivision. I see a few red flags in your plan. My land is zoned agricultural (A-1) which makes life a little easier, but there are still plenty of rules/regs for non-agricultural use on that land, basically to protect my neighbors from me doing stupid things with drainage and land disturbance. In developed subdivisions with neighbors, things are even more restrictive because of architectural and property line rules.
Good luck!