Don't rely on the word of the bldg inspector. We just went a round with our local inspectors regarding house position on our land. He wanted 15 feet from the rear setback, whereas we want to put the house right on the setback. Turns out the 15 feet is just "staff policy" to minimize oversight, and they actually have no teeth to back it up. It's not part of code or law. Of course, they didn't tell me that when they rejected the site plan and told me to move the house 15 feet (resurveying at my expense). I had to lookup the county code, and let them know I knew the rules before they caved in. All to let me do what is my right. I lost respect for these guys.
In your case, if the land is zoned agricultural, I'd tell the building inspector to shove it. Then research the rules/ordinances governing land disturbance on agricultural land, and go from there. At least here in VA, land disturbance involves moving soil and/or changing the erosion/sedimentation characteristics of the land. So there's a distinction between cutting down a tree (OK) and digging out a stump (land disturbance). I'd think that mowing and cleaning up downed trees/logs would not count as land disturbance. You're not really impacting the erosion/sedimentation characteristics of the land.