Gotcha. So that's your garage on the other side of the fence. Bummer. That's one of the problems with being allowed to put buildings so close to property lines. Your neighbor can do the same thing, then neither of you can do maintenance.
I'd imagine if you had to do maintenance, you might be able to force them to allow you to take the fence down temporarily while your repairs are made, as long as it's put back up undamaged, in like condition, and in a time-line.
At our first house, right after we got married, we got a dog. I wanted to fence in the yard and their garage was like yours. The neighbor, an old guy that had lived there about 30 years, showed me the property line, a post near his driveway, which was about 3' south of the garage. I had the fence company located the actual pins. It was 2.5' further north, putting the property line 6" from his garage. Had I fenced that, he'd never have been able to take his trash out to the alley, paint the side of his garage, etc... Since I liked them, I showed them everything and just put the fence 3' in on my side, effectively gifting him the 3'. It wasn't worth losing a good neighbor over.
In your situation, it would have been nice if they had discussed it with you first, and talked about making removable panels (really easy with planned brackets and removable hardware), but it is what it is.
Hopefully, their increased property values will also increase yours and you can benefit from it in the long run.