Kind of confused as to why.... "When I called the building department, they seemed not to care". The following clearly states that permits are required. Maybe back when you biult they weren't. Are you being taxed on the structure?
"A permit is required anytime that a new building is constructed, structural changes made within a building or additions to existing structures. Also, permits are required for electrical upgrades, meter changes, etc. Plumbing permits are required for new plumbing fixtures or for relocating plumbing fixtures. Mechanical permits are required for new equipment, change-outs, running a gas line, etc. For more information, contact the Codes Enforcement Division at (864) 467-7060."