TLJ87, I've unfortunately been through exactly the same scene. Not fun. I had a contractor pull similar work hours and minor progress on a workshop build. We'd see those 6 hour irregular workdays, vacation in the middle of the project, and wanting progress payments before milestones were met.
The saving grace was that I demanded progress payments with milestones detailed, and major final payment in the contract. The contractor wanted to quit long before the building was completed. I told him fine - no more money. It was finished when I said it was finished, not before.
Sure - he took me to court. I won. I had plenty of pictures of re-do work, poor craftsmanship, etc..
My Lesson Learned:
Disregard anything the contractor says if not explicitly written in the contract. Get very detailed in what and how, including material specs. That "My word is my bond" crap isn't worth the air he uses.
Set specific start date, milestone dates, and completion date with penalties. (Expect a fight over that, but insist!!!)
Establish that all modification, additions and deletions must be documented with Field Change Orders, signed by both parties.
Understand - everybody is all 'buddies' before the work begins. Once the first shovel is used, it has changed into a business contract.