Some additional thoughts.
You need a contract to work with a contractor. When it came time to build our house 8 or 9 years ago, I was amazed at the contracts that various contractors proposed. There was almost nothing in them to really specify many details about the house. I had architect's drawings, but much of these where what I would consider generalities.
One area where I was bitten pretty hard was in specifying concrete. They specified concrete psi, rebar number and thickness, finish roughness, etc. but what they didn't specify was flatness of slabs. Before the house was dried in, there was a lot of rain, and the slab for the downstairs floor turned out to have a 1 1/2" deep puddle almost everywhere. The edges were at the right grade, but the concrete guys did not use grade stakes, and the rest of the floor was dished. This probably meets commercial standards, but was not what I wanted. Live & learn. Now, I either do a slab myself or specify the number of square feet of puddles any 10 square foot section can hove and how deep they can be.
These days I would specify permissible slope of floors, both concrete and framed, both end to end, and at intermediate areas ( think waviness), top of windows all at the same height above the floor +/- 0.25" for adjacent windows, +/- 0.5" for the whole house. Walls must form square corners +/- 0.5 degrees, walls must be vertical +/- 0.25 degrees, etc. Notice, a specification without a tolerance does you no good.
Another big sore point can be how much time the contractor actually spends on your job. Get it in writing. Most of the actual work is done by subcontractors who are working with cheap labor and as fast as they can. These guys need to be supervised by your contractor, or by you the owner. If your contractor won't agree in writing to spend at least 10 hours per week at your job site, not just working on your project, get someone else.
We had a construction loan, and even if you can afford to pay cash for the house, a construction loan with a local bank is good to have. If you want to pay it off very early, that is OK. Our bank sent an inspector out to look over the house about once a week, and had a list of recommended contractors we interviewed. The bank was also very familiar with the laws regarding mechanics liens and material supplier liens. They made sure the contractor kept everything kosher on the legal end. They had a standard "release of all liens" form which every supplier and subcontractor had to sign before they got paid. The contractors knew they got a lot of business from bank recommendations and wanted to cooperate with the bank.
* * * * *
If you intend this to be the last house you are ever going to live in, there are some very uncomfortable things you need to address. 36" doors and walk-in showers are a start, but you need to go quite a bit further. At least one toilet on each floor has to have enough room around the toilet for both a patient in a wheel chair and an attendant. Similarly, at least one walk-in shower needs room for a patient and an attendant. You don't have to put up a sign saying this is what you have done, but the feature has to be there.