I agree with the posters stating that it is not a big deal to take off a non quick attach loader. It just takes more time. There are many impliments/attachments that are used on real farm machinary and are not quick attach. They are made that way for a number of reasons but mostly beacause they are only added or removed once or twice a year. If you really don't beleive you will need to be removing the loader on regular bases for some future use then don't let the loader be the deciding factor.
The only thing that may be a consideration is the resale value. If you don't think you may be selling trading it after a few years, it may be harder to sell without the quik attach loader.