You're correct. The Grandcamp contained a hold full of binder twine and some ammunition; it exploded first.
https://www.hsdl.org/?view&did=735172
I researched this back in the 70's when I was working in a research labortory; I was assigned a project whereby I had to test a certain crop defoliant for explosivity; Ammonium Nitrate was used as a control substance, because it's properties were pretty well known at that time. I didn't want to blow myself up, so I spent consideral time getting acquainted with the stuff. My company manufactured something called Sodium Chlorate...a rocket fuel...and was also used to spray on cotton to cause the leaves to drop off. When a fabric covered spray plane spontaneously combusted sitting on the runway, they decided to make it safe to use. To make a long story short, after several months, we came up with a formula that wouldn't set the world on fire.
My recollection is that Ammonium Nitrate is fairly safe as long as (1) you keep it clean and free of organic contaminants, and (2) keep it cool, and away from heat and flames.