I would distinguish between the casing, aka the pipe between the surface and the water table keeping the soil out and the screen down at the bottom keeping the sand/gravel in the aquifer out of your well casing and pump. If you have sand or gravel in your water, it could come from a failed well casing, or it could come from a failed/failing screen. Since screening is pretty light weight, my bet would be on the screen, but it certainly also could be the well casing. The screening is not that hard to fix. Fixing the casing is more complicated, but not undoable, though it is often cheaper and easier just to drill a new well.
To answer your question about the rocks and gravel, once the screen gets a hole, the flow of water will continue to suck sand and gravel in. It is important to remember that in water, rocks and gravel have a lower apparent density due to the displaced water, making them easier to transport. The ability of water to move sediment goes up with the velocity to the sixth power (more or less), meaning a small hole in a screen will be extremely effective at spraying sand into the well casing. The low pump pressure suggests to me that your pump impellers have eroded, probably from pumping the sand and gravel that you have been finding in your filters.
$40 will get you a 16' inspection camera that would let you check what is under your well cap...
Well drillers can, and should in my opinion, have a great knowledge and experience, but speaking from personal experience, it can really vary. I've had guys who not only didn't understand the importance of galvanic isolation, but also could not tell brass from galvanized in their own gear. (A locker of plumbing fittings drifted a foot deep, not kept clean, and not sorted by size, or material. I had to find them the right pieces out of their pile. Even a couple of buckets would have gone a long way.)
Ever heard the one about the well drilling company that wouldn't buy all wheel drive trucks? They didn't want their crews to get too confident and get stuck. I.e. leaving the landowner to tow the two wheel drive truck into position...
And for what it is worth, I'm a fan of VFDs, but in my experience lots of folks dump on them without understanding them in detail, and certainly not understanding the ins and outs. I don't get it. I feel like it is another "well my brand of truck/tractor/whatever is better than yours, period, no matter what you say or the facts are". Are VFDs as simple as a pressure switch? No, but that is the pro and con. Are there bad ones out there? Of course! But there are reputable brands and vendors, too. I think it is illustrated in the following effect;
(Notice that I have opined a bunch here...)
All the best,
Peter