If you have an air compressor handy, take off the line feeding your fuel filters from the tank and back blow it. then put a rag on the tank filler with your air and put just a small amount of pressure to push fuel back through the line and hook it back up. Do the same from the filters to the pump, only take off both ends. put the end on the filter back on and make sure you have fuel to the pump before reattaching the line.
It is my thinking you have a restriction in the line that lets it fill from the tank when sitting for a bit but will not let a steady flow through. I had a piece of float for the gauge get to my 90 degree fitting next to the filter. Would just let enough flow for an idle. When I would open it up it would sound like the injectors were missfiring. Mechanics gave up but gave me all sorts of advice like replace all the injectors and fuel pump. Sure over a grand for a small piece of plastic. Glad I didn't listen to them.
After re reading thought I might point out, about that line from the tank, take it off at the filter to back blow. Don't know your experience, just wanted to make it a bit more clear.