You have a problem with the fuel tank ventilation system. Normally you wouldn't have to undo the fuel cap under any circumstance.
In my opinion 3 or 4 miles is probably not enough to build up pressure, but it could be tank is sucked vacuum and the air inlet is blocked.