How you describe it - the problem is the one way valve on your fuel pump.

When the engine is hot, everything under the bonnet is hot too. Fuel lines are constantly cooled by fresh fuel from the tank. If you turn the car off, the fuel lines will get very warm. The fuel is getting air bubbles. If your one way valve is not working these bubbles will be getting very big and push fuel back into the tank. You're not able to start for a long time because wehn the pump is making pressure the bubbles are compressed but won't push the fuel behind them because they "suck" the pressure when they are compressed.

When you leave the car for a few hours everything will cool down, the air bubbles will vanish and you can start your car without any problems.