There should not be any fuel coming out under any circumstances, whatever you press.
What exactly is the "thing" that you press, it sounds like it could be the fuel pressure regulator, which if duff can cause all sorts of problems with fluctuating fuel pressure as the air fuel ratio will be constantly changing, but if pressing causes a leak it needs fixing like yesterday anyway, unless you are reffering to a little valve the same as you have on a car tyre tp pump up, as these are often the type used to hook up a pressure gauge, it has a "pin" type bit in the centre which if pressed causes the fuel under great pressure to come out.
If you haven't done so look at plugs, for tracking, a fave with Bosch plugs on these engines, for some odd reason, then if not already change the fuel filter for a new one, as no visual check will tell you anything.
tabetha