It's down to personal choice and what you want to get from a game. I thought GT5 was really disappointing. They'd sapped every last ounce of fun out of the game in the pursuit of the 'perfect' sim. I was really close to buying a PS3 just for GT5… until I played it! If I wanted that kind of game I'd have bought an F1 sim.
Loved Forza 3 to bits though, thought it had just the right blend of gameplay and fun could play it for hours and never got bored of it – but Forza 4 is just awesome! It's like 3, but better in every area. There's no way (IMO) that you can compare GT5 to F4.
I don't actually have an Xbox, or PS3, and the last console I bought was a PS2, but every week we have a 'lad's night in' which involves a lot of Xbox. While we play a range of games, we always seem to go back to Forza, and now 4's out I doubt any other game will get a look in for quite some time.