I've owned a straight six and it was a truly epic car. Really want another one day. It had a knack of making you feel incredibly special, and even at 14 years old, it still turned heads. But I'd say the V8 is a vastly improved car, even if the looks don't show it. Nowadays the six's suspension really shows it up - it feels too firm to be a luxury car and too weighty to be a sportscar, so it sort-of sits unhappily in the middle. The V8 was fitted with CATS (Computer Adaptive Technology Suspension) as standard, so does a much better job of fulfilling both wafty ride quality and sporty handling in one package. Don't believe all you read about V8 problems - yes, they did have many, but nowadays the consensus is that if it hasn't broken by now, it's not going to. The three areas you really need to watch with the V8 are the Nikasil issue (get a blow-by test done on it before you buy, and if all's well, it shouldn't be an issue), the timing chain tensioners (these were plastic from the factory but break easily; steel replacements are available so it's strongly advised to replace them if they've not been done already), and the Merc-sourced auto box (when Jag sold the XJR they said the box was sealed for life and that the ATF would never need replacing; curiously, when this box was supplied in Merc's own cars, though, they specified a 60k fluid change interval. If the fluid's not changed the box has been known to throw its toys out just over the 160k mark, so if the fluid has never been changed, get it done as early as possible). Keep these things in mind and the V8 should be no less reliable than the six, and it'll be a much more rewarding car to own and drive.
As already said though, either model is mind-bogglingly juicy, plus of course maintenance isn't cheap (especially if you plan on keeping up a full specialist history), so deep pockets are essential!