Yeah, Phantom players seem to only go online if you arrange it with them or post to say your going on, although you sometimes see a Phantom game pop up. Killer_Loopz and armageddon012 are two fast players who just appear on the servers at random times. Kappeskopp is very fast too (and a member here, but never posts).
The online quitters are a big problem in this game (for me anyway). Seems 90% of players are so afraid to lose, that they quit as soon as you get 3 seconds or so ahead of them. They usually then want to "race" you again!?!
You're ok racing members from here, gamefaqs and Arena I think. It's just the random players online who do it. Personally I deal with it by adding them to my Quitters List (which I'll post eventually), or hosting games using their PSN ID with some insult in it.