Apparently if you drop shadows from Ultra settings to something lower it fixes the problem.
The Person who reported this said all other settings can remain the same.

QUOTE:
UPDATE: I installed the latest driver (v457.09) and it didn't help much. The FPS wouldn't drop quite as low, however it still happened on most tracks.

I dropped all settings to low, and I no longer had the issue. So, I bumped all settings back to Ultra and dropped each setting to Low one at a time to see which setting might be causing it.

Seems like it was Shadows causing the issue, as I no longer got any fps drop or stuttering when looking behind me with the setting lowered. I can even run the setting on Very High without having any issues, so I guess it's just the Ultra setting causing me issues specifically.