I know that this doesn't solve your issue, but I am running a similar system (i5 3570k, GTX 670, 16GB 2400Mhz ram), and my frame rate is fine. I never see a dip. Not in town, not in dungeons. I am also running the newest NVIDIA beta drivers. Might be system related after all.