Generally from your visual and vestibular systems sending conflicting information. I've heard people argue over the reality of "getting your VR legs" (as in, sea legs), but the concept makes sense.
Either way, don't power through it. Otherwise you'll come to associate VR with nausea and never get used to it. Take a break at even the slightest discomfort.