[Scenario]

You can always get that players virtual world, save yours into a variable, and set yours to the same as the other players. When you are done spectating, you can retrieve the information from that variable for your virtual world and set it to it again.