It doesn't work like that though. You'd return to the exact same point in time. At least, people have gone back then returned and it was like that for them.

...I don't think you'd miss them, if that's what you're worried about.