I hate it when I am watching a TV show and the main character walks in on someone just as they deliver the punch line to a joke and everyone around them roars with laughter, but neither the main character (nor I) got to hear the setup so “we” have no idea why that punch line is so funny. If you are like me, you spend the rest of the show focusing on what was so funny about that line rather than paying attention to the rest of the show.
I know the writer was using that scene to reveal how accepted that other character is by the group and how disconnected the main character is from the group, but this also illustrates another point.
Clearly the joke is hilarious, as evidenced by the roaring laughter. But without the setup, it doesn't seem that funny.
Without the setup, the payoff is missing something.
Actually, it's missing everything.
The 6th Sense, starring Bruce Willis, had the biggest setup (the entire movie) which resulted in one of the largest payoffs in cinematic history. Upon subsequent viewings of the movie, we realize that M. Night Shamalan gave us hints along the way. But he presented them in a way that kept us distracted from the truth until the big reveal at the end. If the subconscious hadn't been fed all the little clues throughout the story, the reveal at the end would still have surprised the audience, but not in a good way. And it would have lacked the "knock your breath out" impact that it had.
The same is true for your story. For every amazing reveal or twist that you create, return to earlier points in your story to create the setup that will give that reveal or twist a solid one-two punch. The reader subconsciously picks up on these hints and gains greater satisfaction when you hit them with the payoff.
Without the setup, your payoff is just another coincidence or event in the story that the reader glazes right over never realizing how awesome it really is.