I just got back from my trip today so I’ll share my experiences.
You are correct, and in fact, a third person who rode the first time around can go back on the rider swap pass. It is good for up to 3 people. When you go up to the line you will ask the CM at the head of the line for the rider swap and they will ask you who is returning for the 2nd ride. They will scan each of their magic bands and add the pass to their MDE account. If you look in your account you will see it under fastpasses.
With FP, I had FP for everyone in my party, but my experience was that If you had 3 people with FP+ and 3 without, you could go for rider swap and the 3 without a FP could get the rider swap to ride 2nd. The only reason the CMs seem interested in whether the first group is using FP is to set the correct return time. They give you a return time that starts after however many minutes the first group should be expected to wait in line. I personally did not test this, but what I did notice is that after we did the 2 rides, either my wife or I would still have a rider swap FP remaining on our account. What this told me is that when I went for the 2nd ride with the kids, my FP was “used”, leaving the rider swap FP still hanging around.
So, safe bet is FP for everyone, but my experience says you could get away with FP for half of your party and let the other half ride on the rider swap, so you could effectively double your FPs. I have read some reports around that CMs at FOP and other popular attractions wanted everyone to have FP, but I did not experience that myself.
The biggest benefit I saw with RS with 2 adults, 2 children who could ride and one child who was too small, was that the big kids got to ride twice every time! If you have anyone in your group who will want 2 rides, they can take advantage.