I think the official policy is that the program is for children under the ridable height, and what the cast member told you is correct.
I can only tell you my experience, which is my wife and daughter were always over near the stroller parking and Id walk up and tell the cast member I’d need a ride swap pass because my daughter couldn’t ride and that my wife was going to stay with her. (Sometimes I’d have to point to where they were, but they never checked her height) They never question me at all, they just handed me the pass and I walked with the ride swap. I did this at animal kingdom for avatar and Everest, Hollywood studios at rockin roller coaster, and at magic kingdom for space my and dwarves mine trail. This was in early December this year.