We did USF on a CL4 day (in a tropical storm) and IoA on a CL5. Absolute pleasure with EPs, and much like staying on-site at WDW, we wouldn’t do it again without them (this is in the context that we did everything on the trip on a budget - a single night at RPR for EPs was worth it).
Sure, we’d (probably) have got to see everything (and we did see everything - except the DA shows due to rain) without them as the opening hours are quite generous, but just being able to walk onto rides like Hulk, RRR, Mummy or FJ twice in a row while there were substantial lines, and of course to skip even small lines for everything else was a huge bonus.
EfG, Kong and DDR in particular had moderate waits (20m+) even with EPs, so I wouldn’t like to try it without. If Hagrid’s, or the new raptor coaster gets EP this will be even more of a no-brainier for us if we had limited time there.
However (after all that!), with three days, two at low crowd levels, I’m pretty sure you will be fine without if that’s best for you.