Not knowing the full family dynamics etc are - which in some instances could necessitate an entire estate so that there is enough room. If everyone is fairly close (gets along etc) I would think Poly will be perfectly fine. As noted prior - they have larger rooms and the 2yr old will take up minimum space.
I do suggest that you try to keep the room as tidy as possible and make full use of the wardrobes, closet and put the suitcases out of the way to just have more room. For me I would much rather have 5 at POLY than a suite at AofA.
AKL is a wonderful resort - but the monorail has a special charm to it and you get the benefits of seeing the fireworks etc every evening if you wish. I think you will spend less time traveling and more time enjoying.
IF you plan more time at MK and EPCOT - stay at Poly. If you plan more time at AK or HS - then sure AKL.
I would however stay in those two as compared to AofA. AofA is perfectly nice - but can’t compare to the AKL or Poly IMHO.