In general, prepare for high crowds in Diagon Alley and Hogsmeade.
Since you are not using EPA, I would stay clear of Hogsmeade til after noon when it won’t be quite as crazy.
Bonus is that USF doesn’t have EPA so most of the early crowd will head to IOA Hogsmeade. So the earlier you can arrive at Diagon Alley in the morning the better.
Of course late evening is less busy too.
Since you may spend a lot of time in the WWOHP and your teens are Potter heads, this link to a detailed map of Diagon Alley is helpful so you know where you’re going (and where to find the entrance) ahead of time.