Our whole family is a huge Harry Potter fan (I’ve read the books through 4 times myself…mostly…well, book 7 is a chore and least favorite of the series, but that’s a topic for another day!).
Anyhow, our trip in February, we did all of Diagon Alley and Hogsmeade (starting from park opening) by 3:00 pm. You can read our full trip report here: Trip Report, Universal Feb 5 through 9
Now we were there when crowds were lighter, but I still think you can get through everything easily in a day since there are really only 3 rides (Forbidden Journey, Gringott’s, and Hippogriff). The train is kind of a ride (definitely an experience) and needs to be ridden both directions for a complete experience. The rest of the time will be “experiential”…that is, doing Olivander’s, exploring Knockturn Alley, etc.
You might even have time to do Skull Island or another ride. But if you start in Diagon Alley and then end up back there, you will want to pick rides in the Universal Side. Spidey and Kong are both on the IOA side. Our two favorites at Universal were The Mummy, and Rip Ride Rocket. When you exit Diagon Alley, head to the right to do those. Or, alternately, if you head right to exit, you can hit up MiB (which is a lot of fun) and The Simpsons.
Skull Island is probably one of my favorite non-coaster rides at either Universal or IOA, but you can easily experience it another day. While it was too scary for my wife, DD14 and DS8 to ride, my DS15 and I rode it twice.