That looks like a packed and fun day! I have only been to Universal a couple of times and the Harry Potter areas once so I don't know too much.
I did have luck doing the Castle tour and riding in the Single Rider line for Forbidden Journey. You really can't see the person next to you anyway so you don't miss being together. I also did Gringotts at the end of the night and was able to walk out of an almost empty Diagon Alley which was really cool. If you are going to get a wand I would do that first before exploring the area so you can do the spells - it is really fun. Also make sure that you have enough time for ButterBeer (frozen or ice cream version).
Although you certainly don't have to do any rides - I think it is nice to walk though the Dr. Suess area of the park. I think it is so charming and fun. If you are running behind I would cut E.T. - although it is really cute. It is not as cool as I remember when I was kid. It is a little out of the way too.