HUGE HP fan here. You can absolutely spend an entire day doing only HP stuff. Yes, there are only three rides and the train (with another ride on the way!), but there are also two restaurants, five shows, several stores, spell casting locations, and so much detail that you need time to take it all in. Could you rush through it in half a day? Sure. Would a true fan do that? Never, if they had the option of taking their time.
I fall in the camp that says you start in Diagon Alley and “take Harry’s journey”. Explore, eat, enjoy. Here is a link to another thread where a similar question was posted. You may find it helpful: