I read the manga up to whatever's out, I think it covers maybe 3 Volumes of LN?

Up to where he's an adventurer and reunites with his dad

Rather like it but still find myself disliking the MC quite a bit, so hesitating to read LN.

Assuming he does, how far into the story does he stop talking / worrying about building a harem?
I know it's a harem story but seeing him lust after every female or so is rather offputting.

Also do his mother, sisters or daughter end up part of his harem?

Does they actually build up a romance between him and the girls that end up with him or is that just part of his harem building?

Does he remain calculating and manipulative for most of the story, or does that change at some point?
Like how he waited to save the adventurer in the forest and they ended up dieing (ch25 of manga, dunno if it was same in LN) or how he was planning to mold Sylph how he wanted.

And I assume he stays perverted for the entire series? Lol

Assuming he does, how far into the story does he stop talking / worrying about building a harem?

Soon, he'll have more important things to worry about. I don't remember that being so prominent in the early story either.

About his harem

there's no incest in it.

And I assume he stays perverted for the entire series?

Yes.

I'd recommend you to read the LN as the manga cuts short some things.