I don't know the story but I still remember the ending. The ending is a time skip and MC is an old man already and is sitting at the balcony in some sort of castle is going to die soon but in his side are his granddaughters and grandsons, then MC died and MC turned back into his original self before time skip is finally reunited with the main heroine and all the harem members in heaven.

Yonakano Reiji ni Harem o!!