I'm looking for an older manga It's about this group of siblings trying to go see their families grave and they keep running into different situations like bandits etc.. all I really remember is that the baby was a master ninja and one of the other siblings was a swordsman.

Sounds like Niji Iro Togarashi to me...and even if it isn't the one you had in mind, it's still a great manga. 😀

"[English] not only borrows words from other languages; it has on occasion chased other languages down dark alley-ways, clubbed them unconscious and rifled their pockets for new vocabulary."
-James Nicoll, can.general, March 21, 1992

yeah i agree with tofuQueen, its amazing.