The magician's curse prevents him from using magic and kills all his party members. I believe his entire party was killed 5 times. Because of this he chooses to travel "alone," with his talking sword, which is what he uses since he can't cast spells. Some new adventurers join up with him because they don't believe in the curse - and it finally looks like things might go his way.

I don't think this is exactly it, but maybe Yakushoku Distpiari - Gesellshaft Blue?

Hmm, nope. It's not hidden from his party at all - they're just kinda headstrong.

Bumpity

Bumpty

Ah man.

Wizardry Zeo?

Holy shit, that's it! Thank you so much!

I couldn't figure out the name because the one I was reading originally didn't have a translated title. Instead it had a bunch of weird accented letters I had no way of typing - never mind actually remembering them. But as soon as I saw the first illustration of the main character I knew this was it. Great, woo hoo!