I'm looking for manga where the character looks weak or nonthreatening, but is actually quite strong.

Two good examples are:

Holyland - Yuu's personality which lacks confidence and his lean build give people the impression that hes physically weak.

Hajime no ippo - Ippo gives off a harmless nice guy vibe. It would make one think "this guy couldn't hurt a fly".

I gave two examples where the characters were male, but female characters are welcome as well.

Feed your head.

Change 123 Motoko looks weak, and she is weak, but her other personalities...
Id Id looks like a girl, and no one believes, that he can fight, before they see it
Junkie Fiction Taku has the body of a kid, but he makes up for it, with the supernatural medicines
Übel Blatt Koinzell doesn't look weak exactly, but nowhere as strong as he is
Nurarihyon no Mago Rikuo in his human form
Ai wo Utau Yori Ore ni Oborero! Akira looks weak and cute, but he is very strong

"Thus is our treaty written, thus is agreement made
Thought is the arrow of time, memory never fades
What was asked is given, the price is paid"

I think the male protagonist of Taboo Tattoo looks rather weak.
Maybe SWOT works as well, though he does have red hair in serialization.
Yanki-kun to Megane-chan in regards to Megane-chan
World Embryo works well, since the main character is both weak-looking and doesn't have a particularly strong personality though he shows surprising resilience and cunning...
Tiji-kun!, which is basically a shounen version of Holyland.
Shingetsutan Tsukihime also has a bespectacled hero with serious kick-ass powers.

En Passant
Law of Ueki

I second Id!!

Hayate no Gotoku almost every single character is strong but doesn't look like it, especially isumi

in need of romance?
uncommon, sad, sweet

Eclipse Hunter (Novel) - he's a kid with a small build, but his body is actually robotic underneath, and he's very powerful.

Dragon Ball - anime is full of fillers, manga is epic.

The Legend of Sun Knight / The Legend of Sun Knight (Novel) - he looks to be a weak guy who can't use a sword despite being a knight, but he's secretly very skilled in magic and taking care of things behind the scenes.

Aside from what the others previously mentioned, Asamiya-san no Imouto?

She physically looks naive and weak looking but is really strong.

"Shouldn't be alive but the condom broke."