I really enjoyed watching Puella Magi Madoka Magica and Happy sugar life,Wonder Egg Priority,three animes that i thought would be really cute and wholesome but have a dark/phycological and even horror theme.

So now I want to find more anime series/movies(or even manga) like these

Thanks^^

Well the most obvious is Higurashi no Naku Koro ni. It was originally a VN and there is an anime adaptation as well as mangas. I would suggest the anime unless you can stand really bad art in which case the VN is probably pretty good. I have heard many of 07th Expansion's VNs have a similar vibe although my only other experience with them is Umineko no Naku Koro ni which I would not say fits your request.

Kuro (Somato) Fit's pretty well I would say. It's not particularly deep, but it has the cute horror feel. Sadly it was axed, but I would still say what's out is worth the read.

Mahou Shoujo Site Would also be good I think. It has very mixed opinions, but personally I agree with Choror's review. And absolutely don't watch the anime.

Angel Game - Sayonara to Mirai no Kakera Also seems similar to me.

Voynich Hotel Might fit. It's cute, quirky, dark, and mysterious. It's also hilarious and just generally a good read imo.

Discommunication: Seireihen is also a maybe. It has a cuteness but is also weird and psychological. Not as dark though. As a warning it does have themes many consider upsetting or unacceptable. Basically one of the characters is a pedo. And it's been a while but I don't remember that He ever actually hurts anyone, but he's never punished for it either.

For anime:

Shinsekai yori is probably exactly what you want. It looks cute, but then... (This has a manga adaptation. Ignore it. It has nothing to do with the show and is just random porn)

Black Rock Shooter is more action, but I think there are many similarities. I've only seen the OVA and recommend it, no clue how the anime is.

Utena might also be up your ally. It's psychological and has it's dark parts. I think I lost out on some of its appeal due to not being a fan of normal mahou shoujo, but even if you are in the same boat it's still good. You have to give it a bit of time though, at first my main thought was "what the hell is all this rose BS" lol

Serial Experiments Lain is an amazing show that is very psychological and dark. It has less of the cute aspect, but still some. Also considered one of the best anime ever made by many, and I cannot disagree.

Hope I got what you wanted right. I wish there was more like the ones you mentioned, but it's sadly a small genre. That's why I included things that I'm not sure on or have issues.

The only thing that comes to mind that hasn't been brought up is Gakkougurashi/School-Live!

Don't know if you play games, or if you've already played it but it suddenly occurred to me that Doki Doki Literature Club fits perfectly.