Almost everyone has experienced that. That's why the term "shadow people" is so popular, and why there are so many spooky hat-man stories all over the web. It's not just you.
I believe in ghosts no problem, mostly because I've experienced some really weird stuff myself and ghosts is the explanation that makes the most sense. But unlike many (probably most) ghost believers, I also firmly believe in the here-and-now reality we live in. That means also believing in scientific explanations.
When you see something in your peripheral vision, especially if it moves (which shadows often do), your instinct is to look directly at it. And then it's not there. There is a reason for that, and I'll explain it. No need to be indebted to me, I'll be happy if I can just calm you down. And no, the reason isn't some mental illness.
Peripheral vision is part of our animal survival equipment. Our field of focused binocular (both eyes) vision is pretty limited, in fact we can only focus on a small area at any one time. Our peripheral vision is much wider, but totally unfocused. Our peripheral (or corner of the eye) vision only picks up colours, basic shapes, and especially movement. If you want an illustration, look anywhere just beside your monitor and try to read the next sentence. Go ahead, do it, nobody is looking....
Couldn't read this sentence without focusing on it, could you? Your eye could see it, could probably even decipher that what it saw were typed words, but you had to look directly at the words to actually read them. That's how human vision works.
But vision is worthless without a brain to process the information, and your brain doesn't care that you couldn't focus with your peripheral vision. It knows you saw something, and immediately starts trying to match up what you saw with images stored in your memory. Basically, your brain says "no, I know we saw something, so what was it? It looked like a..." and then searches for a match. All this happens in less time that it takes you to blink, and you can't turn it off.
But when you focus on where it was, your eyes pick up nothing. That really messes with your brain, because your brain knows it saw something and doesn't know what. So it keeps replaying it over and over, trying to figure out what it could swear your eyes saw. Then, inevitably, our old friend the imagination kicks in. Our imaginations are what made us the dominant species on the planet, but they're also the reason we fear the unknown. And when your eyes caught a glimpse of something that isn't there when you look directly, it becomes an unknown.
If it was a real ghost, you would have still seen it when you looked directly.