Not at all, but lack of physical  fitness "can".  some man as they age may have a tendency to develop large suprapubic fat deposits that literally can "bury" part of the penis and get that impression that it's shrinking.
In some Asian cultures there is what's called Koro, where people develop a belief , magical or psychiatric in nature, that penis is shrinking and can dissapear. Can be treated by seeing a psychologist familiar with the culture.