We have 310 million people in this country, but 4 year olds don't work, and neither do 80 year olds. People on disability don't work either. For most of our history, women weren't part of the work force either, many still aren't.
The total estimated work force in the USA is about 155 million (2010).
One can look at this as a ratio between working adults and the total adult population. The recession has obviously taken a big toll on the total number in the workforce. We're at the lowest level since 1984. But that's still a high 59% of the population.
The years after 1984 until the recession, we rode an endless series of economic bubbles, all based on overproduction. Cutting back may ultimately be beneficial.
One should note that in the late 60's we had lower work force to population ratio than we do now, and we had full employment, and GDP growth rates 4 times higher than it is now.