In the past four years, the number of adult Americans who aren't working and who aren't looking for work has swelled by six million, pushing labor-force participation rates to the lowest levels in 16 years.

Now that the job market appears to be improving, some economists believe it is time for labor-market dropouts to drop back in again. "If workers are having some sense that the labor market is starting to pick up you would expect them to come back into the labor force in larger numbers," says Harry Holzer, an economist with Georgetown University and former chief economist with the Labor Department.

But it may not work out that way. That is because a mix of demographic and social trends has been at play in driving Americans out of the labor force, and those same trends could keep them out even as the economy begins to strengthen.

Many of the workers who left the labor force in recent years are the much-talked-about discouraged workers, people who had such a difficult time finding work that they stopped looking. These are the individuals whom many economists expect to flood back into the labor force in the months ahead.

However, a large number of people who have left the job market don't fit neatly into that category. Students, for example, are spending more time at their studies, rather than concocting dot-com dreams in their dorm rooms or taking afterschool jobs. Other Americans are attending to responsibilities at home, retiring early or collecting disability benefits. While the weak job market had undoubtedly been a factor in their decisions, it wasn't the only factor.

The bottom line, some economists say, is that the labor force might well grow at a slow pace even if job growth picks up and even as discouraged workers return.

This has important implications for everything from how fast the economy can grow to how many jobs it needs to create to hold down unemployment. Economists generally believe the economy needs to create about 150,000 jobs a month to keep up with the addition of new entrants to the labor market. But in a new period of slow labor-force growth, the threshold might be 100,000 or below, according to a recent study by the Federal Reserve Bank of Atlanta.

That might sound like good news because it means less competition for each potential worker. But it's not all good, says Dale Jorgenson, a Harvard economist who specializes in something called growth accounting, which breaks down the engines of economic growth -- most notably productivity growth and changes in the labor force.

While productivity is booming and boosting economic growth, a slow-growing labor force will hold back the economy's overall growth rate in the years ahead, he says. That means less consumption and less investment than some might expect.

Some private economists believe the economy can grow at a pace of nearly 4% in the years ahead, but Mr. Jorgenson says the growth rate will be just a little more than 3%.

"We're not going to have a return to the 1990s and we better live with it," he says.

In a new set of calculations, he estimates the number of hours Americans clock at work will grow by 0.7% annually in the next 10 years, a sharp slowdown from the pace of roughly 1.5% growth that prevailed from the 1960s through most of the 1990s.

Such a slowdown has long been foreordained, because baby boomers are creeping closer to retirement. But what is surprising is that it is potentially happening now, before the oldest boomers have reached their golden years.

Labor Department statistics, for instance, show sharp increases in the number of women, age 25 to 54, who are out of the labor force and attending to responsibilities at home. This number was declining through most of the 1990s, but then started to rise again in 1999 -- before the recession set in. Between 1999 and 2003 their ranks jumped 13% to 8.8 million, according to data compiled by Labor Department economist Steve Hipple.

"We're at the end of an historic period where female labor-force participation rates had been steadily climbing," says Mr. Jorgenson.

Teens and young workers in their 20s are another driving force in this trend of slow labor-force growth. A recent study by economists at the Federal Reserve Bank of Chicago found young Americans behaved differently in the recent downturn when compared with the downturn of the early 1990s.

Back then, an increasing number of teens and college-age students chose to go to school and work at the same time. But from 2000 to 2002, on the other hand, they stayed in school and moved out of the labor force in large numbers. The authors -- Daniel Aaronson, Ellen Rissman and Daniel Sullivan -- argue young workers are potentially responding to the higher wages that are being earned by workers with high skills by dedicating more time to school and less time to work.

Meanwhile, the number of adult Americans who say they are out of the work force and disabled increased by 18%, or 1.8 million, from 2000 to 2003. The number of people ages 25 to 54 who said they retired increased by 27%, or 234,000.

It all adds up to a labor-force slowdown that has been much more complicated than many economists believe. The wild card in whether this slowdown will persist is older workers. Surprisingly, the oldest workers -- those 65 and over -- are the ones whose labor-force participation rates have picked up the most in the past few years. They are healthier. And if Social Security reform entails an increase in the retirement age -- as some argue it should -- they might find they have to work longer than they expected.