Crushing student loan debt is making it harder for some Americans to buy a house. With the cost of a college education growing much faster than average salaries and inflation rates, it’s no wonder families are struggling to participate in the housing market as early as they’d like. Student loan debt is usually considered a good investment in the future – but new data confirms it’s one that can come at the expense of other important goals.

A study by the Federal Reserve Bank of New York shows that for the first time in at least a decade households with student loan debt are less likely to have a mortgage than households without debt.

According to a recent National Association of Realtors survey, half of Americans cited student loan debt as a “huge obstacle” when they want to buy a house.

To try to help stop this trend, student loan servicers are being targeted by lawmakers and regulators. According to HousingWire.com, borrowers complain that servicers are making it hard to pre-pay portions of their loans and charge excessive fees for service processing errors.

Richard Cordray, the head of the Consumer Financial Protection Bureau (CFPB), says the agency will investigate and fine servicers engaged in these practices.

The CFPB is also working with other regulators to provide incentives to loan servicers to provide more modifications and refinancing for private student loan debt.