Do we need the "secondary mortgage market"?

Secondary, as in a mortgage trading market, like the stock market. Fannie and Freddie started the thing and look how it turned out. Used to be, up until about 1980, that banks and savings and loans wrote mortgages with their own money. Since their own money was at risk, they used to check out the borrowers pretty closely to insure that the loan would be paid back.
Once a secondary market exists, the original lender can sell his mortage thus avoiding all risk. Which encourages writing riskier mortgages, 'cause once sold, you don't care anymore. Up til now the secondary market has encouraged a whole lot of people to sign mortgages too big to pay off, financed speculators buying houses to flip them, jacked up the price of housing, and financed building acres of houses that can't be sold. And pushed the entire economy over a cliff into the next great depression. Do we need this?
We could eliminate the entire secondary mortgage business with a simple law stateing that mortgages are non transferable. A mortgage is a contract between a single borrower and a single lender, and cannot be sold. As a mortgage holder I like this. I sign a mortgage with the XYZ bank, and that's that. I don't have to worry that XYZ will sell me and my mortgage to someone awful.
The banks, real estate people, builders, and contractors all claim that they need the secondary market to raise the cash to do the mortgages. In effect, the banks borrow the cash on the secondary market and then lend it out again. I say that banks can raise all the money they need by just paying decent interest on savings deposits.
Eliminate the middle man. Make the banks responsible for writing mortgages that will get paid off. Cut the price of mortgages and put all the secondary mortgage market people to work doing something of real economic value.