“Metrics Matter” is an article written by Doug Gordon, former Director Collateral Valuation and Risk Management at Freddie Mac. Mr. Gordon led the Home Valuation Explorer (HVE) team, one of the most robust and tenured automated valuation models (AVMs) in the market.
Paraphrasing his research and paper, he argues that lenders must understand the Forecast Standard Deviation (FSD) of the models they use. Leading researchers have long argued that FSD is a key determinant of confidence in AVMs. Mr. Gordon states in his paper, “FSD is the best measure of expected accuracy of an AVM value in actual use.”
Mr. Gordon knows AVMs better than most people on the planet. Those of us advocating for AVM standards must acknowledge his experience. Any attempt to create standards must include transparent FSDs as the key determinate of confidence.
Two recent presentations at the Collateral Risk Network meeting, held at Valuation Expo in Las Vegas, provided insight into the growing use of AVMs and the challenges confronting further adoption and use.
A presentation by Fitch Rating, “Property Valuations as a Component of the Rating Process” discussed the use of AVMs to facilitate rating mortgage backed securities.
Following the presentation, I asked the following question: “Given that you are contending big data and artificial intelligence are much better today and you use FSD as a test to validate models, has your research shown that FSDs are improving over the years?”
The presenter’s initial response was, “good question,” followed by five minutes explaining how cool big data is and the importance of speed of execution. Of course, I had a follow up question, “I take from your response you have no evidence that FSDs are getting better, is that correct?” His response was, “yes, you are correct.”
Given models have not shown any improvement in FSD over recent years, while big data, computing power, and artificial intelligence is growing and improving, should clearly signal to regulators and policy makers that the science and math used to support collateral lending decisions to ensure a safe and sound economy are not as good as the AVM salesman would try to have us believe.
The second insightful presentation was made by two professors from the University of Northern Iowa and Mr. Lee Kennedy of AVMetrics. Per Mr. Kennedy, this presentation summarized the first of three papers to be written as a framework to help the industry begin to develop a consensus set of standards for AVMs.
AVMetrics is the current third-party market leader in testing the confidence of AVMs for lenders. Three bullet points from the slide “AVM Performance Metrics” are definitive on the current use of algorithms in housing finance:
- AVM performance metrics are not universally defined, nor consistently calculated.
- AVM users are often not able to compare the performance of competing AVMs.
- AVMs deliver measures of precision that often do not meet widely-accepted scientific standards.
These three bullet points are an indictment on the current use of algorithmic valuations in housing finance. AVMs need accepted scientific standards that are universally defined, consistently calculated, and transparently disclosed to all stakeholders. Until such standards are developed, AVMs should be relegated to quality control and low risk transactions.
It certainly feels like the wheels are falling off the institutional mortgage market push to use AVMs to replace appraisers for originating loans. Rumors continue to circulate that the Federal Housing Finance Agency (FHFA) has instructed Fannie and Freddie to roll back their pilots, pending further review of information. Given current AVMs do not use universal definitions, consistently miscalculate their forecasts, and are not able to report non-confusing FSDs, it is hard to believe FHFA will allow the pilots to continue. The marketplace needs a fair, transparent, and equal playing field.
Until a group of academics and industry professionals convene to propose and adopt AVM standards, the use of AVMs to originate loans should be stopped immediately. Their use is reckless, dangerous, and not supported by transparent econometric methods.
Stay tuned. Stand up. America needs professional independent appraisers. Under all is the collateral.
Have any comments or would you like to submit content of your own? Email comments@appraisalbuzz.com for more information!
