Zestimates: home valuations for Mr. Spock.
"Data! I need more data, Captain!"
The Internet's biggest advantage is that it dumps data right in your cubicle or spare bedroom.
The Internet's biggest disadvantage is that it dumps data right in your cubicle or spare bedroom.
Data ≠ knowledge. In fact, data < knowledge.
I went a long time without feeling the need to check the accuracy of Zillow's famous (or infamous) Zestimates, but I'm glad I finally did. Because the results are both enlightening and predictable.
Here's how Zillow defines a Zestimate:
"The Zestimate (pronounced ZEST-ti-met, rhymes with estimate) home valuation is Zillow's estimated market value, computed using a proprietary formula. It is not an appraisal. It is a starting point in determining a home's value. The Zestimate is pulled from data; your real estate agent or appraiser physically inspects the home and takes special features, location, and market conditions into account. Variations in price also occur because of negotiating factors, closing costs, and timing of closing. We encourage buyers, sellers, and homeowners to supplement Zillow's information by doing other research such as:
Sounds innocuous enough—and not by accident—although the obvious question is what Zillow thinks its Zestimates are worth if it encourages buyers, sellers, and homeowners to "supplement" them. What good is a "starting point" if it starts you at the wrong point? How does a Zestimate offer anything new, when all it does is send you to the same old sources?
Or is all this becoming modesty just lawyer-crafted nudge-nudge-wink-wink? A cynic might say that Zillow knows full well that few of its users will go to a home valuation expert once they get their hot little hands on a Zestimate—after all, isn't Internet empowerment the whole idea?—but that Zillow also knows full well it'd better pay lip service to the idea that Zestimates have limitations nudge-nudge-wink-wink.
Zillow is a new breed of real estate Web site, a chastened breed that's needed only ten years or so to figure out that you don't make money off the consumer's crush on real estate by crashing through the industry's showroom window, declaring "this changes everything!" and offering online bling of dubious real-world value to titillate a handful of geeks. Or has it? At least that's Zillow's official line towards the real estate industry: "peace, brother". Let's see how long this uneasy truce lasts.
For several years Zillow and this agent got along just fine, each oblivious of the other's existence. Then, a few weeks ago I meet a buyer for the first time and, as usual, proudly give him the Comparative Market Analysis (CMA) I've compiled for the house we're standing in front of. A CMA is an agent's opinion of the value of a house, based on and showing comparable active listings, pending and recent closed sales, as well as properties taken off the market unsold. It's an example of market "transparency", to use a word the Internet works hard. I do a CMA for every home my buyer clients see, and I'm proud of this service—I don't know of any other agent who prepares CMAs for buyers. Usually it's just sellers who get them, because CMAs take lots of work. I do ten or twenty a week and each one takes about twenty minutes.
So I hand my CMA to the buyer with a flourish, explain what it is and what it does, step back and expect him to gush. He does. Kinda. He exclaims, "Oh! Just like Zillow!"
No, I say. It's ten times better. I don't know why I say it's ten times better and not five times or 23.7 times better, except that anything handcrafted by an expert has got to be maybe ten times better than raw data cranked out on a virtual assembly line.
Then I forget about it. Zillow and I lapse into peaceful co-existence.
Until a week ago, when the same buyer emails me about a home he's just seen at an open house. He'd like my opinion on the price, and he'd like me to get any reports the sellers have. Fair enough, but there's one catch: I can't find the house. It's not in the Multiple Listing Service, at least not as an active listing. What's the price? Who's the agent? What reports do he or she have? How do I get them? The only MLS entry I can find is an old listing taken off the market, unsold, months ago.
I resort to Craigslist. Not there. Then I do what most people do when looking for a needle in a haystack: I look on the Internet. I search by address and, for some reason, up pops a link to a Zestimate for this very house. Intrigued, I click on it. The Zestimate says it's worth $1,137,500. Remember, this house had been on the market, without selling, for 178 days—at a list price of $959k.
This bears investigating. I decide to do four comparisons. The first will see if Zillow can shed light on the current market value of three homes this same client has seen. Why? Because the homes are in a city and price range that hasn't seen much sales activity lately. When sales slow to a crawl, valuing homes becomes even more challenging because you work with few and sometimes no recent comparable sales ("comps"). Sure, you can check macro trends—see how much sales prices have gone up or down in a larger geographic area with more sales—but one of the great things about the mid-Peninsula—and the bane of anyone trying to value a home here—is that almost every city in this area is a collection of widely differing micro-neighborhoods. And even if our neighborhoods were uniform, stamped out cookie-cutter style, values would still vary due to the perceived quality of the school districts, and even between schools within the same district.
Then there's a neighborhood's proximity to shopping, and what kind of shopping is it? A handful of beaten-down strip malls, or an attractive destination downtown? Then there's location. Is the neighborhood next to a freeway? Next to train tracks? Next to a much better area? Next to a much worse area? What's the hot topic in that city or neighborhood? In one city it might be allegations of petty theft by the directors of one of dozens of city-sponsored programs. In a nearby city, so close it shares a zip code, the hot topic might be a spate of gang-related shootings.
So I need a little help here. Let's see if Zillow is up to it.
After that, I'll attempt three more straightforward comparisons. I'll simply compare the sales prices of single-family home sales closed since April 1, 2008 with Zillow's Zestimates for those same homes. To test Zillow's accuracy in a range of markets, I'll use one market that's seen home prices rise recently, Palo Alto; one market that could best be described as "mixed", Burlingame; and one market that could best be described as "hammered", East Palo Alto.
Let's see how those Zestimates perform.
Here's how well they predicted the sales prices of the latest twenty-eight closed Palo Alto SFR sales.
Wowsers! The chart doesn't even begin to convey the discrepancy between Zestimate and reality. Of the 28 data points, eighteen differ more than 10 percent between Zestimate and actual sales price. Five of those eighteen are off by 20 percent or more. And when you're dealing with an average sales price of $1,637,142 (and 86 cents), 10 percent here and 10 percent there and pretty soon you're talking real money.
I will admit that Zestimates absolutely nailed the price twice, within .04 and .2 percent. Nothing to complain about here. So we can say that, based on the Palo Alto sample, Zestimates are deadly accurate 7.1 percent of the time. That's 7 point 1 percent, not 71 percent.
And six Zestimates were within 5 percent of the sale price. That's close enough for government work, don't you think? You wouldn't mind paying $1,765,500 for a home you could've gotten for $1,700,000, would you? Thought not. Or paying $1,728,000 for a house that otherwise would have sold for only $1,667,000. Or bidding $1,447,500 and not a penny more for your dream house and seeing someone else snag it for $1,505,000. At least you didn't shudder overpay!
Okay, so maybe Zestimates can be a little—okay, a lot—off the mark in a rising market. Let's see how they do in a fairly stable market, Burlingame.
Much better! Zestimates nailed the sales price of two of ten data points. That's 20 percent accuracy, and for those of you who aren't into numbers, 20 percent is nearly three times better than 7 percent. And that third data point, off just 1.7 percent, ain't too shabby either. Of course, if there were multiple offers on that third home (and there were—three—including one by my client) and you really really wanted the home, and it was a great home, expensively remodeled and on an unusually large lot and with good schools and close to downtown and okay okay I'll stop, and you offered a Zestimate-correct price of $893,000 and it turned out to sell for just $15,500 more and you calculated just how little that extra $15,500 would've cost you amortized over the life of a thirty year loan at near-historically low interest rates...well, you'd feel fine about that, wouldn't you.
What about the other seven data points? Zestimates have you offering 32.1 percent more than the actual sales price, 29.6 percent less, 20.1 percent more, 19.9 percent less, 9.7 percent more and 9.5 percent less. Hey, I see a trend here: just average all the percentages the Zestimates are off by and you'll get the correct market value!
Okay, so maybe Zestimates do better—but far from consistently great—in a stable market. How about a market where prices are dropping off the table, East Palo Alto?
Eh? Not so good.
East Palo Alto brings up an interesting point. Zestimates give you data, all right, lots and lots of data. Click on "comparable homes" and you'll get a list of fifty or so "comps". Data! I need more data, Captain! Except that when calculating the value of a home, data can just as easily lead you down the slippery slope as down the right path. I'm selective about the comparables I use, and there's good precedent for this. Look at a professional appraisal (as Zillow urges) or something much the same, a Broker's Price Opinion, and you'll see just three comps there or, lately, as many as six. What you won't see is fifty comps, because you don't need fifty comps, especially if forty-eight of them aren't. I'm happy if I can find just four closed sales, as long as they're real comps: similar homes of similar size and similar style and similar condition on similar lots in similar locations in similar neighborhoods with similar schools in the same city.
Really? Is all that stuff important? Yeah, it's gotten to be kind of a custom. The kind of custom the Internet makes a career out of scoffing at. "Customs? We don't need no stinkin' customs!"
Zillow does an I-need-more-data-Captain wide-radius search that doesn't always—in fact, often doesn't—acknowledge real differences between adjacent neighborhoods. To get those East Palo Alto valuations, for example, Zillow crossed 101 into Palo Alto to draw its comps. I won't go into detail about why this is an incredible and unforgivable error except to say that the only similarity between the Palo Alto and East Palo Alto real estate markets is the name "Palo Alto". Otherwise they're apples and oranges, night and day.
Even when a Zestimate uses supposed comps from the same city, many if not most are far outside the neighborhood of the subject property. A Zestimate for a home in an affordable neighborhood routinely uses comps from much more expensive neighborhoods, and vice versa. More apples and oranges, more night and day, the kind of mixing no appraiser or competent agent would ever do, even with a multitude of adjustments to the price.
Then there's one Zestimate, for a home in Palo Alto, that pulls comps from
East Palo Alto
Redwood City and
...all of them much different (and in three cases, much less expensive) markets. At least it didn't pull comps from Pawtucket, Rhode Island.
Here I implore the great god Zestimate to help me value homes in a neighborhood that might be called a "thin market": not much is selling. My client is interested in two of the four homes I've charted. I have a fairly good idea of their value, based both on macro trends and on the list prices of similar homes that also aren't selling, but this approach is far less precise than I like. But you know what? That's what working in a declining market is like. When in doubt, go low—way low. Just make sure it's really a declining market.
Compare each of the Zestimates I've charted to its corresponding real-world list price and you'll see that Zestimates aren't a big help here either. Remember, all four homes have been on the market over one hundred days without selling. That's a long time in these here parts, long enough to confirm that buyers don't feel like saluting those list prices. Yet one Zestimate values a home that won't sell almost 6 percent above its list price. Another Zestimate values a home right at list price. One values a home just below list price, plausible, except that I strongly suspect the home is worth far less than list price. But one Zestimate comes in at the low end of my own range, and somehow I find this oddly reassuring...even with what I know about Zestimates. Never underestimate the power of reinforcement, even if it comes to you courtesy of pseudo-science.
Yes, I know, Zillow gives you the chance to tweak its Zestimates. Sellers might want to think twice before accepting Zillow's offer to make their calculations public. Here's one I stumbled across for a Palo Alto home, with an "owner's estimate" of $1,013,952. You even get to see the owner's worksheet. Good thing the buyer didn't see it, though, because the home ended up listed at $1,250,000. Zillow values it at $977,500, and maybe the seller was trying to pump up the price. It sold for $1,225,000. As a teardown. That cheesy deck the seller's worksheet says adds a few grand to the home's market value? It just means a few more bucks in dump fees when the new owner bulldozes the house.
Woo-hoo! This changes...sorry.
And yes, I know, Zillow gives you paragraph after paragraph of explanations and qualifications. But I wonder how many starry-eyed Zillow-philes read them, and how many of them understand or want to believe even the limitations Zillow admits to.
For example, Zillow allows that "we've never been to your house, never seen your expertise with colors and landscaping. Only you know those things. So we've given you a way to consider them (and other things) in calculating a home's value". Of course, every agent knows that every homeowner is completely objective about his or her home's distinctive and value-adding features. Like the fire-engine red tile countertops in the kitchen that make it look like a 1970s pizza parlor (add 10 percent). Like the yards of pricey English wallpaper ("you can't get this stuff in this country", the owner says proudly, and I never tell him the reason: no one in this country wants it) that has potential buyers leaving the house reeling with laughter (add 10 percent).
Speaking of 10 percent, "Our data shows that the majority of our Zestimate home valuations are within 10% of the selling price of the home. Of course, to a certain extent this depends on the accuracy of the home data we receive..." "To a certain extent"? There's a slight understatement. Since when was the law "garbage in, garbage out" repealed? And "within 10%"? In an area where homes routinely cost $1M or more, that's a margin of error you could fly an air tanker through. And I guess we need to define "majority", because Zillow doesn't: if "majority" means "more than 50 percent", I know four instances (see above) where Zestimates don't meet even this modest goal. And how come it's only "our data" that affirms the accuracy of the "majority" of Zestimates? Why hasn't this been independently confirmed?
Can you take your Zestimate to the bank? "No, you can't. To get a federally guaranteed loan, a law called FIRREA (the Federal Institutions Reform, Recovery and Enforcement Act) requires an appraisal from a professional appraiser." Yeah, probably just some stupid law the real estate industry rammed through to keep useless appraisers employed, right?
So what exactly is a Zestimate good for, assuming yours has a more than 50 percent chance of coming within 10 percent of a home's market value? (You math majors out there, what's the probability of this putting your Zestimate in the right zip code, let alone the right ballpark?) "The Zestimate is our estimate of fair market value, a starting point for home buyers and sellers and anyone just plain interested in the value of houses. You can use it in negotiating, in judging market trends, and in calculating all sorts of things for your personal purposes."
Let's deconstruct these two sentences and see how many weasel words and red flags we find:
"Estimate": hey, just a weasel word to keep the lawyers happy and the appraisal industry off Zillow's back, right? Zillow has to say this nudge-nudge-wink-wink but we know their stuff is the real deal because it comes from the Internet, not from some sleazy agent or scruffy appraiser.
"Fair market value": what the home I own, or want to own, is worth, right? That's clear enough. Despite the weasel word "estimate" preceding it.
"A starting point for home buyers and sellers" or at least buyers and sellers who don't mind starting from behind the eight ball, perhaps tens or hundreds of thousands of dollars off in value—and according to Zillow, that's the best they can hope for.
"and anyone just plain interested in the value of houses" such as just plain bubbleheads, who can ill-afford to be even more just plain ill-informed.
"You can use it in negotiating" from a position of weakness...
"in judging market trends" if you don't mind missing them...
"and in calculating all sorts of things for your personal purposes" like calculating your net worth? Nothing wrong with having a 49 percent chance of being more than 10 percent off when you calculate that, is there? No long-term consequences to that, right?
Where does Zillow's data come from? Zillow shouts "power to the people" and let's count the buzzwords: "It all comes from public data [buzzword 1]. The data is public because it's a consumer's  right  to have access  to information  about what is to be their most important investment —their homes . What? [bonus points for forced folksiness] You've never seen it? [more bonus points for hinting at conspiracy] That's because it is hard to find [more conspiracy bonus points] and hidden [still more conspiracy bonus points] in multiple sources. Zillow [huzzah!] has done the legwork  for you by getting huge  amounts of data  from many sources  and creating  something unique  that the public sources don't provide [nice dig at the "old way"]—a Zestimate of your home  based on the public data ."
And the real estate industry is accused of spinning? That's the hardest-selling summary of methodology I've ever seen. Cripes, the only buzzword Zillow missed is "cool", as in "we do this just because we think it's cool". "Oh, that Porsche out there in the Zillow parking lot? Yeah, it could be mine."
This stuff is fresh, right? You bet! "We receive new data and update the Zestimate regularly to capture new sales in a neighborhood. However [weasel word alert!] there is a delay [red flag!] between when the county is notified of a transaction and when we find out about it. We might not know for some time [weasel phrase alert!] about the sale of that house down the street from you that happened last week." Yep, piping fresh, right out of the oven.
Those of you who were raised on the Internet like I was raised on Captain Kangaroo and Perry Mason—and maybe those of you who weren't—may wonder why I get so worked up about this stuff.
Oh, I don't know. Let's say I come up with some lucrative premise that supposedly offers knowledge and empowerment, yet on cursory inspection seems to offer little of either.
Let's say I come across all aw-shucks and folksy, even though I embody the latest in business school and lawyerly thinking.
Let's say I make sweeping claims that may not deliver all they promise, and cover my rear with weasel words in case they don't.
Let's say I offer easy solutions to problems that don't have easy solutions.
Let's say I encourage the belief that something subjective and complicated is as easy as plugging in a multitude of numbers, the more the better, then tack on a few more weasel words to say I'm not.
Let's say I implicitly dismiss the idea that it takes objectivity, experience and expertise to perform a vitally important commercial function, then tack on still more weasel words.
Let's say my lawyers say all the right things in the calm assurance that few of my customers will read them, even fewer will chose to believe them and fewer still will understand their implications.
Let's say I know my target market, and know it won't ask too many awkward questions.
Let's say I set back empowerment rather than promote it, and that I may be too clueless to know it, even if I care.
Sounds like the typical heartless monolithic corporation, right? But if I do this on the Internet, I'm a folk hero.
copyright © John Fyten 2008 Site Map Home