Zestimates:  home valuations for Mr. Spock.

"Data!  I need more data, Captain!"

The great thing about the Internet is that it dumps data right in your cubicle or spare bedroom.  That's also one of its greatest drawbacks. 

Data ≠ knowledge.  In fact, data < knowledge.

I went a long time without feeling the need to check the accuracy of Zillow's famous (or infamous) Zestimates, but I'm glad I finally did.  The results are both enlightening and predictable.

Here's how Zillow defines a Zestimate:

"The Zestimate (pronounced ZEST-ti-met, rhymes with estimate) home valuation is Zillow's estimated market value, computed using a proprietary formula.  It is not an appraisal.  It is a starting point in determining a home's value.  The Zestimate is pulled from data; your real estate agent or appraiser physically inspects the home and takes special features, location, and market conditions into account.  Variations in price also occur because of negotiating factors, closing costs, and timing of closing.  We encourage buyers, sellers, and homeowners to supplement Zillow's information by doing other research such as:

Sounds innocuous enough—and not by accident—although the obvious question is what Zillow thinks its Zestimates are worth if it encourages buyers, sellers, and homeowners to "supplement" them.  What good is a "starting point" if it can start you from the wrong point?  How does a Zestimate offer anything new, when all it does is send you to the same old place?

Or is this modesty just lawyer-crafted nudge-nudge-wink-wink?  A cynic might say that Zillow knows full well few of its users will go to a home valuation expert once they get their hot little hands on a Zestimate—after all, isn't "empowerment" the whole idea?—but that Zillow also knows full well it needs to pay lip service to the idea that Zestimates have limitations nudge-nudge-wink-wink. 

Zillow is a new breed of real estate Web site, a chastened breed that's needed only ten years or so to learn that you don't make money off the consumer's fascination with real estate by crashing through the industry's plate glass window, declaring "this changes everything" and offering online bling of dubious real-world value that titillates a handful of geeks.  Or has it?  At least that's Zillow's official line vis-à-vis the real estate industry:  "peace, brother".  Let's see how long this uneasy truce lasts.

For several years Zillow and this agent got along just fine, each oblivious to the other's existence.  Then, a few weeks ago I meet a buyer for the first time and, as usual, proudly hand him the Comparative Market Analysis (CMA) I've done for the house we stand in front of.  A CMA is an agent's opinion of the value of a house, based on and showing comparable active listings, pending and recent closed sales, and properties taken off the market unsold.  "Transparency", to use a word the Internet works hard.  I do a CMA for every home my clients see, and I'm proud of this service—I don't know of any other agent who prepares CMAs for buyers.  Usually it's just sellers who get them, because CMAs take lots of work.  I do ten or twenty a week and each one takes about twenty to thirty minutes. 

So I hand my CMA to this buyer with a flourish, explain what it is and what it does, step back and expect him to be overwhelmed.  He is.  Kinda.  He exclaims, "Oh!  Just like Zillow!" 

Rimshot.

No, I say.  It's ten times better.  I don't know why I say it's ten times better, and not five or twenty times better, except that anything handcrafted by an expert has got to be maybe ten times better than raw data cranked out on a virtual assembly line. 

Then I forget about it.  Zillow and I relapse into peaceful co-existence.

Until a week ago, when the same buyer emails me about a home he's just seen at an open house.  He'd like my opinion on its price, and he'd like me to get any reports the sellers have.  Fair enough, but there's one catch:  I can't find the house.   It's not in the Multiple Listing Service, at least not as an active listing.  What's the price?  Who's the agent?  What reports do he or she have?  How do I get them?  The only MLS entry I can find is an old listing taken off the market unsold months ago.

I resort to Craigslist.  Not there.  Then I do what most people do when looking for a needle in a haystack:  I look on the Internet.  I search by address and, for some reason, up pops a link to a Zestimate for this very house.  Intrigued, I click on it.  The Zestimate says it's worth $1,137,500.  Remember, the house had been on the market, without selling, for 178 days—at a list price of $959k. 

Rimshot. 

This bears investigating.  I decide to do four comparisons.  The first will see if Zillow can shed some light on the current market value of three homes this same client has seen.  Why?  Because the homes are in a city and price range that hasn't seen much sales activity lately.  When sales slow to a crawl, valuing homes becomes even more challenging because you work with few and sometimes no recent comparable sales ("comps").  Sure, you can check macro trends—see how much sales prices have gone up or down in a larger geographic area with more sales—but one of the great things about the mid-Peninsula—and the bane of anyone trying to value a home here—is that almost every city in this area is an assortment of widely differing micro-neighborhoods.  And even if our neighborhoods were uniform, stamped out cookie-cutter style, there'd still be large variations in value based on the perceived quality of various school districts, and even between schools within the same district. 

Then there's a neighborhood's proximity to shopping, and what kind is it?  A handful of beaten-down strip malls, or an attractive destination downtown?  Then there's location.  Is the neighborhood next to a freeway?  Next to train tracks?  Next to a much better area?  Next to a much worse area?  What's the hot topic in that city or neighborhood?  In one city it might be allegations of petty theft by the directors of one of dozens of city-sponsored programs.  In a nearby city, so close it shares a zip code, the hot topic might be a spate of gang-related shootings.

So I need a little help here.  Let's see if Zillow is up to it.

After that, I'll attempt three more straightforward comparisons.  I'll simply compare the sales prices of single-family home sales closed since April 1, 2008 (I wrote this a few months ago) with Zillow's Zestimates for those homes.  To test Zillow's accuracy in a range of markets, I'll use one market that's seen home prices rise recently, Palo Alto; one market that could best be described as "mixed", Burlingame; and one market that could best be described as "hammered", East Palo Alto.

Let's see how those Zestimates perform.

Here's how well they predicted the sales prices of the latest twenty-eight closed Palo Alto SFR sales.

The chart doesn't convey the full discrepancy between Zestimate and reality.  Of the 28 data points, eighteen differ more than 10 percent between Zestimate and sales price.  Five of those are off by 20 percent or more.  And when you're dealing with an average sales price of $1,637,142 (and 86 cents), 10 percent here and 10 percent there and pretty soon you're talking real money.

I will admit that Zestimates absolutely nailed the price twice, within .04 and .2 percent.  Nothing to complain about there.  So we can say that, based on the Palo Alto sample, Zestimates are deadly accurate 7.1 percent of the time.  That's 7 point 1 percent, not 71 percent.

And six Zestimates were within 5 percent of the sale price.  That's close enough for government work, isn't it?  You wouldn't mind paying $1,765,500 for a home you could've gotten for $1,700,000, would you?  Thought not.  Or paying $1,728,000 for a house that otherwise would have sold for just $1,667,000.  Or bidding $1,447,500 and not a penny more for your dream house and seeing someone else snag it for $1,505,000.  At least you didn't shudder overpay.

Okay, so maybe Zestimates can be a little—okay, a lot—off the mark in a rising market.  Let's see how they do in a fairly stable market, Burlingame.

Much better!  Zestimates nailed the sales price of two of the ten data points.  That's 20 percent accuracy, and for those of you who aren't into numbers, 20 percent is nearly three times as good as 7 percent.  And that third data point, off just 1.7 percent, ain't too shabby either.  Of course, if there were multiple offers on that third home (and there were—three—including one by my client) and you really really wanted the home, and it was a great home, expensively remodeled and on an unusually large lot and with good schools and close to downtown and okay I'll stop, and you offered a Zestimate-correct price of $893,000 and it turned out to sell for just $15,500 more and you calculated how little that extra $15,500 would cost you amortized over the life of a thirty year loan at near-historically low interest rates...well, you'd feel just fine about that, wouldn't you?

What about the other seven data points?  Zestimates have you offering 32.1 percent more than the actual sales price, 29.6 percent less, 20.1 percent more, 19.9 percent less, 9.7 percent more and 9.5 percent less.  Hey, I know:  just average the percentages all the Zestimates are off by and you'll wind up with the correct market value!

Okay, so maybe Zestimates do better—but far from consistently great—in a stable market.  How about a market where prices are dropping off the table, East Palo Alto?

Eh.  Not so great.

East Palo Alto brings up an interesting point.  Zestimates give you data, all right.  Click on "comparable homes" and you'll get a list of fifty or so.  Data!  I need more data, Captain!  Except that when calculating the value of a home, data can just as easily lead you down a slippery slope as down the right path.  I'm selective about the comparables I use, and there's good precedent for this.  Look at a professional appraisal (as Zillow urges) or something much the same, a Broker's Price Opinion, and you'll see just three comps or, lately, as many as six.  What you won't see is fifty comps, because you don't need fifty comps, especially if forty-eight of them aren't.  I'm happy if I can find just four closed sales, as long as they're real comps:  similar homes of similar size and similar style and similar condition on similar lots in similar locations in similar neighborhoods with similar schools in the same city. 

Really?  Is all that stuff important?  Yeah, it's gotten to be kind of a custom.  The kind of custom the Internet makes good money scoffing at.  "Customs?  We don't need no stinkin' customs!"

Zillow's approach is to do an I-need-more-data-Captain wide-radius search that doesn't always—in fact, often doesn't—acknowledge real differences between adjacent neighborhoods.  To get those East Palo Alto valuations, for example, Zillow crossed 101 into Palo Alto to draw its comps.  I won't go into detail about why this is an incredible and, I think, unforgiveable, error, except to say that the only similarity between the Palo Alto and East Palo Alto real estate markets is the name "Palo Alto".  Otherwise they're apples and oranges, night and day.

Even when a Zestimate uses supposed comps from the same city, many if not most are far outside the neighborhood of the subject property.  A Zestimate for a home in an affordable neighborhood routinely uses comps from much more expensive neighborhoods, and vice versa.  More apples and oranges, more night and day, the kind of mixing no appraiser or competent agent would ever do, even with a multitude of adjustments to the price.

Then there's one Zestimate, for a home in Palo Alto, that pulls comps from

...all of them much different (and in three cases, much less expensive) markets.  At least it didn't pull comps from Pawtucket, Rhode Island.

Here I implore the great god Zestimate to help me value homes in a neighborhood that might be called a "thin market":  not much is selling.  My client is interested in two of the four homes I've charted.  I have a pretty good idea of their value, based both on macro trends and on prices of similar homes on the market that also aren't selling, but this approach is far less precise than I like.  But you know what?  That's what working in a declining market is like.  When in doubt, go low—way low.  Just make sure it's really a declining market.

Compare each of the Zestimates I've charted to its corresponding real-world list price and you'll see that Zestimates aren't a big help here either.  Remember, all four of these homes have been on the market more than one hundred days.  That's a long time around here, long enough to confirm that buyers don't think much of the accuracy and relevance of these list prices.  Yet one Zestimate values a home that won't sell at its current list price almost 6 percent above that price.  Another Zestimate values a home right at list price.  One values a home just below list price, plausible, except that I strongly suspect the home is worth far less than list price.  But one Zestimate comes in at the low end of my own range, and somehow I find this oddly reassuring.  Even knowing what I know about Zestimates.  Never underestimate the power of reinforcement, even if it comes to you via pseudo-science.

Yes, I know, Zillow gives you the chance to adjust its Zestimates.  Sellers might want to think twice before taking up Zillow on its offer to make their calculations public.  Here's one I stumbled across in Palo Alto, a home with an "owner's estimate" of $1,013,952.  You even get to see the owner's worksheet.  Good thing the buyer didn't see it, though, because the home ended up listed at $1,250,000.  Zillow values it at $977,500, and maybe the seller went public to pump up the price.  It sold for $1,225,000.  As a teardown.  That cheesy deck the seller's worksheet says adds a few grand to the market value?  It just means a few more bucks in dump fees when you demolish the house. 

Woo-hoo!  This changes...sorry.

And yes, I know, Zillow gives you paragraph after paragraph of explanations and qualifications.  But I wonder how many starry-eyed Zillow-philes read them, and how many of them understand even the limitations Zillow admits to. 

For example, Zillow allows that "we've never been to your house, never seen your expertise with colors and landscaping.  Only you know those things.  So we've given you a way to consider them (and other things) in calculating a home's value".  Of course, every agent knows that every homeowner is completely objective about his or her home's distinctive and value-adding features.  Like the fire-engine red tile countertops in the kitchen that make it look like a 1970s pizza parlor (definitely add 10 percent).  Like the yards of pricey English wallpaper ("you can't get this stuff is this country" and there's a reason:  no one wants it) that propels potential buyers from the house giddy with laughter (absolutely add another 10 percent).

Speaking of 10 percent, "Our data shows that the majority of our Zestimate home valuations are within 10% of the selling price of the home.  Of course, to a certain extent this depends on the accuracy of the home data we receive..."  "To a certain extent"?  There's an understatement.  Since when was the rule "garbage in, garbage out" repealed?  And "within 10%"?  In an area where homes routinely cost $1M or more, that's a margin of error you could fly an air tanker through.  And I guess we need to define "majority":  if it means "more than 50 percent", then I know four instances (see above) where Zestimates don't achieve even this modest goal.  And how come it's only "our data" that affirms the accuracy of the "majority" of Zestimates?  Why hasn't this been independently confirmed?

Can you take your Zestimate to the bank?  "No, you can't.  To get a federally guaranteed loan, a law called FIRREA (the Federal Institutions Reform, Recovery and Enforcement Act) requires an appraisal from a professional appraiser."  Yeah, probably just some stupid law the real estate industry rammed through to keep useless appraisers employed, right? 

So what exactly is a Zestimate good for, assuming yours has a more than 50 percent chance of coming within 10 percent of a home's market value?  (You math majors out there, what's the probability of this putting you even in the right zip code, let alone the right ballpark?)  "The Zestimate is our estimate of fair market value, a starting point for home buyers and sellers and anyone just plain interested in the value of houses.  You can use it in negotiating, in judging market trends, and in calculating all sorts of things for your personal purposes." 

Let's deconstruct these two sentences and see how many weasel words and red flags we find:

Where does Zillow's data come from?  Zillow shouts "power to the people" and let's count the buzzwords:  "It all comes from public data [buzzword 1].  The data is public because it's a consumer's [2] right [3] to have access [4] to information [5] about what is to be their most important investment [6]—their homes [7].  What?  [bonus points for forced folksiness]  You've never seen it? [unlimited bonus points for hinting at conspiracy]  That's because it is hard to find [more conspiracy bonus points] and hidden [still more conspiracy bonus points] in multiple sources.  Zillow [hurrah!] has done the legwork [8] for you by getting huge [9] amounts of data [10] from many sources [11] and creating [12] something unique [13] that the public sources don't provide [nice dig at the "old way"]—a Zestimate of your home [14] based on the public data [15]." 

And the real estate industry is accused of spinning?  That's the hardest-selling summary of methodology I've ever seen.  Cripes, the only buzzword Zillow missed is "cool", as in "we do this just because we think it's cool".  "Oh, that Porsche out there in the parking lot?  Yeah, it could be mine."

This stuff is fresh, right?  You bet!  "We receive new data and update the Zestimate regularly to capture new sales in a neighborhood. However [weasel word alert!] there is a delay [red flag!] between when the county is notified of a transaction and when we find out about it.  We might not know for some time [weasel phrase alert!] about the sale of that house down the street from you that happened last week."  Yes, piping fresh, right out of the oven.

Those of you who were raised on the Internet like I was raised on Captain Kangaroo and Perry Mason—and maybe those of you who weren't—may wonder why I get so worked up about this stuff. 

Oh, I don't know.  Let's say I come up with some lucrative premise that supposedly offers knowledge and empowerment, yet on even cursory inspection seems to offer little of either. 

Let's say I come across all aw-shucks and folksy, even though I embody the latest in business school and lawyerly thinking. 

Let's say I make sweeping claims that may not deliver all they promise, and that I cover my rear with weasel words in case they don't.

Let's say I offer easy solutions to problems that don't have easy solutions. 

Let's say I encourage the belief that something subjective and complicated is as easy as plugging in a multitude of numbers, the more the better, then tack on a few more weasel words so I can say I'm not. 

Let's say I implicitly dismiss the idea that it takes objectivity, experience and expertise to perform a vitally important commercial function, then tack on still more weasel words. 

Let's say my lawyers say all the right things in the calm assurance that few consumers will read them, even fewer will chose to believe them and fewer still will understand their implications. 

Let's say I know my target market, and know it won't ask too many questions. 

Let's say I set back empowerment rather than promote it, and that I may be too clueless to know it, even if I care.

Sounds like the typical heartless monolithic corporation, right?  But if I do this on the Internet, I'm a folk hero.

copyright © John Fyten 2008        Site Map         Home