Sunday, April 08, 2007

On DataQuick's Changed Methodology

A reader recently sent me this email:

Hi Marinite,

Welcome back to the bubble blogging world!! I am sure you have had lots of positive comments and replies but it is nice to see you back after such a wild set of circumstances. My name is XXX I work for a Bay Area finance and real estate site based out of Foster City. I read your blog and the Sonoma bubble blog quite often to see whats happening in the North Bay. Living in the Bay Area myself (San Jose) I also have a similar personal interest in the housing bubble. I've actually created a section of our site dedicated to Bay Area and some National housing bubble news. I also feel its important to get the word out especially amongst the misinformation thats currently out there and we've had surprisingly good responses from readers and other bloggers. I'd like to help support your blog by dropping you a donation but notice you don't have a donate button or tip jar on your site. Let me know if theres a way to donate (paypal perhaps?)

I also want to encourage you to read some of our posts. We cover a lot of Bay Area as well as National topics that you specifically blog about. Some of the articles that may be of interest to you are:

Marin County Housing Data Conflict
Gloom and Doom in California: Subprime Loan Foreclosure Projections
San Francisco Bay Area Home Sales Fall Again, Prices Still Flat

So let me know how to donate to your blog. I'm certainly glad your back. And I hope you may find some of our information useful. Please feel free to use any of the articles or data/graphs for your blog.

Best Regards,


I have given thought about asking for donations. I figured if I got some donations then I could use the proceeds to get access to that real estate data that only money can buy and then share the results with readers. But I decided not to as it would only fuel the skepticism of some people and I didn't want to have to deal with that.

Regarding that first article cited in the email: If you didn't notice, when DataQuick announced their January, 2007 results they slipped in a little footnote indicating that they were changing the way they calculate the median sale price for their publicly consumed reports (the ones with the data tables I like to show on this blog) but not the ones given out to realtors. They are also changing the way they count a transaction as a sale.

Regarding the New Calculation of the Median
Starting with the January, 2007 data DataQuick is calculating a straight-up median based on everything that sells in the region of interest. A straight-up median was what I've always assumed they were doing all along. But apparently, prior to January, 2007 they were calculating a weighted average of medians across property types. Here is how someone at DataQuick described to me what they were doing in the past:

For a weighted average of medians, we multiplied the number of sales by the median price, for each home type, added the results together and divided by the total number of sales.
So, in other words, they would take the number of properties in category n1 (say, condos), multiply it by that category's median price, then add to it the product of category n2 (say townhouses) times its median, etc., and then divide the whole sum by the total number of properties that sold in the county. If I understand correctly, the equation they must have been using before looks like this:

Where T is the total number of properties that sold in the county, m is the straight-up median sale price for the ith property category (condos, SFRs, townhouses, etc), n is the number of properties that sold in the ith property category, and e is the number of categories.

This is an average of medians. Because averages are more strongly subject to the biasing effects of "outliers" as compared to medians, DataQuick's previous data was potentially noisier than if they had just calculated a plain-ol-vanilla median. In Marin, we have a very positively skewed distribution of sales -- the vast majority of sales bunches up in the low end of the pricing distribution. So our outliers are the über-expensive, multimillion dollar houses.

DataQuick assures us that there will only be about a 1% discrepancy when comparing current calculations of the median to pre-January, 2007 calculations of the median. I believe their claim but only for large counties with large sample sizes of sold properties and which are more normally distributed.

But what about a small county like Marin where a small number of properties sells in any given month and where there is a wide disparity between the "low end" and the "high end"? You should already have a sense of how "noisy" (i.e., variable) our data is. Recall those plots of mine showing DataQuick's year-over-year percent price appreciation (here is one example). Remember how Marin's data is all over the place whereas the data for the entire Bay Area forms a nice, fairly smooth curve? That's what I am talking about when I say "noisy". The noise is due to our small sample size where a small number of hugely expensive houses can throw off the calculations for the whole county.

And because of our positively skewed distribution here in Marin, it can be seen in the above equation that greater weight is being given to the medians derived from the lower priced property categories as compared to the higher priced categories. Perhaps DataQuick did this in an attempt to make the reported county median more representative of what most people were buying. I can understand the rationale for that. But now, with the new straight-up calculation of the median, the "lowly" $500K condo and the $20 million house are both contributing equally to the calculation of the median. So I think we can expect the reported county medians since January, 2007 to be higher than what they would have been under the old average of weighted medians calculation simply because greater weight is no longer being given to the lower end of the pricing distribution.

Furthermore, when calculating year-over-year percent appreciation for Marin, the appreciation rate will be biased larger when comparing calculations based on the new scheme to that of the old scheme. This explains a lot of the recent appreciation activity in Marin lately compared to the larger counties in the Bay Area. It won't be until January, 2008 when the year-over-year comparison is unbiased.

Regarding the New Calculation of Sales
This is what DataQuick says about the new method of counting the number of sales:

To count as an "arm's-length" sale for our sales counts, the logic we've used insisted that there be a seller, a buyer, and that money changed hands. We've now expanded this to include transactions where there was a purchase loan if no price was apparent.

We're also now including multiple sales transactions. If three homes were bought in the same transaction, we now count them as three home sales, not one sale.

These changes increase monthly sales counts by an average of 10 percent. Intra-family transfers are not included, nor are foreclosures until a home is re-sold to a new buyer.

So it should be clear from the above that the number of sales DataQuick now reports each month will be significantly higher (10% is huge) than it would have been prior to January, 2007. Again, what they are doing now vis-à-vis sales makes a lot of sense and I have to wonder why they weren't doing it before.

The Bottom Line
Overall it should be obvious that comparing DataQuick's current calculations of the median and number of sales with those of before January, 2007 will be problematic at best. For Marin, reported year-over-year appreciation rates will be biased in the direction of being higher. Similarly, year-over-year comparisons of sales will be too high, not just for Marin but for most counties. The net effect will be temporarily biased data in the direction of suggesting that things are healthier than they really are. Don't get me wrong: I don't think DataQuick is purposely biasing in that direction; it is just a natural consequence of their (needed and sensible) changes in methodology. But nevertheless, their year-over-year statistics will remain inaccurate until January, 2008 rolls around when they will again be comparing apples to apples, and oranges to oranges.

As a result of all this, I don't think I will be relying on their data as much as I was before. I am now back to where I was originally, wondering if I should be accepting donations to obtain for-pay data services.

Comments? Suggestions?


Blogger Westside Bubble said...

I noticed revisions to the year-ago DataQuick numbers the last two months, presumably recalculated with the new methodology?

Apr 8, 2007, 10:09:00 PM  
Anonymous Anonymous said...


If you can't change the facts, change the way you report them.

There must be a way to make Marin RE hot hot hot!
It never goes down!

Sorry to rant.

Marinite, I will happily send you donations for your work.

You are providing a great service to the disenfranchised like me.

You already saved me $50K on a wasted commission.

Apr 9, 2007, 1:08:00 AM  
Anonymous Anonymous said...

I actually read some of the articles sent to you from the finance guy. There is some quality reading in there. In regards to your comments pertaining to Dataquick, The economist Mike Hudson summed it up well by pretty much stating exactly what you mentioned: Data can easily be skewed if there exists a large disparity. A million dollar house sells on on street and suddenly the rest of the homes that sold get thrown into the same basket, hence making the collective 'median' higher.

Apr 9, 2007, 9:03:00 AM  
Anonymous Anonymous said...


If you can't change the facts, change the way you report them.

Again, I am not prepared to claim that DataQuick is deliberately reporting biased YOY stats. They recognized that they needed to change their methodology and did so. Unless they go back and change the last year's stats to reflect the new methodology, their YOY reports will be necessarily biased. I have not seen in any of their most recent reports any indication that they are going back a year and re-doing the stats so that valid YOY stats can be reported now.

Apr 9, 2007, 10:59:00 AM  
Anonymous Anonymous said...

Calculated Risk -- the best RE blog out there IMO -- on just put in a tip jar. You could ask how he's liking it.

Apr 9, 2007, 12:36:00 PM  
Anonymous Anonymous said...

An epiphany or not ???

Well, I’ve never been one for the conspiracy theory things and I normally give people the benefit of the doubt, but I’m not so sure anymore on this one after this idea flew into my head on the way home from work today. Although everyone has their own opinions as to causes of this massive housing bubble, I think we all have settled on the fact that incredible irresponsible monetary and lending policies were at the heat of this disaster and Sir Allen had his hands on the helm the entire time. So, given that, and given what I think is a rationale argument that Sir Allen is (or was) intelligent (not competent, just intelligent), how could this massive bubble have happened without his and the Fed’s full knowledge and apparent consent?

That’s when it hit me … it most certainly was intentional… and not for (just) the reasons why we have all discussed on these boards which is because the Fed is made up of pompous, greedy bankers who were more concerned with enriching themselves than managing this countries monetary policies (all true). However, as satisfying it is to point to someone or some thing and wag my finger and say “you did this, shame on you”, his incompetence and greed as a Fed Chairman still did not fully explain this housing bubble to me. Nope, I believe that, in addition to the stupidity and greed displayed by Sir Allen and his cohorts, that this bubble was more than that and that it was, in fact, a carefully orchestrated economic occurrence that took both monetary, fiscal and legislative policies to pull it off as planned.

That’s right… that’s my epiphany.. The housing bubble was created to provide a safeguard for the massive shortages projected in Social Security. That’s my new theory, because it all makes sense. Changes in capital gains tax laws, lax (or no) lending regulations and irresponsible monetary policies all designed to allow this country to inflate it’s way out of it’s social security responsibilities.

What has happened with this massive bubble?
(1) It has inflated the hell out of the U.S. dollar, which will eventually crumble and allow them to reshuffle their obligations and debt.
(2) It has falsely increased the baby boomer’s net worth, which could take pressure off of SS system.
(3) It has enslaved the next few generations to ensure they continue to have to work to pay into the system.

Anyway, yes, I believe there was more to this bubble than meets the eye.

Apr 9, 2007, 6:42:00 PM  
Anonymous Anonymous said...

I should have posted "thoughts?" at the end.. fire away at the above theory..

am I giving these clowns too much credit ?

Apr 9, 2007, 6:46:00 PM  
Anonymous Anonymous said...

I should add to the above that where they failed in their plan (they meaning the Fed, Treasury, Key Insider Legislators etc) is that they misjudged the level of greed of the average American and the power of the internet that fueled this thing to the heavens in no time; much quicker than they had anticipated....

my thoughts anyhow..

Apr 9, 2007, 6:50:00 PM  
Anonymous Anonymous said...

Well, some people have sugggested on other sites that the Fed/Gov is deliberately trying to destroy the US currency so as to start a new currency. I don't believe that for a minute, but I'm just saying it so you know you are not alone.

Apr 9, 2007, 7:14:00 PM  
Blogger Marinite said...

I put a perma-link to that eFinanceDirectory site (the folks who sent the email) in the right-hand margin as there is a lot of really great stuff there. Here's also the url:

Apr 9, 2007, 7:20:00 PM  
Anonymous Anonymous said...

Has anyone wondered that over the past few years, China has assembled the largest trade surplus ever while the US has been doing just the opposite? The Chinese may have been helping the US to keep its interest rates artificially low so it could grow powerfully by exporting their products cheap to us. The Americans have been using these low rates to pile up debts. In the long run, who has sold this country, politicians or the citizens?

Apr 9, 2007, 7:27:00 PM  
Anonymous Anonymous said...

10 Sure (Fun) Signs We've been in a Housing Bubble !!

1. Because neighbors are more interested in the price you bought your home for than the first name and age of your son and daughter. And your more interested in your neighbor’s wife than your own… it’s that consumption thing we’ve got going on.

2. Watching parades of enhanced buxom blondes arriving in their Beemers to meet with their tight-knitted groups to go on their morning "net worth" walks in the new developments. Of course, all us guys will secretly miss these events with the ongoing crash, but we can fantasize that some from this group will join something like a “Molly Maid” service to make ends meet and come by and clean our pipes now and then.

3. Having to endure the NAR's sponsored $40M pump and dump campaign to convince you to ignore the man behind the curtain.. in this case, your gut or Mr Common Sense.

4. Flip this House and Flip that House and whatever the hell else is on TV nowadays pumping this disaster. You know the end is near (or past) when shows like this show up. It’s just a matter of time before the lawyers knock them out of their time slots with “Sue your Broker” or “Screw em’ and Sue them All”. Now, that will be a fun show to watch.

5. Having to provide family pictures, family histories, love stories and promises to feed the squirrels and neighborhood rats in order to get the privilege of taking out a mortgage with your house, name and credit on the line. Kiss my a_ _ now Mr/Mrs Seller and Realtor. How about providing me a history of the house, including the location and size of all the nails and whether (and how) the current owners were able to have sex in the shower.

6. Donald Trump on WWW. I know he's got an ego the size of NY, but his recent exploits lend me to believe his cash flow must be in a hurt locker right now. God knows we need to see more of him.

7. Casey Serin being the poster boy for about 20-25% of the buyers in the market over the past 4-5 years. Ooops, no more Casey(s) any more. Who will be my new financial hero? Oh yes, that’s right, the homeless guy in Florida who bought five homes in less than a year… almost forgot about him. Glad there were only two guys like these two knuckleheads out there helping to fuel this disaster.

8. New Century Financial declaring bankruptcy after being the #2 mortgage company in the country. "Really now, there's nothing to worry about here folks, keep on walking"...

9. Inventory levels in Phoenix, OC, San Diego, Boston, Florida, Las Vegas.. etc etc... Yes, what happens in Vegas will stay in Vegas all right. Like those thousands of condos that were recently thrown together in in that hell hole. They will all stay in Vegas all right; empty in Vegas!! Vegas is death (well said).

10. Housing being referred to as sexy. I've heard and read this a number of times and each time I think I'm going to barf. Now, on the other hand, having the pleasure of staring at the backside of a newly minted Realtor climb up another set of stairs in her heels and tight skirt during her 10th McMansion showing only to be told "nah, too much money and what I'm looking for”.... now, that's sexy !

Apr 9, 2007, 10:39:00 PM  
Anonymous Anonymous said...

I also support the notion that the housing bubble was somewhat carefully planned. It was well known that the boomers were set to retire and the fact that the average boomer has less than 40k saved for retirement meant inevitable strain on not only the SS program, but on the economy as well. This was more of a liquidity bubble than anything else, and as also mentioned was meant to transfer money into the pockets of one age group in the form of debt to younger ones.

Had this been all there was to it, then we wouldn't have gotten into near the disastrous condition we're now in.The big mistake was that interest rates were kept low for too long and the boom lasted long enough to give people the false sense that what was highly unsustainable was business as usual. Just like the gold rush, 20's stock market, and tech booms before it, an entire economy was built around it with no room for error. Just like all the booms before it, this boom too shall have an equally unpleasant downfall. At least to those that partook in it anyway.

Apr 10, 2007, 7:41:00 AM  
Anonymous Anonymous said...

Lumber costs plummet.

Apr 10, 2007, 7:52:00 AM  
Anonymous Anonymous said...

If I were you, I would resist that offered contribution after the recent round of 'not-so-nice' threats to you....just my opinion...this one is probably a lovely person but life is short.

Have a good day.

Apr 10, 2007, 8:25:00 AM  
Anonymous Anonymous said...

Dear Anon to Anon. (long rant here.. sorry )

I'm assuming you are referring to my 10 reason post above. Well, ya know, I'm a decent, level headed individual and after reading your post I was going to say I'm sorry for offending some folks, but I've decided not to. I’m sorry you brought violence and threats back onto this blog again however. I wonder if you are in the RE industry yourself? Well, if you are, it's time you swallowed some of the baloney floating around this country nowadays, because of the greed and hype your industry has perpetrated on the rest of us. Remember, no good deed should go undone.

I'm probably a bitter renter in disguise, which will make all the HH's feel better about themselves now knowing that etc, but, I've made a few bucks in RE in the recent past in another state. However, I was outbid a number of times here in 2003/04 when I moved into the area. It is a great area as we all know, but far too expensive as are most places in this country now. Those are facts based on historic data by the way.

I experienced or heard and or felt first hand many of the things on my list above. That's when I took a step back and decided to hold off on all this madness. So, my little rant above was merely an avenue to express some feelings and thoughts that many blogers here and elsewhere share. I was a perhaps a little more crude and graphic in my analogies, but, trust me, "dares truth behind dem dere words". However, no names (or businesses) were mentioned (believe me, I would have liked to), so everyone can keep on walking. You especially, back to your empty RE office.

I'm quite sure that list struck a chord with many readers and fellow blogers however. That's why we blog (or why I blog now), to pass on my thoughts and observations to my partners in arms in this war. And make no mistake about it, it's a war! A war on our money, on our homes, on our future, on our kids' future, on our ideas, on our emotions, on our fears, on our jobs, on our communities and on now on momentum. And here's a clue for you and your RE industry pals, the good guys (eg the prudent, responsible, family oriented, long term thinkers and savers) have stopped your industry’s incessant and destructive advances and now have you in full on retreat. Maybe the momentum is not so obvious yet here in Marin, but you certainly ain’t advancing anymore and many of your partners in this RE war here in Cal and Marin are heading for the hills while they can.

Many of them are snakes and are busy unloading their own properties at the same time their trying to lure other unsuspecting buyers into the market. Just like the insiders at New Century financial who unloaded millions of shares last year before their company imploded. So, if you and your RE cronies don’t like the message, well, I say, tuff nuts to you; deal with it!

So, I say, three cheers for the good guys. Yes, the good guys are us bubble heads; people like Marinite, Ben, Patrick etc. Those are our leaders, our Generals in this war and who will also be the people lauded in the end by the masses and future generations for helping end this BS. As many have noted, this one will be bloody, but we need to see this war through to it’s full on conclusion, so we can get our communities, homes and kids’ futures back in order.

Yes, those are my exact thoughts and I believe in them completely. The RE / Mortgage / Banking industry have screwed this country to royal rafters out of simple greed. I’d like to see many of them fry in court or end up bankrupt themselves, but I doubt seriously much of that will happen.

End rant..

Apr 10, 2007, 5:51:00 PM  
Blogger Lander said...

"It's God's country, what can I say," Leslie Appleton-Young, chief economist for the California Association of Realtors, told an audience of agents Tuesday in Terra Linda. "When is the 30 percent decline in Marin County's market going to happen? Not in my lifetime."

Marin Independent [sic] Journal

Apr 10, 2007, 8:48:00 PM  
Anonymous Anonymous said...

Conspiracy theorists.

I believe it to be historical fact, that when wealth and power are in question there are lies and deceit, the more power and wealth involved the bigger the lies and deceit.

Did I mention that statistics can be misleading recently?

Oh, and if my house that I bought in 2004 fell 30% in value I would still be up 5%.

Freedy T

Apr 10, 2007, 10:22:00 PM  
Anonymous Anonymous said...

Denial by Ms Young is just another sure fire sign of the sigificant additional price drops that are now underway.. she and her cohorts must be greatly concerned or there would be no such comment.. "not in my lifetime"... poor dear, it would seem she's not long of this earth then...

ta ta Leslie..

Apr 11, 2007, 12:08:00 AM  
Anonymous Anonymous said...

IMF: Housing Killing US Economy.

Apr 11, 2007, 9:30:00 AM  
Anonymous Anonymous said...

" we can get our communities, homes and kids’ futures back in order."

I couldn't agree more. The price escalation in my opinion has caused a lot of deterioration for the namesake of creating stable communities.

I've been living here for years and for the last 6 have felt that the financial sacrifice people were making for a house was absolutely nonsensical and ridiculous.

There were a few years that I wondered if my economics was outdated because every time I sat down to do the math pertaining to what the bare-basic expenses would be for buying a bare-bones house anywhere in the Bay Area, I got some incredible number that was about 50% more than I would ever want to pay for anything- homes or whatever. Thousands of dollars per month. Huge personal sacrifices, and usually after decades of saving every single penny. I watched this go on for years. I also heard the same "fountain of youth" analogy of the supposed virtues that made property worth every penny just because it sat within site of San Francisco.

Just like the anon I quoted from, I really questioned if buying here was worth what seemed to me like a huge wall of difficulty. This became especially true when I started making that oh-so-fabulous 6-figure Bay Area salary and after re-configuring the expenses realized that even with this kind of income- something as simple as a VERY modest home in the BA would use up just about all of the income.

So if you do well professionally, live frugally, invest wisely, read the reports, and do your homework, then why on earth does homebuying in the BA fail to meet ANYTHING even remotely close to what would be deemed area economics?

The answer is what we all have known for some time, which is that this funny business was not capable of continuing forward.The party was bound to end, and that eventually a few cracks would form.

My personal conclusions are that those that have chosen to invest in themselves, commit to doing some old-fashioned saving, ignore the hype, and make themselves aware of what realistic costs are associated with living life are going to be sitting pretty for the next few years at the very least. Those house prices are going to fall for years and plummet to a point where people will probably forget that there was ever a boom, and once again, people will invest in themselves, their communities, and stop investing in paper dreams.

You notice that I did mention this entire post in past tense. The boom is and has been over. Now let's get used to the idea and move on.

Apr 11, 2007, 10:58:00 AM  
Anonymous Anonymous said...

It's a war people...

I expect we'll see, when the dust finally settles and the blood is all poured out onto the streets..

1. 50-70% price declines across the board.
2. The disestablishment of the Federal Reserve.
3. Sir Allen in ball and chains (okay, probably not, but just a wish).
4. NYC bankrupt.
5. California bankrupt.
6. Florida bankrupt.
7. The implosion of the U.S. dollar..
8. Gold over $1000 per ounce.
9. Crime and drug waves rampant.

But, on the other hand, many people got to feel rich and important for a while and got to drive fancy SUVs and BMWs..

Mark this post and lets check it in 4 years..

It's time to short HOG and a few more luxury consumer companies..

So sad..

Apr 11, 2007, 6:26:00 PM  
Anonymous Anonymous said...

So this got brought up on a Craigslist thread, how does DQ calculate it's monthly mortgage payment figures?

According to DQ's latest release for the Bay Area "The typical monthly mortgage payment that Bay Area buyers committed themselves to paying was $2,944 last month, up from $2,884 the previous month and down from $2,977 a year ago. Adjusted for inflation, current payments are 16.0 percent above typical payments in the spring of 1989, the peak of the prior real estate cycle. They are 8.6 percent below the current cycle's peak last June."

While it's great that they adjust for inflation, the question was/is do they adjust for the loan type?

What everyone was getting at was the fact that unless DQ is calculating the full amortized monthly amount (to adjust for I/O's, ARMs, and other exotic mortgages) and assuming that exotic mortgages were less prevalent in '89 isn't that 16% actually low? Maybe substantially lower then it should be?

Apr 13, 2007, 10:46:00 AM  

Post a Comment

<< Home

Terms of Use: The purpose of the Marin Real Estate Bubble weblog (located at URL and henceforth referred to as “MREB” or “this site”) is to present and discuss information relating to real estate and the real estate industry in general (locally, state-wide, nationally, and internationally) as it pertains to the thesis that recent real estate related activity is properly characterized as a “speculative mania” or a “bubble”. MREB is a non-profit, community site that depends on community participation and feedback. While MREB administrators do strive to confirm all information presented here and qualify all doubtful items, the information presented at MREB is neither definitive nor should it be construed as professional advice. All information published on MREB is provided “as is” without warranty of any kind and the administrators of this site shall not be liable for any direct or indirect damages arising out of use of this site. This site is moderated by MREB administrators and the MREB administrators reserve the right to edit, remove, or refuse postings that are off-topic, defamatory, libelous, offensive, or otherwise deemed inappropriate by MREB administrators. You should consult a finance professional before making any decisions based on information found on this site.

The contributors to this site may, from time to time, hold short (or long) positions in mentioned and related companies.