Direct Marketing Meets Wholesale Surveillance

A $100K National Science Foundation grant to Geosemble Technologies, Inc.

SBIR Phase I: Exploiting High-Resolution Imagery, Geospatial Data, and Online Sources to Automatically Identify Direct Marketing Leads

Abstract: This Small Business Innovation Research Phase I project will conduct a feasibility study to demonstrate that by combining currently available high-resolution imagery, geospatial data (e.g., parcel data or structure data), and other related online data sources (e.g., property tax data or census data), it is possible to automatically generate highly targeted direct marketing leads for a variety of markets. The plan is to approach this problem by (1) aligning existing geospatial sources with the high-resolution imagery in order to determine the exact location and determine the address of the parcels seen in the imagery, (2) extracting the relevant features from the imagery to provide appropriate leads, such as determining the presence or absence of a swimming pool, the type of roofing materials used, or what types of cars are parked in the driveway, and (3) bringing in other sources of data, such as property tax assessment data to provide additional context.

The primary focus of the phase I project will be to demonstrate the use of machine learning technology for identifying features in high-resolution imagery that can be used for direct marketing. High-resolution aerial imagery is now being widely collected and is available for low cost or in some cases is even free. The challenges are to first to align parcel data with the high resolution imagery to identify the exact address and boundaries of a property, and second to develop feature extraction techniques that can exploit the contextual information to accurately identify novel features, such as roofs, cars, pools, landscaping, etc., that can be used for direct marketing. The ability to accurately identify features in imagery and then relate them to specific properties as well as related sources of information will allow a targeted direct marketing product to be built. The end users of this product will be companies seeking to market products directly to residential consumers. This includes product and services relating to home improvement, both exterior and interior, as well as those products relating to residents of the home, that can be gleaned from imagery available for the parcel in question. This is a large market and includes everyone from home improvement stores to roofing companies, construction companies, automobile dealers, tree trimmers, landscapers, and pool construction companies. Beyond direct marketing, the technology can also be used for other applications that combine imagery, geospatial data, and structured information. For example, it could used for mosquito abatement, which is important to stop the spread of West Nile Virus, by identifying large pools of stagnant water, associating those hazards with the appropriate address, and then mailing abatement notifications to the residents.

Tags: , , ,

Posted on June 19, 2007 at 3:52 PM • 45 Comments

Comments

Anonymous • June 19, 2007 4:27 PM

ah, yes.. spending your hard earned tax dollars to find more ways for telemarketers to intrude upon your life…

…mind you, this could be the seed for a whole new industry in domestic camoflauge products such as pool covers that look like grass or trees from above…

JackG't • June 19, 2007 4:37 PM

Snap images of a residence often enough and, in most cases, it would be possible to develop data concerning the activities and choices of residents and even guests. Now who would be interested in that kind of data, I wonder.

Lou the troll • June 19, 2007 4:46 PM

I knew my Yugo Beater Car Cover® was going to come in handy for covering my Hummer…

I don’t really own a Hummer…

And man, what a use of tax dollars…

Tim • June 19, 2007 5:07 PM

What a waste of money, especially tax money.

But if this goes through, at least I know higher-quality junk mail will go straight to my recycle bin without being read or opened.

Rob • June 19, 2007 5:14 PM

Hey, it’s capitalism (technically the combination of government and corporations is called economic fascism, but not when we do it, of course!).

So stop moaning and buy stock in aerial camouflage companies.

Theuns • June 19, 2007 5:31 PM

Why is this a useful way to spend tax dollars?
Well, other than the interesting science involved, bear in mind that satellite imagery doesn’t just cover the domestic market…

It does make more vivid the loss of another illusion of privacy – what you do in the courtyard of your house isn’t private business any more.

David Wilford • June 19, 2007 5:41 PM

As someone who deals with hi-res urban aerial photos on a daily basis, the current standard (six-inch resolution/pixel) imagery isn’t detailed enough to tell you much more about a roof than its color. Pools though do stand out very nicely and could be detected automatically. Cars/SUVs though, fuggetaboudit!

But the claim that parcel boundaries could be “exactly” determined is bullshit in most cases, not because the technology isn’t capable of doing it, but because the source boundary data itself is usually not even close to being “exact”.

FYI, you can already use tools that many cities provide to look up what your neighbor pays in taxes and compare it to what you pay. That all goes back to the old “open book” days that used to be noted in your newspaper where the tax assessor’s books would be opened for public view on a given day for the curious and stingy to check out to make sure they’re being treated the same as everyone else. Or not…

kashmarek • June 19, 2007 5:56 PM

Its all about marketing. Or the surveilance society in the guise of marketing. And if the targeted marketing mailing of 10,000 didn’t do the job, then 100,000 should, or maybe 1,000,000 and ultimately 10,000,000 a day for the next 100 days will turn the profit faucet on. And of course you pay for all of this in higher prices and taxes.

merkelcellcancer • June 19, 2007 6:05 PM

I am aware that the county of Charleston, South Carolina has used aerial photography (not satellite data) since the early 1990s to keep up with tax assessments. Looking at Google Earth and Google images of the neighborhood I can see that they have been upgraded since I poured a second driveway for the recreational vehicle I purchased five years ago.

Beyond that it is difficult to see the new roofing titles and the fact that I used the best I could afford. The quality of photography and resolution available is just not high enough in this community of over 110,000.

altjira • June 19, 2007 6:08 PM

Thanks, Bruce, for getting me all worked up again.

I hope this a-hole uses Google Maps, and that some diligent Schneier-blog-reading Google employee (there must be a lot of them!) catches him violating terms of service, and I hope they wait to slam him until just after the company he founds (based on a business plan using crucial strategies from a government-funded study) has its IPO, so he can spend the rest of his miserable life fending off investor class-action suits. But that’s a sad dream founded on a naive hope that there is justice in this world. I can’t even get into fantasy details like his new trophy wife leaving him after the courts impound her Ferrari, and suing him for US$100,000 a month alimony.

If people really wanted direct marketing, we would already have Brin and Page billionaires who had founded a company that allowed you to register all your personal details so that you were only exposed to marketing for stuff that you were interested in. Oh wait – isn’t that Brin and Page?

Pat Cahalan • June 19, 2007 6:37 PM

I can see why taxpayer money is funding this, it’s an interesting information science problem. The fact that it will undoubtedly be used solely for the purposes of Evil…

Chris E • June 19, 2007 7:30 PM

I’d like to expound on David’s comments – aerial 6 inch/pixel photos are already in use for West Nile Virus prevention in Oregon, and probably other states / locations. This seems like a more expensive way to do the same thing, while spying on citizens’ buying habits. You can pretty much tell a truck is a truck and a car is a car, but that’s about it. Where are they going to get up-to-date footage from anyway? The Russians?

DigitalCommando • June 19, 2007 8:53 PM

Bruce, if you could provide us with this company’s address, perhaps we could assist them with learning about our eating habits/preferences by all of us mailing our used toilet paper to their corporate headquarters for analysis!

Geoff Lane • June 20, 2007 12:05 AM

I have this wonderful new business idea – custom painted pool covers!

Do you truly want the authorities to think you have a cruise missile in your back yard? Well, my custom painted pool covers will do the job.

Peter • June 20, 2007 2:16 AM

This is a wind-up isn’t it ? Someone went through the NSF grant process to see how much they could scam for an “anti” project, right ? The phrase “direct marketing” is used too many times in the abstract Bruce reproduces to be real… I really do hope.

Neighborcat • June 20, 2007 6:01 AM

Perhaps there is a bright side to this. Marketing in general becomes obnoxious when the cost per recipient drops too low. That’s why email spam is considered almost universally annoying, but Super Bowl ads are often discussed more than the game itself.

Hopefully, the direct marketers want to filter recipients so they can increase the quality/cost of their ads, and this can bring valued resources into the recipients lives.

Example: I consider junk mail and the numerous “free” papers thrown on my lawn daily to be an annoyance, but to many struggling in the 3rd world, the same ads would be a daily delivery of cooking fuel.

Our task then, it to figure out the best means of collecting and using the resources provided by marketers. Home heating with junk mail is not a new idea, but can the junk-mail flex fuel vehicle be far behind?

Bring on those thick, glossy, full color packets stuffed with BTUs. DVDs, CDs, or those fake credit-cards? You bet! Can I have an extra one for my mom?

geomark • June 20, 2007 6:09 AM

Isn’t this what Google is up to with Street View? I’m sure that integrating Street View with Google Earth isn’t just a public service effort. I don’t know how much Google is spending but I’m sure it is orders of magnitude more than this study and they will probably roll out a paid service to advertisers in the not to distant future with similar functionality.

jmr • June 20, 2007 6:40 AM

Why are you people bemoaning this project? It truly is interesting science; getting a computer to understand imaging data and relating it to human endeavors would win any computer-related science fair project, so why shouldn’t it be fair game for an NSF grant? If you disagree with the idea of the NSF at all, say so. I am not going to present my opinion on that matter here.

The government may have other ideas for using this technology, like getting a computer to predict military activity or evaluate readiness by examining high resolution imagery of the target country. Or maybe evaluate tax circumstances without having to pay for tax assessors across the country? A piddling $100K for those capabilities would go a long way; think of the number of analysts or tax assessors the gov’t could fire if a computer could take over those sorts of duties.

Science and technology can be used for evil or for good. Creating capability isn’t morally wrong; it’s the use of the capability that is concerning. This sort of grant could advance the state of AI significantly, and that is a good thing. Using this technology commercially for marketing purposes may not be a good thing, but it’s something that can (and probably should) be regulated by policy, not by eliminating the ability for humankind to advance its understanding of the universe.

MrDude • June 20, 2007 7:29 AM

Tax dollars funding this. Thats nothing. Drug companies need to charge huge prices to cover the ~billions that drug trails cost. The thing is they don’t pay most of that bill.

@jmr
Yea but the context is so thay can spam you. We don’t really like the idea that some company knows more about the state of my pool than i do. Will i be able to get the information? Probably not. If its wrong can i get it at least corrected? Doubt it.

I see no reall reason to support R&D for a marketting tool.

Oh and the real sugar will be when compaines say that product X cost Y dollars to “recover the huge advertising and markting campains”.

NotOutraged • June 20, 2007 8:46 AM

This new game of putting NSF award abstracts on Bruce’s blog is an interesting exercise in civics. Part of the deal is that taking government money obliges the company to reveal something of what they are planning. But I am pretty sure the company was not expecting the kind of scrutiny that Bruce has exposed them to. Perhaps they will welcome the publicity, or perhaps they will feel awkward. The idea, to use an understatement beloved of British detective writers, is no better than it should be. There will be worse science and more questionable applications that get NSF support. I am also sure that commercial data miners are doing plenty of this kind of thing out of the light of day.

FooDooHackedYou • June 20, 2007 9:39 AM

WoW. So, maybe I should start putting decoys in my yard like fake cars, cardboard cutouts of swimming pools… Hmm… this could be fun. Honeypots anyone?

Peter • June 20, 2007 10:05 AM

Camouflage covers? Isn’t it already a felony to deliberately lie to government investigators
in the course of an investigation, whether you are a target, person of interest, or not?
This could support an entirely new domestic prison industry!

JackG't • June 20, 2007 10:26 AM

Calling the project Small Business Innovation Research strikes me as disingenuous, as does throwing in mosquito abatement as an example application. Surely large personal data brokerages would be prime beneficiaries. Some agencies of government would have a pipeline to the data. We’d likely have difficulty getting access to our privately-held personal information, as MrDude points out. Personal data that government agencies would glean from the database(s) would likely be classified or be access-restricted in some way, resulting in there being more potentially harmful information on us that we’d be unable to see.

derf • June 20, 2007 11:32 AM

With this kind of intrusiveness, how long will it be until someone takes up a collection to pay China to blow the satellites from orbit?

Fred • June 20, 2007 11:40 AM

Peter –
Is there a thread? Your other posts get at this idea in other ways. It seems we have the Tweedle-Dee-Dum-conundrum of who determines what (what) means, and who is held to account for the uses of that meaning, versus the meaning that others know or intend. Could this shake apart hundreds of classifying distinctions as data migrates through one ethical or activity context after another?

Anonymous • June 20, 2007 12:32 PM

@jmr
If you think the military is just now getting into this level of imaging, you are way, way, way, behind the times. Go do some smple research on spy satellites and what was possible using film cameras back in the middle of last century…

Fred F. • June 20, 2007 12:35 PM

SBIR’s are intended to help small company’s develop basic technology that they can in later phases commercialize to give the US an edge. The WHOLE purpose is to use tax money to advance the state of the art for American companies.

Phase ones are typically small ($30k or so) and run for 6 months (or thereabouts) and the purpose is to flesh out and idea. Then you present a final paper and request to get a Phase II where you actually do implement the idea in a prototype and position the company to get in bed with a larger company in a Phase III to actually sell the product.

Other than having the right connections to get a Phase I approved, you usually answer to an RFP put out by the government entities (DARPA, NAVY, ARMY, etc) where they list things they need but are not available. I don’t know if this was answering a request or it was their original proposal.

Ohh the other thing is that under an SBIR the government will grant you protection for your idea for 4 years I think so it is a sweet deal. You get paid to develop it and then you get to market it. Still I think it IS part of the government duty to pay for basic research in areas it identifies as critical but that private industry doesn’t have an incentive to tackle.

Anonymous • June 20, 2007 12:36 PM

@jmr
If you think the military is just now getting into this level of imaging, you are way, way, way, behind the times. Go do some smple research on spy satellites and what was possible using film cameras back in the middle of last century…

Chris E • June 20, 2007 1:25 PM

I’m still incredulous as to the scientific validity of this idea. For all intents and purposes it’s like facial recognition for satellite imagery. Software analyzes certain pixel arrangements and matches (guesses?) against a database – but instead of a realtively high-res close range photo, you’re using blurred images of large structures that may or may not be up to date. If all this can accomplish is send me ads to fertilize my lawn in December because the last satellite data was from August and my grass was dead, it’s a waste of my tax dollars.

Roxanne • June 20, 2007 2:42 PM

Isn’t this what Google Earth is for?

Meanwhile, ChemLawn has been doing this for years. I think their marketing dept had us on speed dial until they disallowed telemarketing; the guy stops by in person every spring.

Sorry, my kids are allergic, and I don’t care what my lawn looks like. Go away.

JackG't • June 20, 2007 4:35 PM

I believe that some large corporations would not meekly accept any declaration that the grant recipient’s idea is original and protected.

The whole project smacks of grantsmanship and a ploy to be acquisitioned after developing the product package at the expense of the public.

jmr • June 20, 2007 6:21 PM

@Anonymous,

I said nothing about imaging capabilities, but something about potentially improving data-mining capabilities. Just because you have the ability to take pictures at two inch resolution doesn’t mean you have the capability to understand what the images mean.

@MrDude

My point wasn’t whether the business capabilities that resulted are ones you want businesses to have, but whether an NSF grant to improve the state of science was warranted. I assert that grants for unpopular but valid science are important /provided/ that you agree to the existence of such a granting agency. Along the same lines, just because you don’t -like- particular grants that support the arts doesn’t mean they shouldn’t happen, again assuming you want public funds to support the arts.

I only state that the advancement of science that comes out of this grant is worth the $100K /if/ you agree to publicly support science. Just because it also has business possibilities doesn’t negate its scientific value.

npw • June 20, 2007 9:47 PM

I agree with jmr. So what that a stinky $100k of our tax dollars went to bunch of scientists who’re advancing the technology. In the long term, these technologies are what that would keep US economy ahead of all the other countries. Otherwise, if the next technological leap happens outside US, we’ll be crushed by China, India, Japan or Singapore in a wink! Do you think we have our edge because of our brilliant politicians or our real-estate agents, or stock brokers or lawyers! No buddy, it is because of our technology colleagues and NSF, on whom we’re picking! No wonder they call us geeks and nerds.

Clive Robinson • June 21, 2007 4:44 AM

@stacy

“Why are tax dollars funding this?”

How about to raise more tax dollars?

Read the last paragraph where it say’s,

“Beyond direct marketing, the technology can also be used for other applications that combine imagery, geospatial data, and structured information.”

Yup, you have a nice garden that’s got to be worth another 1000USD a year of property tax…

In case you think I am slightly off balance in the U.K. they are sereiously proposing to charge Council Tax (Propert Tax) based on things like if you have double glazing, central heating, air conditioning, nice decoration and nice views out of your windows…

jmr • June 21, 2007 7:17 AM

By the way, just for completeness’s sake, I don’t particularly like most data mining applications either, including this one.

kashmarek • June 21, 2007 9:13 AM

In at least one location in the State of Iowa, a city passed an ordinance adding a tax to property (it was called a rainwater fee) based on the area of non-impervious material exposed to rainfall (roof, driveway, sidewalk, tennis court, etc.) In that same county, they plan to go to arial photography for regular property tax assessments (probably in addition to regular walkby or driveby data collection).

slingshot • July 3, 2007 12:17 AM

I dont really get why this is generating so much controversy. Our houses are already mapped by who knows how many US and foreign satellites, not to mention flyovers by aircraft with high tech cameras a la google earth and yahoo maps. This NSF project is nothing more that the front end of what’s to come – which is probably why they funded it.

m.a.wajid • January 24, 2008 8:32 AM

we are seriously looking for marketting a product which may require intensive marketting workout,much groundwork,still wanted to take up the challenge ,we prefer the highly paid product which includes ,servicing in the later stage, anticipating early respond,wajid

Leave a comment

Login

Allowed HTML <a href="URL"> • <em> <cite> <i> • <strong> <b> • <sub> <sup> • <ul> <ol> <li> • <blockquote> <pre> Markdown Extra syntax via https://michelf.ca/projects/php-markdown/extra/

Sidebar photo of Bruce Schneier by Joe MacInnis.