DuckDuckGo, a search engine built with BOSS

DuckDuckDuckGonduck.v104Editor's note: Rahul Hampole, Senior Product Manager for Yahoo! Search BOSS, interviewed DuckDuckGo founder Gabriel Weinberg by email for this guest post. Gabriel is a serial entrepreneur and angel investor who was educated at MIT and resides in Valley Forge, PA, USA.

What is DuckDuckGo and its value proposition?

GW (Gabriel Weinberg): DuckDuckGo is a general purpose search engine designed to be a drop-in replacement for Google, Yahoo! or Bing. Our value proposition is way more instant answers, way less spam, and real privacy. I'll briefly touch on each.

We believe that for any given query there is usually a vertical search engine out there that answers that query better than a general search engine (via regular algorithmic results). So, for each query, we try to get you data from that vertical site in the form of instant answers that we put above the links in a red box.

For spam and irrelevant results, we've been way more aggressive at banning content farms, parked pages and any domain with essentially zero content (usually full of ads). We maintain a list of over 70 million such domains.

We also don't track our users. That is, we do not collect or share personal information, e.g. IP addresses and user agents.

Taken together along with our less-cluttered user interface, we believe we deliver a compelling alternative search engine in the form of a better overall search experience.

How did you end up using Yahoo! Search BOSS?

GW: We actually started doing our own general crawling before BOSS came out. Needless to say, that's time consuming and expensive!

When BOSS came out it became clear that by using it we could focus our attention on the instant answers, spam and interface elements that really differentiate DuckDuckGo. So we largely dropped our general crawl and buckled down on those efforts. Now our crawling is mainly limited to deep-crawling semi-structured content for instant answers and for spam removal.

Over time, our use of BOSS has become more complicated as well. We currently have a rather sophisticated intelligence layer built on top of it. This is why when you compare our results to BOSS they are often not close.

How does Yahoo! Search BOSS appeal to a developer like you (both on the product side and the business side)?

GW: On the product side, it enables a developer to deliver a fully functioning general-purpose search engine to a consumer, which is pretty amazing if you think about it. As such, it really opens up the possibility to innovate in search.

On the business side, the pricing structure enables you to find a profitable niche. Third-party advertising is allowed, and you can also use the Yahoo! Search ads, which should cover costs.

How easy was it to get up and running with it?

GW: As a developer, BOSS is extremely easy to use. It's packaged in an easily consumable API that can easily drop-in to pretty much any application. We program in Perl and, as I imagine in any modern language, there are helper libraries already in place for the OAuth and JSON/XML interactions. Also, the mailing list is rather active with other developers ready to help out if required.

Advice for developers building search applications?

GW: The consumer doesn't really care where results come from as long as they are good and they like the interface. This fact means that by using BOSS mixed with your own stuff you can actually deliver something to them that they like better than their current search engine! It really opens up innovation on the product side in my opinion, both for general and specialty search.

alt="Gabriel Weinberg avatar"> class="note">Gabriel Weinberg
Founder and CEO, DuckDuckGo