Inside DuckDuckGo, Google’s Tiniest, Fiercest Competitor…


d

From FastCompany

In 2008, launching a search engine seemed like a crazy idea. DuckDuckGo proved the critics wrong… with hardcore privacy…

When Gabriel Weinberg launched a search engine in 2008, plenty of people thought he was insane. How could DuckDuckGo, a tiny, Philadelphia-based startup, go up against Google? One way, he wagered, was by respecting user privacy. Six years later, we’re living in the post-Snowden era, and the idea doesn’t seem so crazy.

In fact, DuckDuckGo is exploding.

Looking at the chart of DuckDuckGo’s daily search queries above, the milestones are obvious. A $3 million investment from Union Square Ventures in 2011. Just prior to that, a San Francisco billboard campaign. Inclusion in Time‘s 50 Best Websites of 2011. Each of these things moved the traffic needle for DuckDuckGo, but none of them came close to sparking anything like the massive spike in queries the company saw last July. That’s when Edward Snowden first revealed the NSA’s extensive digital surveillance program to the world. The line on the chart hasn’t stopped climbing north since.

“Every year, we’ve grown 200-500%,” Weinberg says. “The numbers keep getting bigger.” As of early February, DuckDuckGo was seeing more than 4 million search queries per day. One year ago, that number had just barely broken 1 million.

Surprisingly, the sudden success didn’t send the site crashing down. Nor did it change the company’s stripped-down, razor-sharp focus. Here’s how one small company is slowly, surely beating its way into the most monopolized category in technology. 

Three Ideas In One: Where DuckDuckGo Came From

Weinberg didn’t originally set out to build a search engine. After shuttering one failed startup and selling another to Classmates.com for $10 million in 2006, the MIT grad found himself exploring several new ideas. Across multiple projects, he focused on structured data, Quora-style Q&A, and programmatically combating spam. “I started all of these projects independently and none of them really took off,” Weinberg says. “Then I realized, maybe if I put them all together, there might be an interesting search experience there.”

The result was DuckDuckGo, a search engine offering direct answers to people’s queries, rather than merely delivering a list of links. Below these so-called “Instant Answers,” the site still displays traditional, link-by-link search results syndicated from third parties like Bing and Yandex but, crucially, they’re filtered and reorganized to reduce spam.

Looking at the modern history of the search space, Weinberg noticed that several companies in the early 2000s had tried–and failed–to rival Google by mimicking its method of mass-indexing the web. Instead of following this ambitious (and very expensive) path, Weinberg decided to let other companies’ infrastructure do most of the heavy lifting so that his startup–initially, just him coding at home with his newborn son nearby–could focus on building a superior experience for finding information online. The key, as he saw it, would be Instant Answers.

“When you do a search, you generally want an answer. You don’t necessarily want to click around links,” Weinberg says. “It’s our job to try to get an answer. Our grand vision is that that happens for 80% of queries, even for very niche things.”

To deliver these answers, DuckDuckGo relies on a mixture of third-party data sources and the deep–sometimes bizarrely arcane–knowledge of its growing community of users and developers.

“We have recipes and Lego parts and other weird stuff that we don’t know about as a team,” Weinberg says. “The only way that works is if we have a community of people who are interested enough to know about these subjects. And then come up with ideas about what the answer should be, suggest the sources, and even develop them.”

With Instant Answers, he was clearly onto something. Today, when you run a Google search for “Galileo” or “How tall is Big Ben?” the usual list of blue links is accompanied by a tidy white box displaying a mini-bio of Galileo or, in the latter case, the answer: 316 feet.

This is the biggest risk for any startup who dares challenge a giant head-on: At any point, the Googles or Facebooks or Apples of the world can just mimic what made you different, slam-dunking your shattered dreams into the wastebin of tech history.

Weinberg and his small team seem undeterred. After all, DuckDuckGo has one asset that Google could never copy, even if it wanted to.

DuckDuckGo’s Secret Weapon: Hardcore Privacy

When you do a search from DuckDuckGo’s website or one of its mobile apps, it doesn’t know who you are. There are no user accounts. Your IP address isn’t logged by default. The site doesn’t use search cookies to keep track of what you do over time or where else you go online. It doesn’t save your search history. When you click on a link in DuckDuckGo’s results, those websites won’t see which search terms you used. The company even has its own Tor exit relay, allowingTor users to search DuckDuckGo with less of a performance lag.

Simply put, they’re hardcore about privacy.

But things didn’t start out that way. Weinberg, who says he has “always been a privacy-minded person,” wasn’t particularly concerned with search privacy issues when he first started building the service. In fact, he knew very little about the matter at all. Then early users started asking questions.

“Some of the first questions I got were about privacy,” Weinberg says. “I had launched and it was on Reddit and Hacker News. So it was tech-heavy and there were a lot of privacy-minded people.”

Did the site use tracking cookies? Did it log IP addresses? These are things Weinberg hadn’t thought much about, but clearly mattered to his newfound user base. So the self-described tech policy geek decided to dig deeper into the privacy practices of Google and other search engines. He didn’t like what he learned.

“If you look at the logs of people’s search sessions, they’re the most personal thing on the Internet,” Weinberg says. “Unlike Facebook, where you choose what to post, with search you’re typing in medical and financial problems and all sorts of other things. You’re not thinking about the privacy implications of your search history.”

This common functionality, combined with the possibility for accidental data leaks and hacks, made Weinberg nervous. At the same time, he realized that it was still possible to build a viable business model around search without tracking users. One might not reach Google-levels of profitability without large-scale targeted advertising, but the fundamental logic of delivering search ads–user searches for motorcycles, you show them a motorcycle ad sold by keyword–is still sound. And, of course, the profitability of such a model only grows as more people search more.

It quickly became clear that taking a no-holds-barred approach to privacy would give DuckDuckGo a unique selling point as Google gobbled up more private user data. So the company positioned itself accordingly and started amassing attention as the issue of online privacy slowly ballooned in the public’s consciousness.

“It was extreme at the time,” Weinberg says. “And it still may be considered extreme by some people, but I think it’s becoming less extreme nowadays. In the last year, it’s become obvious why people don’t want to be tracked.”

How DuckDuckGo’s Community Informs Its Product–And Payroll

One person who didn’t think the pro-privacy tack was extreme is Caine Tighe. After hearing about DuckDuckGo in 2009, the young programmer decided to reach out to see if Weinberg needed any help.

“I met Gabe like I meet a lot of people: on the Internet,” says Tighe. “I thought it was interesting that he was working on a new search engine. He didn’t really have that much stuff for me at the time, as he was still fleshing out the idea. Later on, I helped him write the first Android app.”

By 2011, armed with its first infusion of venture cash, DuckDuckGo was ready to leave Weinberg’s basement and grow into a real company. Having worked on multiple projects with Weinberg, Tighe was an obvious choice to become the company’s first full-time employee. He soon became the Director of Core Components, a role in which he oversees DuckDuckGo’s infrastructure, from its underlying web technology to the functionality of its mobile apps.

Tighe’s trajectory is one that all future employees would follow: One by one, members of the DuckDuckGo community have gradually increased their involvement with the site–building plug-ins, recommending obscure data sources, even contributing code–eventually positioning themselves as no-brainer job applicants the next time the company is hiring.

“If you want to work with us, we find that the most motivated people come directly from the community,” says Zac Pappis, DuckDuckGo’s Director of Marketing and Community. “Anybody can get their feet wet.” Indeed, Pappis himself was heavily involved in the DuckDuckGo community and did part-time work for Weinberg in the evenings before being offered a full-time position in 2012.

It’s in this capacity that Pappis helps oversee the growing community of users and developers that contribute to DuckDuckGo. Over time, this community has taken on an increasingly vital role in the evolution of the company’s core product. Not only does it pluck new employees from this group of devotees, but a subgroup of it has splintered off into a platform all its own: DuckDuckHack.

Letting Users Hack Their Own Search Engine

“It started very casually,” says Weinberg. “I just made some APIs and tried to get these guys involved coding stuff for it. And then it clicked that this could actually be a platform that people could work on.”

 Branded as an “open source DuckDuckGo,” the DuckDuckHack platform lets developers build their own Instant Answer plug-ins using the company’s suite of (mostly Perl-based) APIs and documentation. It’s essentially a toolkit that lets anybody programmatically define how to interpret whatever people may ask a search engine, as well as where to find the information online.

Some examples are obvious: Information about movies comes from IMBD, or Rotten Tomatoes. A search for “The Beatles” pulls in a biographical blurb from Wikipedia or Last.fm. Simple computations come from Wolfram Alpha. Tighe seems particularly excited about their recent integration with Forecast.io, the data source behind the popular weather prediction app Dark Sky.

Then there’s the more obscure, less obvious stuff. That’s where the community comes into play.

“The Wolfram Alpha experience really opened my eyes to the fact that you could have answers for really esoteric stuff,” says Weinberg. “And we started including them. You could have answers for tons and tons of topics that our team knew absolutely nothing about. Celebrities, for example. We’re not big on pop culture here.”

One user, for instance, was a big Lego nerd. As Lego enthusiasts know, each piece of the iconic building block set has a unique ID number. To make looking up those pieces easier on DuckDuckGo, he crafted a plug-in that pulls from a database of Lego pieces built by Lego fans every bit as geeky as he is.

Scrolling through the list of existing Instant Answer queries–known as “goodies” on DuckDuckHack–you begin to understand why DuckDuckGo has opened up this part of its process to others: Some of this stuff is just too weird for one company to tackle.

Inside DuckDuckGo: How Its Tiny Team Works

Stepping inside DuckDuckGo’s headquarters in Paoli, Pennsylvania, it’s hard to imagine that its two modest, sparsely occupied floors could be used to launch an insurgency against one of the biggest tech companies on the planet. (see slide show here)

But they are–sort of. Most of DuckDuckGo’s 20 or so employees work remotely. Doug Brown, a front-end dev specialist, lives in Toronto, making the trip to the small Philadelphia suburb periodically. The company’s newest hire, a developer named Jag Talon, started working for the company from the Philippines before his family relocated to New Jersey last year. When we visited their office in January, fewer than a dozen people roamed the halls. This, we were told, was more than usual.

Day to day, the team’s priorities are dictated by what they call their “critical path”–the short list of focal points that prevent them from getting distracted from DuckDuckGo’s core mission. Surprisingly, their recent surge in traffic hasn’t altered the critical path very much, other than to initially double-check that the site was well-architected enough to handle the load (It was).

One recently checked-off item from the priority list was the consolidation of the company’s public-facing community sites into a single Web interface. Next up is a visual redesign of the DuckDuckGo website, which they’re aiming to relaunch in the second quarter of this year.

Like any company with a mostly remote team, DuckDuckGo experiments with all the latest online collaboration tools.

Skype. Yammer. HipChat. Asana. “We’ve tried everything that we know of,” says Pappis.

Lately, they’ve been toying with Sqwiggle, an online collaboration tool that uses persistent video and periodic screenshots to let coworkers see each other–or know who’s away from their desk.

“Skype is more like picking up the phone, whereas Squiggle is more like texting,” says Brown. “You can see people’s faces and you can chat with them really quick without having to set up a formal time because you know they’re right there. That’s pretty much the key to everything. And making sure that everybody knows what you’re doing and when you’re doing it.”

On the tech side, all of the team’s code–mostly Perl and JavaScript–is managed and discussed through Git and GitHub Enterprise. Every change, no matter how minute, is pushed through the team’s HipChat room so everybody can see what’s going on at all times. This practice, everyone seems to agree, is incredibly handy.

“It’s interesting because that’s a by-product of the fact that the team is remote,” says Tighe. “It wouldn’t exist if everybody were in this office, because everybody would just be pinging each other about what was going on locally, as opposed to being like: We need to have every single piece of information that goes out available to everybody on the team that could potentially be responsible for it, asynchronously.”

When it comes to the tools individual developers use to build out DuckDuckGo’s features, Weinberg isn’t picky.

“We try not to control people’s environments,” he says. “People are remote. We give everyone their own dev environment that’s a full stack of DuckDuckGo. It’s a full machine. Some people are using Emacs. Some people are connecting directly to the machine. Some people are developing locally. People work in all sorts of different ways.
~~

Follow

Get every new post delivered to your Inbox.

Join 4,560 other followers