Duck Tales: How we use AI at DuckDuckGo while staying true to our principles (Ep.45)
This episode first appeared on Inside DuckDuckGo, where you can watch or listen to it. The transcript below is mirrored from there.
In this episode, Marc (Engineering) and Zach (Data Science) discuss how we use AI internally: our most common uses, how we use it privately, and why our principles matter more when using AI, not less.
The transcript is only lightly edited and may contain minor inaccuracies or transcription errors.
Marc: Hi, and welcome to Duck Tales, where we go behind the scenes at DuckDuckGo and discuss the stories, technology, and people that help build privacy tools for everyone. In each episode, you’ll hear from employees about our vision, product updates, engineering, or approach to AI. My name is Marc and I’m on the engineering team. I have with me Zach. Zach, you want to introduce yourself?
Zach: hi, yeah. I’m Zach Deane-Mayer on our data science team here at DuckDuckGo. I am still a Duckling, so I’ve been here for about ten months. and I work on building our internal AI systems.
Marc: Great. yeah, so today we’re gonna talk even more about AI, which is Zach’s specialty. I think today we’re gonna talk specifically about how we use Duck or sorry, how we use AI internally at DuckDuckGo. so yeah, I guess to start out, as a company, you know, we’ve adopted AI pretty pretty heavily internally as a tool to help us move faster. can you talk a little bit, Zach, about the the approach that we’ve taken here? Given that I think this you were in on the ground floor with this, right?
Zach: Yeah, yeah. So DuckDuckGo as a company takes I I think a somewhat unique approach to AI as we take a unique approach to everything we do. We very much have a culture of of things like first principles thinking and and you know our own spin on things.
Marc: Right.
Zach: And a very important part of what we do are our privacy and security. so before we rolled out many of these tools, we spent quite a bit of time internally getting comfortable with their failure modes and the blast radius potential for for those failures. One of the first places we started using AI a lot was with coding tools. So right off the bat with Cursor and then Claude Code. And you know, one, the models are very, very good at writing code. But then two, if you are effectively using tools like git and GitHub, an agent that’s limited to writing code can only do so much damage if your code base is backed up in GitHub and you don’t let, for example, people force push to main or whatever like that. Like if your GitHub is properly configured. And so we had a lot of questions like that early on. And then, you know, once once we were confident, we really were able to kind of let open the floodgates and let people really start start using both Cursor, Claude Code and now to a lesser degree Codex to to write code and automate as much of engineering as they can.
Marc: Yeah, yeah, that’s great. you mentioned privacy, right? Like that is obviously it’s a core tenet of our products. It’s also a core tenet of how we operate here. and yeah, to your point, we do tend to do things a little bit differently. so I’m curious how the privacy kind of shaped our internal workflows with respect to AI, given that, you know, w we even though we don’t collect user data, we do still have things that we obviously don’t want to to share more broadly.
Zach: Yeah. Yeah, absolutely. so our privacy engineers very much think of like a an information hierarchy of stuff that is you know very I don’t know what the word is I want to use for the scale, but there’s a there’s a high to low scale. and so there’s certain information that we don’t even save to disk in our production servers. There’s certain information that is stored in memory and is never retained anywhere in our systems.
Marc: Right.
Zach: and that is fundamentally the safest form of data. If you don’t if you don’t store a piece of information, it’s it’s very hard to accidentally store it in the wrong place and in a way that you don’t you don’t want to do. and and that kind of principle thinking applies to AI too. Like if if an AI system doesn’t have access to certain data, it It can’t do the wrong thing with that data. and one of those ways to prevent access is just by by fundamentally not not storing the data in the first place. As you go through that hierarchy, you know, there there’s information that we’re more and more comfortable with working on internally, like for example, you know, time series traffic of you know how many users are on our various websites, like that sort of aggregated data that can’t be tied to an individual person. That’s a lot safer to start saying, hey, I’m gonna, you know, I’m on the data science team, I’m gonna do some data science work and look at, you know, whatever, whatever it is that the set of questions I have. and then I think the kind of the bottom piece that is incredibly important is we pay a lot of attention to our data processing agreements that we have with all of our vendors, but especially our AI vendors. and the contractual agreements you have with people about what they are and aren’t allowed to do with your data are are also very important. so so yeah, there’s there’s a spectrum of things and you go talk to any privacy engineer, I mean you can very quickly learn how just how deep the problem goes.
Marc: Yeah, yeah, it’s interesting you mentioned principled thinking, right? Like I I think that’s something that is a core part of the culture here. And I I think it’s fair to say that we have, you know, spent a lot of time talking about how to adapt AI to our principles as opposed to adapting our principles to AI, right? which I feel like you see a lot in the industry right now.
Zach: Yeah, yeah. Yeah. Yeah. And and and and that’s I mean that’s a that’s a great point. Like two of our principles that matter a lot to me personally are questioning assumptions and validating direction.
Marc: Absolutely, yeah.
Zach: and you know, I’m I’m like the the AI, you know, waving, you know, the flag and the the pom-poms and saying, let’s go AI internally.
Marc: Right.
Zach: but like I I find I have to like, you know, remind people, just because we’re using AI doesn’t mean we throw our principles out the window. Like,
Marc: Absolutely.
Zach: I am you know, again again as a leader here, I am more interested in like like I want to validate direction too. And so I am very interested to hear from people places where AI isn’t working. Like, like I want to see, you know, the PR that’s a 7,000 line of code wall of text where a team’s going, whoa, like we can’t we can’t review this, let alone ship it.
Marc: Right.
Zach: And those those kinds of failure modes, like AI is rife with them. And we don’t, you know, we don’t just look at an AI’s output and say, well, an AI made it, so let’s ship it. Like you you should use AI as as you know, a lever against, you know, with your principles, not not as a as a reason to throw your your principles out the window. But like again, like just questioning assumptions and validating direction, like you know, AIs are notoriously bad at questioning assumptions.
Marc: Right.
Zach: Like this will happen all the time, even with the latest models, where you’ll be like, hang on a sec. Like you said X and then you said Y, which implies not X, like, but you never connected the pieces together in in in in the chain of logic.
Marc: Sure, sure.
Zach: and so I I think I think your principles matter more when you’re using AI.
Marc: that’s interesting. Yeah. Yeah, that I think that’s really well put. actually leads us into our next question to a degree. there’s yeah there’s a lot of hype right now, obviously with AI. I mean to put it mildly, right? the hype cycle is has been super interesting. there’s a lot of concern about AI in the workforce today, right? And we’re certainly not immune to that, although we don’t really see it that way. as a company. So maybe can you talk a little bit about how we’re how we’re getting so many people to use it effectively and and you know where we’re trying to get to.
Zach: Yeah. Yeah, so AI is a very powerful tool. and it’s also in some ways a a slippery tool. Like it’s it’s changing so fast. It there’s not like a long history of like I don’t know, I I can’t I can’t think of an equivalent technology, but there’s there’s not like The field has changed so fast, everybody’s brand new to it.
Marc: Yeah.
Zach: And there’s a lot of surface area of of unknown unknowns, of things you don’t know that that can go go wrong.
Marc: Well.
Zach: I I heard a joke I really like that like working with AI, it’s like a cross between Amelia Bedelia and the guy from Memento.
Marc: That’s good.
Zach: and and it’s a great joke, but it’s like, you know, one, interprets everything you said too literally. and then two, it forgets tomorrow everything that you said to it today. And like, like those two problems, like, you know. It’s you know, I I’ve been playing around with Fable a lot recently. It’s an amazing model for one-shotting really difficult programming problems. But you still get that sometimes over literality, like like what you say wasn’t quite interpreted the way you meant it, and you have to be careful of that. And then the other side of like, you know, you come in tomorrow and you have to curate context that you take over between sessions. And I I think honestly that that latter part has been the part for me that has been the most difficult to adapt to is just what do you remember and how do you remember and how do you persist that memory?
Marc: Sure, sure.
Zach: And that’s very much something you have you have to build for yourself. and and you know from the individual level to the team level, like how are we storing information in, you know, when an AI goes wrong Where do we put that learning so that we don’t, you know, forget about it again in the future?
Marc: Right. Right, right. Yeah, I think from an engineering perspective, I mean, you know, we’re both engineers, so I I I think it’s safe to speak about first hand experience there. it’s definitely changing the field, right? Like everybody acknowledges that. I don’t think it’s obviating the field. I’ve heard so many analogies about how, you know, AI is like this or like, you know, whatever.
Zach: Yeah.
Marc: One I heard that resonates a little bit again from a software engineering perspective is like nobody hand compiles code anymore, right? Generally.
Zach: Right.
Marc: And I feel like, yeah, it maybe not a perfect analogy, but it didn’t make like that didn’t obviate the need for strong engineers, right? It just changed the nature of what they’re doing on a daily basis. And I think at at most that’s what we’re looking at here potentially, but obviously it’s a hot topic.
Zach: Yeah. Yeah. Yeah. No, so I I was literally just listening in a podcast, I’m on a podcast, I was listening to podcast, I love it, this morning with Fiona Fung, who’s the manager of the Claude Code team.
Marc: Mm-hmm. Okay.
Zach: And she had just like this wild and one-off anecdote. She’s like, my old boss, like my first engineering manager I ever had, you know, called me up the other day and and she said, This guy started his career programming in punch cards. Like
Marc: Yeah, right.
Zach: Programming in punch cards.
Marc: Right.
Zach: And the guy called her up, was hey, I’m using Claude Code and you run the Claude Code product team. I just want call you tell you how cool Claude Code is.
Marc: Ha ha ha. Yeah.
Zach: and I thought that was like a really great piece of perspective. Like just like imagining the transition between programming in punch cards to programming in Python. And like I’ve never, you know, I’ve never, you know, I didn’t experience that because I wasn’t I wasn’t around. but you you forget how long the history is of of computing and how much it already has changed.
Marc: Absolutely. Absolutely. To think about people. I mean, I think most of us have at least tried some assembly, right? Like it I don’t know if they still teach that in university today, but they certainly did. even if it was just to expose you to it. But yeah, that’s not something that, you know, the average person or the average engineer is gonna be doing. But anyway.
Zach: Mm. Yeah.
Marc: So yeah, I guess we could start wrapping up. I think what we’ve talked a lot about engineering examples. I’m I’m wondering if you know, we have some non engineering examples of how we’re using this at the company.
Zach: Yeah. one of my favorites is is spreadsheets. I spent a lot of time early in my career building models in Excel. and I definitely developed like an aesthetic for like what I want an Excel model to be like. and then I stopped using Excel and both the interface changed, but like my muscle memory, you know, I never got to like the keyboard shortcut level is always point and click. But like I I’ve been frustrated a lot recently with trying to build a model in Excel and I can’t find the commands I want or necessarily even remember them. and what’s really interesting to me is is a lot of Excel models, you can you can give a data set to Claude, you can describe the problem you want to solve, you can describe the shape of the model that you want. You know, I want one tab with my input parameters, one tab with my data, and one tab with the calculations and output or something like that. And it’ll do a r a remarkably good job of building things like that. and there’s you know, there’s a lot of work that still happens in spreadsheets. and it’s really nice to be able to go to Claude Code and give it a spreadsheet and say, Hey, can you change the formatting? I’ve got all these cells where the important information is bolded and I want to remove the bolding and replace the bolding with like a red background. And that’s really tedious to do with pointing and clicking. and Claude Code, like it’s just reading the XML. And so it’s just like it’s very good at, you know. A a spreadsheet is an XML file and it can just say, Cool, yeah, I’m just gonna change around the XML structure and now instead of bolding, you’ve got red highlight.
Marc: Yeah, yeah. Got it. Yeah. Yeah. No, that’s interesting. I th I feel like yeah, as much as I’ve embraced it in, you know, my daily workflow, I still have not yet gotten the nerve up to trust it with something like a spreadsheet. But I know that a lot of people do, so I probably need to get over that.
Zach: Yeah. I mean, if your spreadsheets are in version control, as mine all are, because I’m a huge nerd, like there’s there’s limited room for it to really, really screw screw something up. Yeah. The other one so I just
Marc: Yeah. Sure. Sure. All right, great. Well, you know, please go ahead.
Zach: w one more is just like summer camp forms. Like I’ve got these forms, like I look actually no, it was a school form. I got a form from my kids’ school and it was like it was like a scan. But it like a little bit sideways, so all the fields are a little bit off. And so, you know, it like filling out a PDF form is annoying enough when there’s like the little boxes that you can type in. But it’s like,
Marc: Right. Yeah.
Zach: okay, I don’t want to print this thing out, handwrite everything. My handwriting’s terrible. Scan it, send it back. And I was like, I actually did, I said, you know, Claude, I said to Claude Cowork, here’s the form. I dictated to it what I wanted in each field. And I said, can you just iterate? Actually, it was Claude Code, not Cowork. Can you iterate on you know, take my text, turn it to text box, put it in the PDF, and then, you know, line them all up correctly with the fields, which was not easy. And it built itself a little loop where it was like, you know, it got like my information to bullet points is pretty easy. Map them to the fields is pretty easy.
Marc: Yeah.
Zach: And then it just did an iteration of okay, write them out to the spreadsheet and then just like adjust the X and Y coordinates of the boxes until it lined up well enough that I could email it back to the school. Yeah. Yeah.
Marc: Wow, that’s awesome. Yeah, I have not I wouldn’t even have thought to do that with AI. So I feel like I need to be a little more ambitious.
Zach: Yeah. Yeah. It was the same where like I I spent five minutes talking to Claude, gave it the file, sent it off in the background and it came back and it was done and I sent it. I just, I checked the data, I was like, Yep, that is correct and you know, emailed it to the school and instead of, you know, I mean, who knows? It would have been probably twenty minutes of printing it out, filling it out, scanning it, sending it back, plus a bunch of steps and I didn’t want to do
Marc: Yeah, sure, sure. All right, great. I yeah, I feel like we could talk about this all day.
Zach: me too.
Marc: but we should probably wrap up. So yeah, thanks a lot for your time and thanks for everyone else for for listening in.
Zach: Yeah. Great, great talking with you as always, Marc.