Janani Sekar:
Okay. Hello everyone, and welcome back to the series on Agentic AI. In this module, I'm very excited to talk to you all about exactly how to use Agentic AI, but also what are some caveats and things that we should be wary of as we are using these tools. So coming in, one framework that I find that is pretty helpful to think about the different tasks that AI agents can do is this idea of which tasks can we augmented with AI and which tasks can we automate with AI? And as researchers, a lot of the tasks that we do involve things like critical thinking and writing and programming. And as a bit of a spoiler, most of these tasks can be augmented instead of being fully automated. And so what does it look like? What do I mean when I say tasks that are augmented or automated?

Can we put a little bit more color to this? Absolutely. Some examples of augmented tasks include things like programming. When you ask an agent or you ask a language model to run a regression, it is on you to human supervise that a little bit by being specific about the structural form of that model that you're trying to run or that regression. Otherwise, you might get an agent or a program that makes certain assumptions that you may or may not agree with or had planned to make. Writing is pretty similar. If you've ever tried to use deep research or one of those agents to write an outline for a paper or even write a paragraph for your paper directly, you've probably experienced that it's not necessarily the best quality. And if you didn't review that outline before telling it to continue writing, what you might get is pretty bad.

And we don't want that type of thing to make it into a paper or even a draft. We definitely don't want to put that in front of our PIs. And then in critical thinking, a lot of folks, especially with newer models that have stronger reasoning capacities, use these agents to poke holes in their logic, to try and soundboard, to give them some writing and say, "Hey, is there anything that I'm missing here? Any information that we should explore further?" And this is also going to be a heavily human involved task. If you don't have your logic written down, then your agent is obviously not going to be able to identify gaps in it. In contrast, here are some things that we can maybe automate. The one on the left is something that I personally do all the time. So installing software and checking software, including doing things like submitting jobs on your compute cluster, I just gave my Claude code, it's a desktop coding agent, the documentation for the Mercury Booth compute cluster.

And now when I tell Claude, "Hey, can you submit this job to Mercury, which is our compute cluster booth, it's able to do that immediately." Data maintenance is another example. Suppose that you are repeatedly fetching data from the Federal Reserve or some other database in equivalent daily intervals, and you've already done that once or twice. You have a script that works, you know that it works, you don't need to keep checking it. You can outsource that to an agent, just make sure that it runs that program once a day to get you that data. So if you can think of any other tasks that you can automate, I definitely encourage you to try. I would say the main thing that you should be mindful of here is, can I do this task with very, very little or minimal human oversight? There are a lot of papers that discuss this augmented, automated space of tasks that AI can do in research.

I'll highlight a few here, Coronet 2023 and 2025. Both are really, really great resources. They also come with cookbooks, so actual implementation plans. If you are new to this space and you want to see how to use an AI agent to do a literature review or write a draft, definitely check that out. The other source that I wanted to flag here is Dell in 2025. The paper is called Deep Learning for Economists. And it's actually not about AI agents. It's more about how we can use machine learning methods to advance economics research, but there is a brief discussion of things like good prompting strategies in this paper. And what I really like about this paper is it breaks down the insides and the theory behind how language models and agents work. So if you want to know more about these tools and you want to feel less like they're impenetrable black boxes, this is a really great paper to read.

It will also be linked and referenced at the very, very end of this presentation. So we said the tasks that we do as researchers are difficult to augment exactly or difficult to automate. Why exactly is that? And what are some things that go wrong when we try to automate tasks that we shouldn't? Well, think about things like literature reviews and synthesis. What we find is that when we outsource these tasks to agents, sometimes they might return spurious papers. We have other things that can go wrong in the OCR space. Sometimes an entire document is left out, parts of the document aren't done correctly. We have to human verify what happens. We can try to label data sources or annotate data using language models and AI. Things often goes wrong when we do that. We'll talk about what exactly goes wrong there in a little bit more detail on the next slide.

And the same kind of pitfalls arise when we get models to interpret results. And what exactly are these pitfalls? What types of errors do agents make in this space? There's a little bit more literature that I wanted to reference that discusses this in more detail. Again, I'm not going to summarize all of these papers in great detail, but the goal is rather to direct you all to them so that if you're interested in understanding what goes wrong here, both empirically and theoretically, what have other people in this discipline found as best practices for using these models? I'm going to shout out a few papers right now. The first is Bowman et al. In 2025. Specifically, this is a great paper which documents how changing the prompt with which we talk to an agent or a language model for annotating our data produces wildly different results.

So imagine that you're using a language model to try and tell you, is this news positive, negative, or neutral about this company to try and use that as a way to predict stock price movements. Depending on how you phrase that prompt, the language model's annotation of that data point might be entirely different, which means that we are not creating findings or presenting findings that are very robust to the choice of words with which we communicate. That is obviously not a really great position to end up in. So you should be mindful of trying to test out different prompts and seeing how much that changes your results. And in fact, as you do that and you see that your results do change, there are also a body of literature that discusses how to debias your labels in this process. So you can use your language models and your agents to label your data if you do have some human validated ground truth.

And if you want to understand how to better do this in a statistically significant and mathematically defensible way, there are two really great papers that I've mentioned over here. And then the third category of paper that I will highlight is that bit about using LLMs to interpret results. So you might say, "That sounds like a terrible idea to use an LLM to annotate my data to begin with. I would never do that. I wouldn't even let an LLM or an agent run a regression for me. I'm still trusting myself more. I'm going to do that by myself." All I'm going to do is give it the output table and ask it, "What does it mean if this was the sign on my coefficient?" You can't even do that without validating the results because there is a lot of evidence that shows that LLMs and agents will systematically overgeneralize the findings from a analysis or from a model in a way that actually has meaningful impacts on the statistical significance of those results.
So we don't want problems with making causal statements when those things don't exist. We don't want problems with interpreting the sign of our coefficients in the wrong way. So please be very mindful when you think about using your language models and agents for these tasks.

So if it is this challenging and there are this many things that can go wrong when we're using agents for science and scientific discovery and research, why even bother? Well, it's not to say that they can't help us. It's more to say that there are principles for responsible use that we should keep in our heads as we use these tools. And this is a really nice framework from an AI initiative at the University of Cambridge on responsible LLM use for science. And the three main pillars that they highlight are data quality. So making sure that the language models that you use were trained on good data, making sure that you're using language models that were evaluated and were validated against what we call benchmarks. And so benchmarks are large data sets that have been curated by domain experts to assess a model's ability on a particular task.

And so there are benchmarks for medical science and benchmarks for legal work and benchmarks for how good a model is at doing the SAT or doing math or solving proofs. And if you're using your AI agent or language model in one of those domains, you can defend the fact that the information it provides is reasonably of quality if it performs well on an agreed upon benchmark in that domain. And finally, as long as you report the use of AI and are transparent about where you used AI in the work that you produce and that you put out into the world, it is going to be okay to use these tools, obviously, as long as you disclose their use. All right, fabulous. So if there are any takeaways that I have for you all from this module, it is not to say that we should not use AI.

It's not to say that these mistakes are the end of the world and that that should preclude us from exploring this new technology, but rather that with new technology also comes new categories of error and we need to know how to navigate this landscape and be sophisticated users that understand the limitations as well as the power of this incredible, incredible set of tools.