Pod Insights Practical AI Exploring the COVID-19 Open Research Dataset About this episode In the midst of the COVID-19 pandemic, Daniel and Chris have a timely conversation with Lucy Lu Wang of the Allen Institute for Artificial Intelligence about COVID-19 Open Research Dataset (CORD-19). She relates how CORD-19 was created and organized, and how researchers around the world are currently using the data to answer important COVID-19 questions that will help the world through this ongoing crisis.
Sponsors:
Linode https://linode.com/changelog – Our cloud of choice and the home of Changelog.com. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2019 OR changelog2020. To learn more and get started head to linode.com/changelog https://linode.com/changelog. AI Classroom https://datadan.io – An immersive, 3 day virtual training in AI with Practical AI co-host Daniel Whitenack. Get 10% off using the code PRACTICALAI10. To learn more and purchase tickets go to datadan.io https://datadan.io/. Featuring:
Lucy Lu Wang – Website https://llwang.net/, GitHub https://github.com/lucylw, LinkedIn https://www.linkedin.com/in/lucylw, X https://x.com/lucyluwangChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes:
Lucy Lu Wang - Google Scholar https://scholar.google.com/citations?user=REBtJOYAAAAJKaggle: COVID-19 Open Research Dataset Challenge (CORD-19) https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challengeCORD-19 Explorer https://cord-19.apps.allenai.orgSemantic Scholar | CORD-19 https://pages.semanticscholar.org/coronavirus-researchAllen Institute for AI https://allenai.orgUpcoming Events:
Register for upcoming webinars here https://practicalai.fm/webinars!
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Speaker 00 The scientific engine has really spun up to handle this current situation. As far as I know, there's been more than 4,000 papers released since January on COVID-19.
0:10 Daniel Whitenack? Wow.
0:11 Speaker 00 And that the number of papers continues to grow, but more importantly, the number of papers released every day continues to grow. So we're up to maybe more than like several hundreds of new papers a day.
0:25 Speaker 02 Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com and we're hosted on Linode cloud servers. Head to linode.com slash change log.
0:43 Speaker 02 Do not underestimate the power of the independent open cloud for developers. Yes, I'm talking about Linode. Linode is our cloud of choice and it's the home of change log.com. What we love most about Linode is their independence and their commitment to open cloud. Open cloud means being unencumbered by outside investment and maximizing value for the community, not shareholders. And that's exactly what Linode represents. No vendor lock-in, open at every layer. If you want to learn more, head to linode.com slash open. Again, linode.com slash open.
1:26 Speaker 02 Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM. Okay, take it away, guys.
1:52 Daniel Whitenack? Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with S I L International and I'm joined as always by my co-host, Chris Benson, who is a Principal AI Strategist at Lockheed Martin. How are things down in Atlanta, Chris?
2:10 Chris Benson? Doing very well down in Atlanta. Got a bit of a cold, so may cough my way through the episode, but other than that, doing great. Spring has sprung. It's beautiful.
2:18 Daniel Whitenack? Hopefully just a cold.
2:19 Chris Benson? Hopefully, yeah, we're just crossing our fingers. Took my daughter a couple of weeks, I'm hoping it was her cold, took her to a pediatrician a couple of weeks ago. We actually had to go in because it kept going and it was a frightening thing to say, "Well, it could be strep, could be this, or it could be COVID-19. We can't exclude that." And as a parent, that was like, whoa. So I've been, I've gotten through that. Just a cold. We're good.
2:41 Daniel Whitenack? Good, good.
2:42 Chris Benson? Rolling forward.
2:43 Daniel Whitenack? Well, we're surviving here in lockdown in Indiana. It's actually pretty nice outside. It's mushroom season here, so there's these like wild mushrooms that come out in Indiana just around this time. They're called morel mushrooms and we go every year hunting. So my family has some property that's all forested. No one else is there, so we've found some good times just going out there and walking through the forest and getting outside. So that's, that's been nice.
3:12 Chris Benson? Yeah, that sounds nice, whether you find any mushrooms or not.
3:14 Daniel Whitenack? Exactly. Well, I guess a related topic because it's really the only topic these days affecting us in all of our lives in a big way. We've had another episode a couple of weeks ago about the COVID QA system, which is a question answer system related to COVID-19. And they were also using this data set called CORD-19. And today we've got Lucy Liu Wong from the Allen Institute for AI. She's a research scientist there and we're going to be talking all about the CORD-19 data set, the ins and outs and the story behind it. So welcome, Lucy.
3:52 Speaker 00 Hi. Thank you, Daniel and Chris, for having me on the show.
3:56 Daniel Whitenack? Yeah, it's great to have you here. Appreciate you joining us. This is, of course, a big topic. Everyone on Twitter and all around is talking about this data set and how it's being used. So we're really excited to talk about it a little bit more here. But before we do that, I'd love to hear a little bit about your background, how you got into AI-related things and ended up at the Allen Institute.
4:21 Speaker 00 Sure. Yeah. So I guess my background's maybe a little less traditional. I started out more in biomedical engineering and physics and worked in a host of biomedical startup companies, creating medical devices. And over time, I was doing more simulations and incorporating more data science and machine learning techniques into my work and found that that was what was very motivating for me. So I decided to pursue a PhD in biomedical informatics, where I focus primarily on biomedical applications of natural language processing techniques and creating models to try to connect these automated methods
5:03 Speaker 00 with the type of improvements in clinical care and biomedical text mining that we so desperately need these days.
5:11 Daniel Whitenack? Yeah. So when you're talking about NLP for biomedical applications, are we talking mostly here about medical records and doctor's notes or whatever that is and trying to extract relevant information from those and patterns and mine those for useful things? Is that the main sort of drive there?
5:31 Speaker 00 That's definitely one aspect of things. I am also very interested in looking into the scientific literature and trying to extract entities and relationships and useful information out of that body of work. And I think that's what really I'm working on at the Allen Institute for AI, or AI2. I'm part of a team called Semantic Scholar, which I think a couple of weeks back, you had an episode about Semantic Scholar. And it's a literature search engine project. And for Semantic Scholar, we've indexed, you know, 180 million papers. There's a really rich corpus of texts to work with. And as part of the research team there, I've created a number of tools and work on a number
6:14 Speaker 00 of projects to kind of understand more about the content of that text. And that's kind of what brought us to the Chord 19 dataset is we have this kind of underlying infrastructure for processing scientific text. And we were asked to contribute some of that expertise to creating the dataset.
6:32 Daniel Whitenack? Awesome, yeah. And I'm curious, so I'll definitely reference the Semantic Scholar episode in our show notes so people can listen to that because I think that provides a really good baseline for like the sort of data that you came into this recent work having, which is a really great way to find and discover scientific information and related data across scientific literature, which is amazing. I was wondering if you could comment before we jump into Chord 19 specifically. I know we're kind of in a really interesting time where a lot of people are publishing a lot of things about COVID-19 very rapidly. What does that situation currently look like?
7:14 Daniel Whitenack? Is it like, you know, are we talking thousands of papers over how much time? How rapidly are they coming out?
7:23 Speaker 00 Yeah, I think the scientific engine has really spun up to handle this current situation. As far as I know, there's been more than 4,000 papers released since January on COVID-19.
7:36 Daniel Whitenack? Wow.
7:36 Speaker 00 And that the number of papers continues to grow, but more importantly, the number of papers released every day continues to grow. So we're up to maybe more than like several hundreds of new papers a day.
7:49 Daniel Whitenack? Wow.
7:49 Speaker 00 And it's kind of intimidating to look at this source of information and kind of see what people are discovering.
7:57 Chris Benson? So the fact that you have so many new coming in every day, are you refreshing the dataset? Is the dataset static at some point in time? Or is it something that you're constantly updating and refreshing?
8:07 Speaker 00 Yeah, so I guess maybe folks already know what the CORD 19 dataset is, but it's a kind of a collection of papers about COVID-19 research, including historic coronavirus research. So we have a collection of historic research, and then we also have all of the new research that is being released daily. We update the dataset currently at a weekly cadence, but we are rapidly moving to a daily cadence since there's just so many new papers released every day.
8:35 Daniel Whitenack? Yeah, so since we kind of went that direction, I was wondering if you could maybe tell a little bit of the story of how this dataset came about. I mean, obviously, you have data about scientific literature within Semantic Scholar, and you're already doing certain things as relating to, you know, tracking entities or topics covered in those. How did the idea for CORD 19 come about? And I know that there's others involved in this, too. So there's Allen AI, but there's also, I think, Microsoft and the Chan Zuckerberg Foundation and others. So how did this come about?
9:08 Speaker 00 Yeah, so the entire project is kind of a coordinated effort by the White House Office of Science and Technology Policy. And I think sometime in early March, a group at Georgetown, the Center for Security and Emerging Technologies, Georgetown CSET, they reached out to us at Allen AI to help coordinate the release of this dataset along with a couple of different organizations. You mentioned MSR, Microsoft Research, Chan Zuckerberg. CAGL was also involved, and the National Library of Medicine, which is part of the NIH. So all of these groups were going to come together to essentially create this dataset to help, I guess, create text mining and information retrieval tools that could assist medical experts
9:51 Speaker 00 in understanding more of what was going on with the epidemic. And for Allen AI, the way that we got involved... Is we had recently sort of created a new pipeline to revamp our research corpus, our open research corpus. So we had a pipeline for essentially taking these paper documents, which are traditionally in kind of a PDF format, not very easy for text mining, not very accessible, and converting them into kind of like a structured full text format where you could run these natural language processing models on them more easily. So that's sort of our major contribution to the data set is the pipeline for both harmonizing the paper metadata that we've collected over the years and also producing
10:36 Speaker 00 these structured full text parses so that we can run our models over that text.
10:42 Daniel Whitenack? And I know one of the big things that we talked about when we talked about Symantec Scholar was the ability to kind of find relevant data that might be buried in the wealth of scientific literature that we have about a certain subject that is of interest. So when you came to CORD 19, there's the extraction of the metadata, the actual content of the paper, but then how do you even go about saying like, these are all the papers related to coronavirus? I mean, I know, and I'm a little bit ignorant on this subject, so you'll have to forgive me. I know that coronavirus is kind of a family of things.
11:22 Daniel Whitenack? It's not just this COVID-19, which is associated with coronavirus, but there's coronavirus associated with the common cold and all these things. So how do you go about saying like, this is what we're scoping down our data set to and finding that and along with that, deciding what you're going to exclude, I guess, as well?
11:43 Speaker 00 It's a great question, and I think it's a question with very open answers. So what we started with were a couple of trusted sources that we knew needed to be included in this data set. And those sources were a collection of papers curated by the World Health Organization on COVID-19 specifically. And we also performed searches over PubMed Central, which is a biomedical paper repository run by the NLM, the National Library of Medicine. As well as these pre-print servers Bio Archive and Med Archive, which were publishing the latest research on COVID-19. And we went out and collected papers from these sources
12:28 Speaker 00 using a set of essentially like keyword searches to make sure that they were relevant to both COVID-19 or the family of coronaviruses in general. Because I think historical coronaviruses like SARS and MERS are also extremely relevant in the current case.
12:46 Chris Benson? I'm kind of curious, as you reached out and made the dataset available, and you look across some of your partner websites, like Kaggle has a call to action and stuff, and you're trying to get AI practitioners and data scientists to focus on kind of important questions that need answering for this purpose. How do you provide guidance in that way for people who are going to engage on the dataset? Is it something where people just grab it and do whatever they want? Is there any kind of organization across teams? You know, there's a lot of human factors involved in this. And how was that conceived?
13:17 Speaker 00 Yeah, so there were a lot of challenges. For Kaggle, when we opened the challenge initially, the COVID-19 challenge, there was a set of kind of 10 slightly open-ended clinical questions which were given to the community. And the engagement at Kaggle, the response we've received, has been like absolutely incredible. There's been like millions of views on the landing pages. The dataset has been downloaded 70,000 times or more. There's been lots of teams that have cropped up and self-organized to work on this dataset. I think there's a group called Corena Y. That's like several hundred data scientists and medical experts who've bonded together
14:00 Speaker 00 to work on the COVID-19 dataset and other coronavirus datasets. And we really want to just offer support to these community members. So there's, you know, a couple of sources of information that we've created to help facilitate these things. So on Kaggle, the forums have been super active. There have been a lot of people answering questions for each other, including from the organizations that have created these datasets. We've also established like a discourse to answer questions specifically about COVID-19, the dataset.
14:32 Speaker 00 So that's a great place to get answers. And finally, I think for these Kaggle challenges and for these shared tasks, one of the things that we're really trying to do by hosting these shared tasks is to connect ML experts with the medical community and kind of experts who can judge the answers that are being retrieved and extracted by these machine learning experts and see whether they have practical application in the clinic. So that's been a challenge.
15:05 Daniel Whitenack? What's up? This is Daniel Whitenack, one of your Practical AI co-hosts, and I hope you're enjoying this episode and staying healthy during these crazy times. I'm working on some pretty cool AI stuff here from my home office, but I've also found that I'm having to get a bit creative and be intentional when it comes to honing my AI skills and virtually connecting with the AI community. If you're in a similar situation or you've been inspired by the Practical AI we talk about on this show, I want to invite you to a live online AI training event I'm hosting this May called AI Classroom. In AI Classroom, I'm going to teach you the practical skills I've learned over the years using the latest open source AI technology.
15:48 Daniel Whitenack? You'll learn AI theory along with practical, hands-on implementations in both PyTorch and TensorFlow. And after the training, you'll be able to understand the latest AI models, implement your own models in code, train computer vision and NLP models, create model inference servers, and experiment with state-of-the-art methods like reinforcement learning. AI Classroom is taking place this May. It'll be taking place live and completely online in a high-quality virtual classroom, so no travel is required. There'll also be two cohorts with convenient time zones for Eastern and Western hemispheres. Don't miss out. Tickets and more information are available at datadan.io. That's datadan.io.
16:32 Daniel Whitenack? And Practical AI listeners can use the code PRACTICALAI10 for 10% off. See you online in AI Classroom.
16:51 Daniel Whitenack? So Lucy, you just brought up something that I think is really interesting, which is the sort of interaction between the AI community and the medical community. And I was actually wondering while you were talking about like, okay, this Core 19 dataset exists, but I know I have some AI expertise, but I don't necessarily have a lot of medical expertise outside of knowing like that I should wash my hands and these other things that kind of the top five that have been going around. I guess I was wondering, as you've got more experience with this kind of intersection between the AI community and the medical community, what has that interaction been like in the past? Has there been much overlap between the AI
17:34 Daniel Whitenack? community and medical practitioners? And then, secondly, as we enter into this new Core 19 challenge, has that changed in any sort of way or been rapidly, you know, advancing in any sort of way?
17:48 Speaker 00 I've sort of worked on the intersection of these communities for a number of years, and I think there's a lot of great collaborations going on. I think a lot of folks in the computing community are incredibly motivated by these like very practical questions that need to be addressed, ways to improve patient care, ways to help with drug development or vaccine development and questions of these nature, this nature. As for kind of COVID-19 specific initiatives, so I can kind of give you two anecdotes for ways that we had annotators, or I guess medical experts interact with computing experts. So for the Kaggle challenge, seems that what is
18:33 Speaker 00 happening is a lot of people are developing different systems, different information retrieval, different information extraction systems, and those systems need to be reviewed by experts for usefulness. So in Kaggle, there's essentially kind of like an army of medical students and other people who are willing to provide their medical expertise and volunteer their medical expertise. Who are actually going through and manually reviewing a lot of the extractions that are coming out of these CAGL challenges and creating these kind of living systematic review pages with the answers to some of these questions. So if you go to the CAGL page, you can see these reviews kind of being created in real
19:17 Speaker 00 time and updated in real time as new literature is released. And another thing that I've been involved in lately is we're kind of running a TREC challenge on this dataset. So TREC is the Text Retrieval Conference, and it's been a project at NIST, which is the National Institute of Standards and Technology for the last 20 years. And these folks are really good at information retrieval and judging information retrieval systems. And the way that these systems are judged is by having expert medical annotators review all the results and provide gold rankings of what is most relevant to query and what is the kind of least relevant.
20:01 Speaker 00 So there's a lot of this, like how incorporating experts in the loop, incorporating humans in the loop to kind of bolster our machine learning systems. And that is not something that we're going to be moving away from anytime soon.
20:15 Chris Benson? As you're talking here, I'm looking at the various questions there that are listed on CAGL, the tasks to go answer, and kind of extending this thing about this collaboration between the AI community and the medical community. The questions themselves, where did they originate from? How were they decided as the important questions that we could all give a shot at going and answering with the dataset? How was that?
20:37 Speaker 00 Yeah, I may be wrong, but I believe this set of questions originated from the White House Office of Science and Technology Policy in collaboration with CAGL. And you have to understand, so this challenge and the dataset, during the early days, we literally had just a few days to turn around this dataset, put it out there, and kind of publish this challenge. We wanted people to start looking at this as quickly as possible. So there were, you know, a lot of the questions that you see on CAGL right now are very open-ended, they can be interpreted in different ways, and as time has gone on, as we've learned in this last month. There's actually, some of those questions are more useful in clinic. Uh, some of those questions are less useful.
21:20 Speaker 00 Like clinicians already know the answers to some of these questions. So now, as we move into the second month of this challenge, there will be new, a new batch of questions released, um, to kind of motivate new work and the kind of questions that have not yet been answered by the community.
21:37 Daniel Whitenack? Yeah, so I, I'm curious with that. It seems like you could have various bottlenecks in, in this situation. And one of those, I think you highlighted is this sort of useful interaction between medical practitioners and the AI people that are trying to do something with the, the data set. So I was wondering, do you have a sort of healthy community of, of medical practitioners that are, are very deeply involved in kind of looking at what's coming through Kaggle or these other teams that are kind of self-organizing? Cause I know that's one, having worked at a nonprofit for a bit. One of the things I've seen is like, oh, people really get behind like the social good like challenge
22:18 Daniel Whitenack? and they work on it like on the hackathon on the weekend and then the project kind of dies. So, um, how, how are you kind of, what are the ways for people? If I want to step into the, some cord 19 related work, what are the ways that I can kind of step into that? But also get connected with the right sort of medical people to make sure that what I'm doing is useful and not just like an kind of interesting weekend thing?
22:45 Speaker 00 Yeah, of course. Uh, I think like now that COVID-19 has sort of taken over all of our lives, um, a lot of people are feeling very motivated to do something in this, uh, direction, um, contribute their skills. And I think I, I mentioned some groups earlier, groups like Corona Y, which is self-organizing to analyze this type of data. And that's a group made up of data scientists, machine learning experts, medical practitioners and so on. And so it's like a good place to get feedback on one's work. And CAGL forums, similarly, I think there are a couple of threads out there essentially continuing this discussion of how to connect to medical experts, how to verify results and
23:29 Speaker 00 so on. A lot of people have, you know, taken it upon themselves to build systems. There's tons of systems out there for searching cord 19, for extracting information out of cord 19. And, you know, to be perfectly honest, not all of those systems are going to be like highly usable or used by any kind of clinical audience. But we, as a community, need to essentially figure out which of those systems are most promising, figure out where to expend additional energy, like more development time and so on. And that's where our annotators come in.
24:03 Chris Benson? Going back to what you said a moment ago, you have put this together so quickly and gotten it out there and the whole world has, has kind of dived into it, the whole world of data science at least. And, you know, that's very different from how most communities, you know, form. And I'm kind of wondering, with hindsight, totally recognizing that you had no choice, you had to get it out there and you did a fantastic job with what the kind of pressure that you were under. If you could go back, knowing what you know today, what are some of the things around community that you would like to have done and that maybe going forward as you're looking at like the tasks and how to move us going into the second month, how to move us in the next set of directions that you need people to take? What are some of the ideas that you're planning to implement there to kind of evolve this process?
24:48 Speaker 00 Let me try to unpack that.
24:49 Chris Benson? No worries. Any way you want, it's fine.
24:52 Speaker 00 Sure. I think we've learned a ton over the last month. Speaking with some of our collaborators at CAGL, like Anthony Goldblum was mentioning, how they've really fallen, or CAGL has fallen into this place with this challenge where they're kind of in new territory. Like, the type of challenge that the cord 19 challenge is, is very unlike most of the challenges hosted by CAGL. And there's a very open-ended nature of it. We're like trying to discover answers, but there's no sense of like what a gold answer is. And they've reacted to that in a very like, kind of wonderful way, by essentially harnessing these medical students as a resource to make judgments on people's extractions and
25:37 Speaker 00 kind of putting an effort there where it seemed like the results were most useful or were going to be the most useful. So I guess that's one thing, which is trying to figure out as early as possible. Where the most useful results are and putting in additional effort there and maybe even like abandoning things that are not worth pursuing. And then I don't even remember what the second half of that question was.
26:01 Chris Benson? No, it was things that you might be thinking going forward, but no worries, we can come back to that.
26:06 Speaker 00 That's right. I mean, I have lots of thoughts on that as well. So certainly, we're going to be supporting cord 19 for as long as it makes sense to do so, certainly until the epidemic seems to wind down a bit. There have been lots of requests for additional features and additional content. So that is one of our priorities. Additional content comes in two forms. One is simply providing more faithful parses of the papers. So right now, first things might be including things like inbound and outbound citations, tables and figures, other places where the answers might be. And these are, these have been requested by a lot of folks.
26:51 Speaker 00 Another is content in the sense of more papers. So one thing that we've been very grateful for is that a lot of publishers have made their COVID-19 articles open access. And by making them open access, they've allowed us to release a dataset like cord. But the fact is, if you look kind of at the dataset and at the papers that are being cited by the papers in the dataset, there's actually a lot of content that are outside of this direct core set of articles on COVID-19 and coronaviruses that are also very relevant to the content of the dataset. So it would be great if we could work with publishers or they could work with us essentially
27:35 Speaker 00 to provide additional content that could be useful for kind of discovering information about COVID-19 and its treatments.
27:43 Daniel Whitenack? Yeah, so it's like you've amassed this kind of central hub, but from that hub of papers, obviously those papers cite other papers and those papers cite other papers and there's other related work and you can kind of go down a rabbit hole. I know we talked on the Semantic Scholar episode about the sort of graph of relations and that sort of thing as papers cite each other. I mean, there is a wealth of papers already in the dataset. What is the kind of current size of the dataset? And you know, you mentioned a few different sources. Could you just give us a sense of kind of the descriptive statistics of the dataset at this point, I guess?
28:21 Speaker 00 Sure, probably should have started with that.
28:24 Daniel Whitenack? Yeah.
28:24 Speaker 00 So the dataset currently consists of more than 50,000 papers and approximately 40,000 of those papers have full text content available. And these papers, as I mentioned, come from a diversity of sources. There's a list of WHO COVID-19 papers, several hundred of those that have been curated by the WHO. And there's preprints from Bio Archive and Med Archive, that's numbers in the thousands. And actually, the vast majority of papers come to us via PubMed Central. And this number will actually continue to grow because
29:05 Speaker 00 many publishers are now depositing all COVID-19 content into PubMed Central.
29:11 Chris Benson? So Lucy, I'm kind of curious. So recognizing that your responsibility has been putting this together and getting it out into the world, I imagine that you've probably talked with various teams or at least observed some of the efforts and work that's going on. And so I'm kind of curious, what are some of the more interesting or innovative or, you know, pick your adjective of the efforts that you've heard of and kind of gone, wow, that's a pretty cool way of approaching this? Any stories to tell us on that?
29:38 Speaker 00 I think folks have done a really great job diving deeply into this dataset. There's so many different kind of search engines that have cropped up over this dataset. If you go to the dataset landing page on Semantic Scholar, there's Like a list of maybe several dozen that we've heard of and are enumerating there. I'm sure there's many more that we aren't aware of. People are pursuing lots of different technologies for these kind of search engines. Some are kind of using the latest kind of state of the art transformer models, neural models for ranking. I think COVID X from Waterloo and NYU is using kind of the latest side T5 model. Very cool stuff.
30:22 Speaker 00 And some of the search engines are actually using very traditional methods, using Lucene or Elasticsearch and focusing more on how to search and filter using entities or other paper features. And we've kind of a funny thing that we heard from Kaggle is that for many of the questions on the CORD 19 challenge, something simpler methods, the more traditional methods have actually worked better for kind of extracting answers. And this came as a surprise to me.
30:55 Chris Benson? So I have a follow up on that. And that is like for working better, you know, kind of putting that in quotes. What does that mean? Who out there is looking at the various results that are coming back from teams and making those evaluations and kind of, and obviously you've already said that you'll be steering in the new tasks on Kaggle, learning what we know. Who's making those evaluations and the decisions associated with that to keep everything focused?
31:22 Speaker 00 So I think this is a primarily work that has been taken on by organizers at Kaggle and kind of medical students that they have had come in and evaluate some of these answers. So really like they put in a ton of effort in curating these results. And I think as for like the metrics that we use to judge these results, I think currently like they're mostly kind of information retrieval based metrics of success.
31:56 Daniel Whitenack? Yeah, to give people an idea, I'm on the Kaggle website and just looking at some of the tasks to maybe for those listeners out there that don't have a good idea of the kind of scope of these, there's information or some of the tasks that are listed are what do we know about COVID-19 risk factors, what do we know about vaccines and therapeutics, what has been published about medical care? And if I kind of dive into each of these, there's a number of submissions, but things like, you know, cord 19 analysis with sentence embeddings, COVID-19 literature clustering, full text search of research papers. And
32:37 Daniel Whitenack? so you can kind of already see, and here's another one, BERT squad for semantic corpus search. So you get a sense, I think, of what you were talking about, Lucy, that some people are kind of going after these sort of transformer-based, maybe extractive QA sort of things. Others are maybe using full text search capabilities that have been out there for a while, like elastic search sort of capabilities, like we talked about on the previous episode. Also, if I'm looking at the data, just to kind of have something in people's mind, I'm seeing, of course, you have the categories of source, like the bio archive and that sort of thing. But if I'm looking at the individual papers, I can see the abstract of the paper and the
33:21 Daniel Whitenack? body text. And if I'm looking at, you know, the body text of this one, I'm reading things like to assess the effects of truncation of the poly C tract on replication, blah, blah, blah, which some of that doesn't mean a lot to me, but this is the sort of data that's in there. I was wondering, as a complete noob to, you know, a lot of this medical terminology, what are maybe some good ways, I know you've got this kind of co-vis from the Allen Institute, which is helping kind of explore some of the, looks like the genes and cells and diseases and chemicals that are connected throughout the data set. What are some good ways of
34:02 Daniel Whitenack? kind of onboarding into the cord 19 work for people that might be sort of new to medical terminology? Are there ways to kind of? Pick up some of that or explore some of it and form some of those connections in a reasonable sort of way?
34:20 Chris Benson? The domain knowledge?
34:21 Speaker 00 Yeah, I think domain knowledge is one of the greatest barriers for working on this data set. And thanks for mentioning the CoViz project. That's a tool that was released by Alan AI for exploring this data set in a slightly more meaningful way. And for that tool, we essentially ran models to perform extractions of these entities from the text, entities of different classes like drugs, genes, diseases, phenotypes, things of that nature, and created a visualization to allow you to browse the relationships that are most prevalent between pairs of these entities.
35:06 Speaker 00 So that's a great way to kind of explore what's in the data set. There's also just like exploring the articles. We have a kind of a CORD 19 Explorer to help you do that. But in general, I think unless you're willing to spend sort of a couple of years of your life in medical school, it is very hard to understand what some of these terms mean. Certainly knowing what class or category of entity is being mentioned is important. So knowing something is a protein, knowing something is a receptor, knowing something belongs to a particular biological pathway. These are kind of key for gaining an initial understanding of what is being said in a text snippet. But that's also why we need medical experts to assess the actual utility of some of these extractions.
35:49 Chris Benson? So I'm wondering, I happen to have a daughter who is a third year medical student. And I've told her very recently, because we had the other episode about this and stuff, but she hadn't been aware of it. Is there any need to connect with medical schools? Has anybody kind of taken that on to try to gather those together and stuff? Because it's been, obviously, there's been an enormous effort in a very short amount of time, totally recognizing the constraints of the reality that we're in today.
36:16 Speaker 00 Yeah.
36:17 Chris Benson? Is that something that y'all are kind of thinking about in terms of going forward? Maybe for stage two or whatever you want to call it?
36:22 Speaker 00 Yeah, absolutely. I'm sure your daughter knows that, I mean, I don't know if she's still, if medical school is continuing as usual, but I think during the third and fourth years, you're mostly in clinic.
36:33 Chris Benson? Yeah.
36:33 Speaker 00 So I know of a lot of medical schools where there are these kind of more senior medical students who really want to Contribute how they can, but really aren't able to be in clinic at this moment.
36:47 Chris Benson? Yeah, they've been kicked out of the ER for, you know, and it's they're working with the health department locally. And I think that kind of alternative work is really common right now for advanced medical students.
36:57 Speaker 00 Yeah, exactly. So for the track task that I was mentioning that we're hosting, we actually are enlisting medical students from a number of institutions, from the Oregon Health and Science University and University of Texas and University of Washington to kind of help with provide annotations on some of these extractions. So I think depending on where your daughter is or where some of these medical students are, there's probably going to be other initiatives like this one that really need their help. So I would definitely encourage anyone to look out for that.
37:33 Chris Benson? I guess I have one more follow up to that that you just mentioned. Do you think recognizing that and recognizing that we're going to be past this moment at some point, this kind of very unique moment in our history, but just as, you know, the kind of the widespread introduction of like open source software really changed, you know, industry itself from being highly proprietary to being, you know, open source became not only a part of business models, but even underlying part of a lot of commercial software that's out there. And it fundamentally changed how that worked. Do you think this is a moment just because COVID-19 passes us, you know, and we get past this, that that maybe there are other challenges, whether they be things that we've been dealing with a long time,
38:18 Chris Benson? like cancers or or new things that may come, that this may fundamentally change how we attack, you know, really hard medical challenges with AI and that integration with the communities that that has happened out of necessity?
38:31 Speaker 00 I'm certainly hoping that to be the case. So we've definitely seen what people can do when they come together for a month or two. And it's it's incredible. Like there's so many people being engaged and building like interesting tools and useful tools. I think there's a couple of maybe things that I'd love for us to be able to extend into the future. One thing is definitely publishers coming together to release more open access content on really kind of.
39:03 Speaker 00 Important topics such as COVID-19 and then the community coming together, especially crossing boundaries, crossing boundaries between computing, the computing community, the medical community and policymakers to really build something useful.
39:20 Daniel Whitenack? So I'm curious, a little bit of a follow up, I guess, to that, that question as well. I know there've been things, as I've worked on related work with SIL where I was thinking, oh, if I would have done this prior to this crisis, like I would be able to do something better than what I'm able to do now. In hindsight, it's easy to, to see those opportunities. I'm curious, on your side with your own research and work. I mean, you were kind of, I'm assuming you were working on various things related to Semantic Scholar prior to this crisis and now your, your head's down working on, on Cord 19 and, and getting this in shape. What are you interested in exploring in the future?
40:05 Daniel Whitenack? Not necessarily Cord 19 related, but how has this whole process shifted, like what you want to work on in your own research in the future?
40:15 Speaker 00 Yeah, so I think this brings up a slightly earlier point, which is we really became involved in the creation of this data set because we had at Semantic Scholar built a bunch of infrastructure for scientific papers. And a collaborator of mine, Kyle Lowe, and I had also been working for the past nearly year on a way of essentially creating a full text extraction pipeline for some of these papers that we are using in Cord 19 today. So a lot of this was infrastructural work. It's not particularly glamorous, but it is really important and it really became more important in light of what happened in the last few months. I guess one thing is infrastructural improvements
40:59 Speaker 00 can be really important, even if it's not particularly sexy.
41:04 Speaker 00 And then kind of going forwards, there's certainly things I care about besides kind of creating data sets of papers. And my research focuses on making scientific literature, making this content more available to biomedical researchers and more understandable. And as you mentioned before, there's so many entities, so many kind of like very domain specific words and relationships that exist in the biomedical literature. And even for someone who is a domain expert, some of those terms can be very hard to parse through and understand. So a lot of my ongoing projects are trying to create systems that understand particular types of relationships. For example, those that understand drug-drug interactions
41:49 Speaker 00 or can mine them out of the literature, those that can understand medical images better. These are the types of projects that I am hoping to continue to work on in the future.
41:59 Daniel Whitenack? Awesome. We're for sure going to have the links to the main dataset website in the show notes, along with the Kaggle challenge and the various other projects and groups that you've talked about. Really appreciate you coming on the show and describing a bit more about the dataset, how it came about, and your own work with it. Really encouraged by the work that the Semantic Scholar team and collaborators are doing. And thank you for your hard work on this and taking time to talk to us.
42:29 Speaker 00 Yeah, thank you so much for having me. And we really hope that other folks are encouraged to kind of contribute and become involved in this project.
42:36 Daniel Whitenack? Yes, please do.
42:41 Speaker 04 Thank you for listening to this episode of Practical AI. More like this at changelog.com-practicalai. There you'll find our latest, as well as lists of our most popular episodes and the ones we recommend. If this show has helped you on your AI journey, please leave us a five-star review on Apple Podcasts, heart us on Spotify, star us on Overcast, and tell a friend what they're missing out on. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo. And our music is brought to you by The Beat Freak, Breakmaster Cylinder. We have awesome sponsors. Please support them. They support Thanks again to Fastly, Linode, and Rollbar. If you and your organization could benefit from speaking directly to all the AI practitioners out there, you should sponsor the show.
43:25 Speaker 04 Podcast advertising is one of the most effective ways to spread your message in an authentic way. Plus, you get the added bonus of supporting something you love. That's all for now. We'll talk to you next time.
Previous Achieving provably beneficial, human-compatible AI Next Reinforcement learning for chip design Page updated October 26, 2026 · transcript and summary are machine-generated