Unleash the Power of GPT 3.5: Scrape ArXiv Papers!

Table of Contents

1. Introduction
2. Approaches for Getting Archive Papers
   1. Dumb Approach
   2. Sophisticated Approach
3. Dumb Approach: Searching on Archive Using Python
4. Sophisticated Approach: Extracting References from Papers
5. Using a Large Language Model to Extract References
6. Performing a Google Search for Paper Titles
7. Building a Graph of Relevant Papers
8. The Archive Object and its Functions
9. Getting Paper Information from Archive API
10. Conclusion

Approaches for Getting Archive Papers

In the next part of the AI System project, we will discuss how to obtain all the archive papers necessary to feed into our AI assistant's memory. There are two approaches that can be taken to accomplish this task: a dumb approach and a more sophisticated approach.

Dumb Approach: Searching on Archive Using Python

The dumb approach involves searching on Archive using Python. We perform an advanced search on the categories related to computer science and computation and language. We retrieve the top 1000 papers sorted by date. While this approach is simple, it may not be suitable if we want to target a specific field of study.

Sophisticated Approach: Extracting References from Papers

The more sophisticated approach involves extracting references from papers. We start by selecting one paper of interest and extract the archive ID. Then, we pass the paper through a large language model and ask it to return the references Mentioned in the paper. From these references, we perform a Google search to find the actual IDs of the papers and retrieve them.

Using this approach, we build a graph of relevant papers, similar to the process of reading a paper and diving into its references to gain a deeper understanding of the topic.

Dumb Approach: Searching on Archive Using Python

To retrieve archive papers using the dumb approach, we use the Archive API in Python. We perform a search with the category as "cscl" and sort the results by date. We retrieve the top 100 results and obtain information such as the authors, categories, and ID of each paper. We also download the PDF of the papers using the provided PDF URL.

Sophisticated Approach: Extracting References from Papers

In the sophisticated approach, we extract references from papers using a large language model. We split the paper Based on the "references" keyword and extract the relevant information. We then use this information to feed into the large language model, which helps us extract titles, authors, and years of papers mentioned in the references section.

Performing a Google search for the paper titles, we retrieve the archive IDs of the papers and download them. This process allows us to Create a graph of relevant papers based on their references.

Using a Large Language Model to Extract References

To extract references from papers, we use a large language model like OpenAI's GPT-3. We provide the model with a prompt template that includes the references section and ask it to extract the titles, authors, and years of the referenced papers. The extracted information is then used to build our graph of papers.

The use of a large language model enhances the accuracy and efficiency of extracting references compared to the dumb approach.

Performing a Google Search for Paper Titles

To find the archive IDs of the papers mentioned in the references, we perform a Google search using the paper titles. We extract the archive paper links from the search results and obtain the IDs. This allows us to identify and retrieve the specific papers required for our AI assistant.

The Google search helps us locate papers that may not be directly available on Archive, providing a comprehensive collection of relevant papers.

Building a Graph of Relevant Papers

By extracting references and retrieving the archive IDs of the mentioned papers, we can build a graph of relevant papers. This graph connects papers based on their references, creating a knowledge graph that emulates the process of diving into related papers for better understanding.

The graph allows our AI assistant to access a vast network of knowledge, providing comprehensive and in-depth information on specific topics.

The Archive Object and its Functions

To streamline the process of obtaining archive papers, we create an "archive" object that contains functions for handling various tasks. These functions include getting paper IDs, initializing extractors for extracting references, retrieving paper information from the Archive API, and more.

The object helps in organizing the code and simplifying the retrieval and processing of archive papers.

Conclusion

Getting archive papers for our AI assistant involves choosing between a dumb approach of searching directly on Archive or a more sophisticated approach of extracting references from papers. Both approaches have their advantages and can be used depending on the desired level of specificity and efficiency.

By building a graph of relevant papers, we create a structure that mimics the process of expanding knowledge through references. This graph enhances the AI assistant's ability to provide accurate and comprehensive information on various topics.

Continuing our discussion, we will explore these approaches in Detail and analyze the pros and cons of each method.