我正在寻找一种编程方式来使用 R 抓取 archive.gov 上数据文件系列的所有可用文件。archives.gov 似乎使用 javascript。我的目标是捕获每个可用文件的 URL 以及文件名。

家庭抵押贷款披露法案数据文件系列有 153 entries

在浏览器中，我可以单击“导出”按钮并获得具有以下结构的 csv 文件:

first_exported_record <-    
    structure(list(resultType = structure(1L, .Label = "fileUnit", class = "factor"), 
    creators.0 = structure(1L, .Label = "Federal Reserve System. Board of Governors. Division of Consumer and Community Affairs. ca. 1981- (Most Recent)", class = "factor"), 
    date = structure(1L, .Label = "1981 - 2013", class = "factor"), 
    documentIndex = 1L, from.0 = structure(1L, .Label = "Series: Home Mortgage Disclosure Data Files, 1981 - 2013", class = "factor"), 
    from.1 = structure(1L, .Label = "Record Group 82: Records of the Federal Reserve System, 1913 - 2003", class = "factor"), 
    location.locationFacility1.0 = structure(1L, .Label = "National Archives at College Park - Electronic Records(RDE)", class = "factor"), 
    location.locationFacility1.1 = structure(1L, .Label = "National Archives at College Park", class = "factor"), 
    location.locationFacility1.2 = structure(1L, .Label = "8601 Adelphi Road", class = "factor"), 
    location.locationFacility1.3 = structure(1L, .Label = "College Park, MD, 20740-6001", class = "factor"), 
    location.locationFacility1.4 = structure(1L, .Label = "Phone: 301-837-0470", class = "factor"), 
    location.locationFacility1.5 = structure(1L, .Label = "Fax: 301-837-3681", class = "factor"), 
    location.locationFacility1.6 = structure(1L, .Label = "Email: cer@nara.gov", class = "factor"), 
    naId = 18491490L, title = structure(1L, .Label = "Non-restricted Ultimate Loan Application Register (LAR) Data, 2012", class = "factor"), 
    url = structure(1L, .Label = "https://catalog.archives.gov/id/18491490", class = "factor")), .Names = c("resultType", 
    "creators.0", "date", "documentIndex", "from.0", "from.1", "location.locationFacility1.0", 
    "location.locationFacility1.1", "location.locationFacility1.2", 
    "location.locationFacility1.3", "location.locationFacility1.4", 
    "location.locationFacility1.5", "location.locationFacility1.6", 
    "naId", "title", "url"), class = "data.frame", row.names = c(NA, 
    -1L))

然后在这 153 个条目的每一个后面，都有文件单元页面，其中包含多个可供下载的文件。例如，第一个导出的记录指向:

https://catalog.archives.gov/id/18491490

但是这两个页面似乎都是 javascript，所以我不确定我是否需要 phantomjs 或 selenium 之类的东西，或者是否有一些技巧可以使用更简单的工具(如 rvest)导出目录？

当我知道每个文件的 url 时，我可以毫无问题地下载它们:

tf <- tempfile()
download.file( "https://catalog.archives.gov/catalogmedia/lz/electronic-records/rg-082/hmda/233_32LU_TSS.pdf?download=false" , tf , mode = 'wb' )

这个文件名是

"Technical Specifications Summary, 2012 Ultimate LAR."

谢谢!

更新:

具体问题是我如何以编程方式从第一个链接(系列 ID)获取该系列中所有可供下载的文件的标题和 URL。我尝试了 rvest 和 httr 命令，但没有显示任何有用的东西..:/谢谢

最佳答案

这里不需要加载和解析页面，因为记录是通过简单的 Ajax 请求加载的。

要查看请求，只需通过 devtools 监视它们并选择第一个返回一些 JSON 的请求。然后使用jsonlite库请求与R相同的URL，会自动解析结果。

列出 153 个条目的所有文件(描述 + URL):

library(jsonlite)
options(timeout=60000) # increase timeout to 60sec (default is 10sec)

json = fromJSON("https://catalog.archives.gov/OpaAPI/iapi/v1?action=search&f.level=fileUnit&f.parentNaId=2456161&q=*:*&offset=0&rows=10000&tabType=all")
ids = json$opaResponse$results$result$naId

for (id in ids) { # each id
    json = fromJSON(sprintf("https://catalog.archives.gov/OpaAPI/iapi/v1/id/%s", id))
    records = json$opaResponse$content$objects$objects$object

    for (r in 1:nrow(records)) {  # each record
        # prints the file description and URL
        print(records[r, 'description'])
        print(records[r, '@renditionBaseUrl'])
    }
}

关于javascript - 如何使用 R 从国家文件馆 (archives.gov) 中抓取目录系列中的所有文件，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/48463895/