我是 R 和 stackoverflow 的新手，所以请保持温和，我会尽量保持这篇文章的正确性。 我正在开展一个项目，将全外显子组测序 (WES) 结果与蛋白质组数据进行比较。我们的 WES 设施仅以 html 文件形式提供数据，因此我需要将其读入 R 以继续我的工作。
我试图跟随 DataCamp tutorial for rvest但我认为问题可能是 html 文件太复杂了，因为我得到的是\t\t\tn\n\t 之间的一些文本。我想问题是 html_node 不正确？
这是我的 R 代码，后跟经过缩短和变体修改的 HTML。
我想要得到的是一个与 html 中具有相同列的数据框。如示例中所示，某些变体会影响多个转录本，在这些情况下，单行/转录本将是完美的，但无论如何都不是必须的。
非常感谢您的帮助!
塞巴斯蒂安
library(tidyverse)
library(rvest)
htmlALL <- read_html("Example_html")
getDATA <- function(html){
html %>%
html_nodes(".table") %>%
html_text() %>%
str_trim() %>%
unlist()
}
df_html <- getDATA(htmlALL)
<!DOCTYPE html
PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en-US" xml:lang="en-US">
<head>
<!-- add title in the brower tab bar -->
<title>Homozygous variants of sample XXX </title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<!-- change style to look nice -->
<style type="text/css">
html {
text-align: center;
vertical-align: middle;
height: 100%;
width: 100%;
}
body {
background: #eee url('http://i.imgur.com/eeQeRmk.png'); /* http://subtlepatterns.com/weave/ */
font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif;
font-size: 62.5%;
entry-height: 1;
color: #585858;
padding: 22px 10px;
padding-bottom: 55px;
}
::selection { background: #5f74a0; color: #fff; }
::-moz-selection { background: #5f74a0; color: #fff; }
::-webkit-selection { background: #5f74a0; color: #fff; }
br { display: block; entry-height: 1.6em; }
input, textarea {
-webkit-font-smoothing: antialiased;
-webkit-text-size-adjust: 100%;
-ms-text-size-adjust: 100%;
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
outentry: none;
}
blockquote, q { quotes: none; }
blockquote:before, blockquote:after, q:before, q:after { content: ''; content: none; }
strong, b { font-weight: bold; }
h1 {
font-weight: bold;
font-size: 3.6em;
entry-height: 1.7em;
margin-bottom: 10px;
text-align: center;
}
h2 {
font-weight: bold;
font-size: 2.6em;
entry-height: 1.7em;
margin-bottom: 10px;
text-align: center;
}
/** big white sheet everything is on **/
.wrapper {
display: block;
width: 95%;
background: #fff;
margin: 0 auto;
padding: 10px 17px 100px;
box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-webkit-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-moz-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-ms-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-o-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
overflow-x: auto;
overflow-y: visible;
}
/* smaller box the family information is on */
.info{
display: block;
width: 800px;
background: #f2f2f2;
margin: 0 auto;
padding: 10px 17px 10px 10px;
box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-webkit-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-moz-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-ms-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-o-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
font-size: 1.8em;
margin-bottom: 10px;
}
/* this is what actually contains the info */
.table {
display: table;
margin: 0 auto;
width: 99%;
font-size: 1.2em;
margin-bottom: 15px;
border-collapse: collapse;
overflow: visible;
}
/* one row of the variants */
.tablerow {
display: table-row;
overflow: visible;
border: 1px solid gray;
width: 100%;
}
/* header are bigger and may in the future be clickable to sort accordginly*/
.tableheader {
display: table-cell;
background: #f2f2f2;
padding: 3px 10px;
margin-bottom: 25px;
font-size: 1.8em;
box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-webkit-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-moz-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-ms-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-o-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
}
/* in the following each column gets specified to increase readablity*/
.position {
display: table-cell;
padding: 3px 10px;
font-size: 1.4em;
height: 100%;
text-align: center;
vertical-align: middle;
}
.variants {
display: table-cell;
height: 100%;
vertical-align: middle;
overflow: visible;
white-space: nowrap;
}
.stacked {
display: table;
height: 50%;
width: 100%;
}
.center {
display: table-cell;
vertical-align: middle;
width: 100%;
padding: 0px 5px;
}
.consequences {
display: table-cell;
height: 100%;
vertical-align: middle;
padding: 3px 10px;
}
.gene {
display: table-cell;
padding: 3px 15px;
height: 100%;
vertical-align: middle;
font-size: 1.4em;
font-weight: bold;
}
.transcripts {
display: table-cell;
vertical-align: middle;
height: 100%;
}
.list {
height: 100%;
width: 100%;
display: table;
table-layout: fixed;
}
.row {
display: table-row;
overflow: visible;
vertical-align: middle;
}
.entry {
display: table-cell;
vertical-align:middle;
padding: 0% 1% 0% 1%;
white-space: nowrap;
text-overflow: ellipsis;
overflow: hidden;
}
.cdspos {
display: table-cell;
vertical-align: middle;
height: 100%;
}
.exon {
display: table-cell;
vertical-align: middle;
height: 100%;
}
.hgvs {
display: table-cell;
height: 100%;
vertical-align: middle;
}
.hgvs .list .row{
display: table-row;
vertical-align: middle;
}
.polyphen {
display: table-cell;
height: 100%;
vertical-align: middle;
}
.polyphen .list .row{
display: table-row;
vertical-align: middle;
}
.sift {
display: table-cell;
height: 100%;
vertical-align: middle;
}
.sift .list .row{
display: table-row;
vertical-align: middle;
}
.allelefreq {
display: table-cell;
height: 100%;
vertical-align: middle;
}
/* Tooltip container */
.tooltip_gene, .tooltip_allelefrq ,.tooltip_qual{
position: relative;
display: inline-block;
border-bottom: 1px dotted black; /* If you want dots under the hoverable text */
}
.tooltiptext{
visibility: hidden;
overflow: auto;
min-width: 400px;
background-color: #ffb380;
color: black;
text-align: left;
padding: 5px 10px;
border-radius: 6px;
font-size: 12pt;
font-weight: normal;
/* Position the tooltip text - see examples below! */
position: absolute;
z-index:1;
/* shadow */
box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-webkit-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-moz-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-ms-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
-o-box-shadow: 2px 2px 3px -1px rgba(0,0,0,0.35);
opacity: 0.95;
filter: alpha(opacity=95);
}
/* Tooltip text */
.tooltip_gene .tooltiptext {
top: -5px;
left: 105%;
}
/* Tooltip text */
.tooltip_allelefrq .tooltiptext {
top: -5px;
right: 105%;
min-width: 120px;
}
/* Show the tooltip text when you mouse over the tooltip container */
.tooltip_allelefrq:hover .tooltiptext, .tooltip_gene:hover .tooltiptext {
visibility: visible;
}
.clin {
display: table-cell;
height: 100%;
vertical-align: middle;
padding: 0% 1% 0% 1%;
white-space: nowrap;
text-overflow: ellipsis;
overflow: hidden;
}
</style>
<body>
<div class="wrapper">
<!-- add info about patients -->
<h1>Homozygous variants of sample XXX</h1>
<h2>Tue Jan 23 09:01:56 2018</h2>
<div class="info">
Patient only<br>
</div>
<!-- variants table start -->
<div class="table">
<!-- table header start -->
<div class="tablerow">
<div class="tableheader">
Position
</div>
<div class="tableheader">
Variant
</div>
<div class="tableheader">
Cons
</div>
<div class="tableheader">
Gene
</div>
<div class="tableheader">
Transcript
</div>
<div class="tableheader">
HGVSC
</div>
<div class="tableheader">
HGVSP
</div>
<div class="tableheader">
PolyPhen
</div>
<div class="tableheader">
SIFT
</div>
<div class="tableheader">
AF
</div>
<div class="tableheader">
Clin
</div>
</div>
<!-- table header stop -->
<!-- var loop start -->
<div class="tablerow" >
<!-- position start -->
<div class="position">
<a href="http://gnomad.broadinstitute.org/region/1-117635467-117635507">1:117635487</a>
</div>
<!-- position stop -->
<!-- variants start -->
<div class="variants">
G->T
</div>
<!-- variants stop -->
<!-- consequences start -->
<div class="consequences" style="background: rgb(196, 197, 198);">
synonymous
</div>
<!-- consequences stop -->
<!-- gene start -->
<div class="gene" >
<div class="tooltip_gene">
<a href="http://www.genecards.org/cgi-bin/carddisp.pl?gene=TTF2" >
TTF2
</a>
<span class="tooltiptext">GeneCards Summary<hr>
TTF2 (Transcription Termination Factor 2) is a Protein Coding gene.
Diseases associated with TTF2 include Sexual Sadism and Narcissistic Personality Disorder.
Among its related pathways are Human Thyroid Stimulating Hormone (TSH) signaling pathway and Insulin secretion.
GO annotations related to this gene include hydrolase activity and DNA-dependent ATPase activity.
An important paralog of this gene is HLTF.</span>
</div>
</div>
<!-- gene stop -->
<!-- transcripts start -->
<div class="transcripts">
<div class="list">
<div class="row">
<div class="entry">
<a href="http://grch37.ensembl.org/Homo_sapiens/Transcript/Summary?db=core;t=ENST00000369466">ENST00000369466
</a>
</div>
</div>
</div>
</div>
<!-- transcripts stop -->
<!-- exon start -->
<!-- <div class="exon">
<div class="list">
</div>
</div>-->
<!-- exon stop -->
<!-- hgvsc start -->
<div class="hgvs">
<div class="list">
<div class="row">
<div class="entry">
c.2940G>T
</div>
</div>
</div>
</div>
<!-- hgvsc stop -->
<!-- hgvsp start -->
<div class="hgvs">
<div class="list">
<div class="row">
<div class="entry">
c.2940G>T(p.%3D)
</div>
</div>
</div>
</div>
<!-- hgvsp stop -->
<!-- polyphen start -->
<div class="polyphen">
<div class="list">
<div class="row">
<div class="entry">
</div>
</div>
</div>
</div>
<!-- polyphen stop -->
<!-- sift start -->
<div class="sift">
<div class="list">
<div class="row">
<div class="entry">
</div>
</div>
</div>
</div>
<!-- sift stop -->
<!--.allelefreq start -->
<div class="allelefreq">
<div class="tooltip_allelefrq">
0.00000
<span class="tooltiptext">allele counts<hr>ht: <span style='float:right;'>0</span><br>hm: <span style='float:right;'>0</span><br>wt: <span style='float:right;'>0</span><hr>inhouse:<span style='float:right;'>0.00118</span></span>
</div>
</div>
<!--.allelefreq stop -->
<!--.allelefreq start -->
<div class="clin">
</div>
<!--.allelefreq stop -->
</div>
<!-- table row stop-->
<div class="tablerow" >
<!-- position start -->
<div class="position">
<a href="http://gnomad.broadinstitute.org/region/1-149898435-149898475">1:149898455</a>
</div>
<!-- position stop -->
<!-- variants start -->
<div class="variants">
<a href="https://www.ncbi.nlm.nih.gov/projects/SNP/snp_ref.cgi?rs=rs143105666">G->A</a>
</div>
<!-- variants stop -->
<!-- consequences start -->
<div class="consequences" style="background: rgb(196, 197, 198);">
synonymous
</div>
<!-- consequences stop -->
<!-- gene start -->
<div class="gene" >
<div class="tooltip_gene">
<a href="http://www.genecards.org/cgi-bin/carddisp.pl?gene=SF3B4" >
SF3B4
</a>
<span class="tooltiptext">GeneCards Summary<hr>
SF3B4 (Splicing Factor 3b Subunit 4) is a Protein Coding gene.
Diseases associated with SF3B4 include Acrofacial Dysostosis 1, Nager Type and Acrofacial Dysostosis Syndrome Of Rodriguez.
Among its related pathways are mRNA Splicing - Major Pathway and Gene Expression.
GO annotations related to this gene include nucleic acid binding and nucleotide binding.
</span>
</div>
</div>
<!-- gene stop -->
<!-- transcripts start -->
<div class="transcripts">
<div class="list">
<div class="row">
<div class="entry">
<a href="http://grch37.ensembl.org/Homo_sapiens/Transcript/Summary?db=core;t=ENST00000457312">ENST00000457312
</a>
</div>
</div>
<div class="row">
<div class="entry">
<a href="http://grch37.ensembl.org/Homo_sapiens/Transcript/Summary?db=core;t=ENST00000271628">ENST00000271628
</a>
</div>
</div>
</div>
</div>
<!-- transcripts stop -->
<!-- exon start -->
<!-- <div class="exon">
<div class="list">
</div>
</div>-->
<!-- exon stop -->
<!-- hgvsc start -->
<div class="hgvs">
<div class="list">
<div class="row">
<div class="entry">
c.390C>A
</div>
</div>
<div class="row">
<div class="entry">
c.519C>A
</div>
</div>
</div>
</div>
<!-- hgvsc stop -->
<!-- hgvsp start -->
<div class="hgvs">
<div class="list">
<div class="row">
<div class="entry">
c.390C>A(p.%3D)
</div>
</div>
<div class="row">
<div class="entry">
c.519C>A(p.%3D)
</div>
</div>
</div>
</div>
<!-- hgvsp stop -->
<!-- polyphen start -->
<div class="polyphen">
<div class="list">
<div class="row">
<div class="entry">
</div>
</div>
<div class="row">
<div class="entry">
</div>
</div>
</div>
</div>
<!-- polyphen stop -->
<!-- sift start -->
<div class="sift">
<div class="list">
<div class="row">
<div class="entry">
</div>
</div>
<div class="row">
<div class="entry">
</div>
</div>
</div>
</div>
<!-- sift stop -->
<!--.allelefreq start -->
<div class="allelefreq">
<div class="tooltip_allelefrq">
0.00021
<span class="tooltiptext">allele counts<hr>ht: <span style='float:right;'>57</span><br>hm: <span style='float:right;'>0</span><br>wt: <span style='float:right;'>277082</span><hr>inhouse:<span style='float:right;'>0.00236</span></span>
</div>
</div>
<!--.allelefreq stop -->
<!--.allelefreq start -->
<div class="clin">
</div>
<!--.allelefreq stop -->
</div>
<!-- table row stop-->
<!-- var loop stop -->
</div>
<!-- variant table stop -->
</div>
</body>
</html>
最佳答案
这是我能为您提供的最好的。请注意，输出包括将鼠标悬停在 Gene 列中的数据上时弹出的“工具提示文本”。
library(rvest)
# I saved your sample to my Desktop as test.html
pg = read_html('~/Desktop/test.html')
# count rows (including header):
n_rows = pg %>% html_nodes('div.tablerow') %>% length
# sprintf-friendly format to get the %d-th node matching
# //div[@class="tablerow"] (same as div.tablerow in CSS)
# All of the /div after this are columns
xp_fmt = '//div[@class="tablerow"][%d]/div'
# div.tableheader nodes contain column names
col_names = pg %>% html_nodes(xpath = sprintf(xp_fmt, 1L)) %>%
html_text %>% trimws
# rows 2:n contain the actual data; gsub is
# stripping leading/trailing whitespace and
# any duplicate internal whitespace
rows = lapply(2:n_rows, function(ii) {
pg %>% html_nodes(xpath = sprintf(xp_fmt, ii)) %>%
html_text %>% gsub('^\\s+|\\s{2,}|\\s+$', '', .)
})
# can't forget those pesky factors
DF = as.data.frame(do.call(rbind, rows), stringsAsFactors = FALSE)
names(DF) = col_names
DF
# Position Variant Cons
# 1 1:117635487 G->T synonymous
# 2 1:149898455 G->A synonymous
# Gene
# 1 TTF2GeneCards Summary\nTTF2 (Transcription Termination Factor 2) is a Protein Coding gene.\nDiseases associated with TTF2 include Sexual Sadism and Narcissistic Personality Disorder.\nAmong its related pathways are Human Thyroid Stimulating Hormone (TSH) signaling pathway and Insulin secretion.\nGO annotations related to this gene include hydrolase activity and DNA-dependent ATPase activity.\nAn important paralog of this gene is HLTF.
# 2 SF3B4GeneCards Summary\nSF3B4 (Splicing Factor 3b Subunit 4) is a Protein Coding gene.\nDiseases associated with SF3B4 include Acrofacial Dysostosis 1, Nager Type and Acrofacial Dysostosis Syndrome Of Rodriguez.\nAmong its related pathways are mRNA Splicing - Major Pathway and Gene Expression.\nGO annotations related to this gene include nucleic acid binding and nucleotide binding.
# Transcript HGVSC
# 1 ENST00000369466 c.2940G>T
# 2 ENST00000457312ENST00000271628 c.390C>Ac.519C>A
# HGVSP PolyPhen SIFT
# 1 c.2940G>T(p.%3D)
# 2 c.390C>A(p.%3D)c.519C>A(p.%3D)
# AF
# 1 0.00000allele countsht: 0hm: 0wt: 0inhouse:0.00118
# 2 0.00021allele countsht: 57hm: 0wt: 277082inhouse:0.00236
# Clin
# 1
# 2
请注意，它不适用于此处，因为您的所有列似乎都是 character 类型，但更复杂的方法会将此处的行转换为常规文件(例如 csv )，然后使用 read.table(或者更好，fread)读入文本并自动检测列类型。
关于html - 使用 rvest 将复杂的 html 文件读入 R，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/52297953/
我正在学习如何使用Nokogiri，根据这段代码我遇到了一些问题:require'rubygems'require'mechanize'post_agent=WWW::Mechanize.newpost_page=post_agent.get('http://www.vbulletin.org/forum/showthread.php?t=230708')puts"\nabsolutepathwithtbodygivesnil"putspost_page.parser.xpath('/html/body/div/div/div/div/div/table/tbody/tr/td/div
我有一个Ruby程序，它使用rubyzip压缩XML文件的目录树。gem。我的问题是文件开始变得很重，我想提高压缩级别，因为压缩时间不是问题。我在rubyzipdocumentation中找不到一种为创建的ZIP文件指定压缩级别的方法。有人知道如何更改此设置吗？是否有另一个允许指定压缩级别的Ruby库？ 最佳答案 这是我通过查看rubyzip内部创建的代码。level=Zlib::BEST_COMPRESSIONZip::ZipOutputStream.open(zip_file)do|zip|Dir.glob("**/*")d
类classAprivatedeffooputs:fooendpublicdefbarputs:barendprivatedefzimputs:zimendprotecteddefdibputs:dibendendA的实例a=A.new测试a.foorescueputs:faila.barrescueputs:faila.zimrescueputs:faila.dibrescueputs:faila.gazrescueputs:fail测试输出failbarfailfailfail.发送测试[:foo,:bar,:zim,:dib,:gaz].each{|m|a.send(m)resc
很好奇，就使用rubyonrails自动化单元测试而言，你们正在做什么？您是否创建了一个脚本来在cron中运行rake作业并将结果邮寄给您？git中的预提交Hook？只是手动调用？我完全理解测试，但想知道在错误发生之前捕获错误的最佳实践是什么。让我们理所当然地认为测试本身是完美无缺的，并且可以正常工作。下一步是什么以确保他们在正确的时间将可能有害的结果传达给您？ 最佳答案 不确定您到底想听什么，但是有几个级别的自动代码库控制:在处理某项功能时，您可以使用类似autotest的内容获得关于哪些有效，哪些无效的即时反馈。要确保您的提
假设我做了一个模块如下:m=Module.newdoclassCendend三个问题:除了对m的引用之外，还有什么方法可以访问C和m中的其他内容？我可以在创建匿名模块后为其命名吗(就像我输入“module...”一样)？如何在使用完匿名模块后将其删除，使其定义的常量不再存在？ 最佳答案 三个答案:是的，使用ObjectSpace.此代码使c引用你的类(class)C不引用m:c=nilObjectSpace.each_object{|obj|c=objif(Class===objandobj.name=~/::C$/)}当然这取决于
我试图在一个项目中使用rake，如果我把所有东西都放到Rakefile中，它会很大并且很难读取/找到东西，所以我试着将每个命名空间放在lib/rake中它自己的文件中，我添加了这个到我的rake文件的顶部:Dir['#{File.dirname(__FILE__)}/lib/rake/*.rake'].map{|f|requiref}它加载文件没问题，但没有任务。我现在只有一个.rake文件作为测试，名为“servers.rake”，它看起来像这样:namespace:serverdotask:testdoputs"test"endend所以当我运行rakeserver:testid时
我的目标是转换表单输入，例如“100兆字节”或“1GB”，并将其转换为我可以存储在数据库中的文件大小(以千字节为单位)。目前，我有这个:defquota_convert@regex=/([0-9]+)(.*)s/@sizes=%w{kilobytemegabytegigabyte}m=self.quota.match(@regex)if@sizes.include?m[2]eval("self.quota=#{m[1]}.#{m[2]}")endend这有效，但前提是输入是倍数(“gigabytes”，而不是“gigabyte”)并且由于使用了eval看起来疯狂不安全。所以，功能正常，
我正在尝试使用ruby和Savon来使用网络服务。测试服务为http://www.webservicex.net/WS/WSDetails.aspx?WSID=9&CATID=2require'rubygems'require'savon'client=Savon::Client.new"http://www.webservicex.net/stockquote.asmx?WSDL"client.get_quotedo|soap|soap.body={:symbol=>"AAPL"}end返回SOAP异常。检查soap信封，在我看来soap请求没有正确的命名空间。任何人都可以建议我
关闭。这个问题是opinion-based.它目前不接受答案。想要改进这个问题？更新问题，以便editingthispost可以用事实和引用来回答它.关闭4年前。Improvethisquestion我想在固定时间创建一系列低音和高音调的哔哔声。例如:在150毫秒时发出高音调的蜂鸣声在151毫秒时发出低音调的蜂鸣声200毫秒时发出低音调的蜂鸣声250毫秒的高音调蜂鸣声有没有办法在Ruby或Python中做到这一点？我真的不在乎输出编码是什么(.wav、.mp3、.ogg等等)，但我确实想创建一个输出文件。
Rails2.3可以选择随时使用RouteSet#add_configuration_file添加更多路由。是否可以在Rails3项目中做同样的事情？ 最佳答案 在config/application.rb中:config.paths.config.routes在Rails3.2(也可能是Rails3.1)中，使用:config.paths["config/routes"] 关于ruby-on-rails-Rails3中的多个路由文件，我们在StackOverflow上找到一个类似的问题