我正在针对二进制预测问题运行一些监督实验。我使用 10 折交叉验证来评估平均精度的性能(每折的平均精度除以交叉验证的折数 - 在我的例子中为 10)。我想绘制这 10 次折叠的平均精度结果的 PR 曲线，但我不确定执行此操作的最佳方法。

A previous question在 Cross Validated Stack Exchange 网站上提出了同样的问题。一条建议通过 this example 解决的评论从 Scikit-Learn 站点绘制跨交叉验证折叠的 ROC 曲线，并将其调整为平均精度。这是我为尝试这个想法而修改的相关代码部分:

from scipy import interp
# Other packages/functions are imported, but not crucial to the question
max_ent = LogisticRegression()

mean_precision = 0.0
mean_recall = np.linspace(0,1,100)
mean_average_precision = []

for i in set(folds):
    y_scores = max_ent.fit(X_train, y_train).decision_function(X_test)
    precision, recall, _ = precision_recall_curve(y_test, y_scores)
    average_precision = average_precision_score(y_test, y_scores)
    mean_average_precision.append(average_precision)
    mean_precision += interp(mean_recall, recall, precision)

# After this line of code, inspecting the mean_precision array shows that 
# the majority of the elements equal 1. This is the part that is confusing me
# and is contributing to the incorrect plot.
mean_precision /= len(set(folds))
# This is what the actual MAP score should be
mean_average_precision = sum(mean_average_precision) / len(mean_average_precision)

# Code for plotting the mean average precision curve across folds
plt.plot(mean_recall, mean_precision)
plt.title('Mean AP Over 10 folds (area=%0.2f)' % (mean_average_precision))
plt.show()

代码运行，但在我的例子中，平均精度曲线不正确。出于某种原因，我指定用于存储 mean_precision 分数的数组(ROC 示例中的 mean_tpr 变量)计算第一个元素接近零，所有其他元素除以折叠数后为1。下面是根据 mean_recall 分数绘制的 mean_precision 分数的可视化。如您所见，绘图跳转到 1，这是不准确的。 所以我的预感是在每次交叉验证时更新 mean_precision (mean_precision += interp(mean_recall, recall, precision) ) 会出错，但是目前还不清楚如何解决这个问题。任何指导或帮助将不胜感激。

最佳答案

我遇到了同样的问题。这是我的解决方案:在循环之后，我计算所有折叠结果的 precision_recall_curve，而不是对折叠进行平均。根据 https://stats.stackexchange.com/questions/34611/meanscores-vs-scoreconcatenation-in-cross-validation 中的讨论这是一种通常更可取的方法。

import matplotlib.pyplot as plt
import numpy
from sklearn.datasets import make_blobs
from sklearn.metrics import precision_recall_curve, auc
from sklearn.model_selection import KFold
from sklearn.svm import SVC

FOLDS = 5

X, y = make_blobs(n_samples=1000, n_features=2, centers=2, cluster_std=10.0,
    random_state=12345)

f, axes = plt.subplots(1, 2, figsize=(10, 5))

axes[0].scatter(X[y==0,0], X[y==0,1], color='blue', s=2, label='y=0')
axes[0].scatter(X[y!=0,0], X[y!=0,1], color='red', s=2, label='y=1')
axes[0].set_xlabel('X[:,0]')
axes[0].set_ylabel('X[:,1]')
axes[0].legend(loc='lower left', fontsize='small')

k_fold = KFold(n_splits=FOLDS, shuffle=True, random_state=12345)
predictor = SVC(kernel='linear', C=1.0, probability=True, random_state=12345)

y_real = []
y_proba = []
for i, (train_index, test_index) in enumerate(k_fold.split(X)):
    Xtrain, Xtest = X[train_index], X[test_index]
    ytrain, ytest = y[train_index], y[test_index]
    predictor.fit(Xtrain, ytrain)
    pred_proba = predictor.predict_proba(Xtest)
    precision, recall, _ = precision_recall_curve(ytest, pred_proba[:,1])
    lab = 'Fold %d AUC=%.4f' % (i+1, auc(recall, precision))
    axes[1].step(recall, precision, label=lab)
    y_real.append(ytest)
    y_proba.append(pred_proba[:,1])

y_real = numpy.concatenate(y_real)
y_proba = numpy.concatenate(y_proba)
precision, recall, _ = precision_recall_curve(y_real, y_proba)
lab = 'Overall AUC=%.4f' % (auc(recall, precision))
axes[1].step(recall, precision, label=lab, lw=2, color='black')
axes[1].set_xlabel('Recall')
axes[1].set_ylabel('Precision')
axes[1].legend(loc='lower left', fontsize='small')

f.tight_layout()
f.savefig('result.png')

关于python - 如何在 Scikit-Learn 中绘制超过 10 倍交叉验证的 PR 曲线，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/29656550/