以下内容如有错误，恳请指出。
最近在学习一些sklearn的特征降维与特征提取的事情，可以参考前面几篇发的博客。在之前也提及到，对于一个文本，可以将其编码为词频向量，或在是权重向量，也可以将其编码为哈希向量。对于最后的哈希向量可以理解为一个降维的特征提取过程。
首先说一下，我暂时完全没有接触过nlp的相关任务，也还没有使用过rnn，lstm等时序网络。但是现在，已经把一个文本也可以称得上为一个样本编码为一个向量，配合我们之前学到的各种机器学习分类器与深度学习，那么不就可以对这个特征向量进行分类训练，从而实现文本分类的任务了吗？
所以，不会rnn或者lstm的搭建也没有关系，现在可以通过其他的方法将文本特征提取出来，那么就已经足够了。我就可以拿这个特征来做下一步的事情。这就像视频理解领域的C3D，将视频编码为一个4096的特征向量，那么就拿这个特征向量来进行其他的下游任务。同理，这里也是一样的，将其编码为哈希向量之后，我就可以使用分类器对其进行一个文本分类的NLP任务。
理论分析结束，接下来就是实战验证。对于哈希编码的特征向量，我使用了svm、随机森林与搭建了个神经网络分别测试了分类的效果。此外还用svm测试了权重编码的效果，同时查看了一下那些特征会影响分类结果。最后，贴上了一个官方的代码以供学习。
@[toc] 这里我使用的是sklearn自带的一个数据集，属于一个新闻分类数据，有20类，18846个样本，其中11314个训练样本。其类别信息分别为：
['alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', 'comp.sys.ibm.pc.hardware', 'comp.sys.mac.hardware', 'comp.windows.x', 'misc.forsale', 'rec.autos', 'rec.motorcycles', 'rec.sport.baseball', 'rec.sport.hockey', 'sci.crypt', 'sci.electronics', 'sci.med', 'sci.space', 'soc.religion.christian', 'talk.politics.guns', 'talk.politics.mideast', 'talk.politics.misc', 'talk.religion.misc']
这里会选取其中的4个类别进行文本分类处理："alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space"
1. 文本向量提取
1.1 数据集导入
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.targetps：这里通过remove删除文件的标题、签名快与应用块，使得数据更加的真实。这是因为，如果不删除分类器会过分拟合许多的内容：
几乎每个组都通过诸如 NNTP-Posting-Host:和之类的标题是否Distribution:经常出现来区分。
另一个重要特征涉及发件人是否隶属于大学，如其标题或签名所示。
“文章”这个词是一个重要的特征，基于人们引用以前的帖子的频率是这样的：“在文章 [文章 ID]，[名称] <[电子邮件地址]> 写道：”
其他功能与当时发帖的特定人员的姓名和电子邮件地址相匹配。
有了如此丰富的区分新闻组的线索，分类器几乎不需要从文本中识别主题，而且它们都在相同的高水平上执行。所以，这里删除了('headers', 'footers', 'quotes')的信息。
1.2 哈希编码
from sklearn.feature_extraction.text import HashingVectorizerfrom sklearn.feature_extraction.text import TfidfVectorizervectorizer = HashingVectorizer(stop_words="english", alternate_sign=False, n_features=2 ** 10)X_train = vectorizer.fit_transform(data_train.data)X_test = vectorizer.transform(data_test.data)print("X_train.shape:{}, X_test.shape:{}".format(X_train.shape, X_test.shape))# 输出：# X_train.shape:(2034, 1024), X_test.shape:(1353, 1024)1.3 卡方过滤
from sklearn.feature_selection import SelectKBestfrom sklearn.feature_selection import chi2select_kbest = SelectKBest(chi2, k=200)X_train = select_kbest.fit_transform(X_train, y_train)X_test = select_kbest.transform(X_test)print("X_train.shape:{}, X_test.shape:{}".format(X_train.shape, X_test.shape))# 输出：# X_train.shape:(2034, 200), X_test.shape:(1353, 200)哈希编码内容与卡方过滤内容详细可以见我之前的两篇笔记，链接如下：
1. sklearn特征提取方法汇总（包含字典、文本、图像的特征提取）
2. klearn特征降维方法汇总（方差过滤，卡方，F过滤，互信息，嵌入法）
2. 机器学习模型训练
2.1 SVM模型测试
from time import timefrom sklearn.svm import LinearSVCfrom sklearn.metrics import accuracy_scoret0 = time()clf = LinearSVC(dual=False, tol=1e-3)clf.fit(X_train, y_train)train_time = time() - t0print("train time: %0.3fs" % train_time)t0 = time() pred = clf.predict(X_test)test_time = time() - t0print("test time: %0.3fs" % test_time)test_sroce = accuracy_score(y_test, pred)train_sroce = accuracy_score(y_train, clf.predict(X_train))print("test accuracy: %0.3f" % test_sroce, "train accuracy: %0.3f" % train_sroce)输出：
train time: 0.020stest time: 0.001stest accuracy: 0.662 train accuracy: 0.775
2.2 随机森林模型测试
from time import timefrom sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import accuracy_scoret0 = time()clf = RandomForestClassifier(verbose=1, random_state=42)clf.fit(X_train, y_train)train_time = time() - t0print("train time: %0.3fs" % train_time)t0 = time() pred = clf.predict(X_test)test_time = time() - t0print("test time: %0.3fs" % test_time)test_sroce = accuracy_score(y_test, pred)train_sroce = accuracy_score(y_train, clf.predict(X_train))print("test accuracy: %0.3f" % test_sroce, "train accuracy: %0.3f" % train_sroce)输出：
train time: 0.466stest time: 0.032stest accuracy: 0.617 train accuracy: 0.964
3. 搭建神经网络测试
其实，文本分类只要是将文本信息编码为特征，但是这一步已经在哈希编码与卡方过滤后提取了出来。最后每个文本信息被编码为了一个200维的特征信息，那么现在就可以利用这个特征信息构建一个神经网络来进行训练。
所以下面会简单的搭建一个多层感知机来进行训练，这里我就搭建了一个3层的全连接层，没有特别的设计（其实我也不会特别设计，哭...）
3.1 神经网络搭建
import torch.nn as nnclass MLP(nn.Module): def __init__(self, embedding=200, n_class=4): super().__init__() self.model = nn.Sequential( nn.Linear(embedding, 256), nn.Dropout(p=0.6), nn.ReLU(), nn.Linear(256, 512), nn.Dropout(p=0.6), nn.ReLU(), nn.Linear(512, n_class), ) def forward(self, x): return self.model(x)
3.2 神经网络训练
数据格式转换
在训练之前，需要进行数据格式的转换。由于卡方过滤出来的矩阵是一个稀疏矩阵，也就是一个稀疏的格式矩阵，所以需要进行.toarray()转换为numpy的array格式。eg：X_train = X_train..toarray()
之后具体是转什么样的格式，就执行在出错的位置看error的提醒就可以了，或者直接按照我这里的数据格式转换，基本是正确的。
import torchX_train = torch.tensor(X_train.toarray()).float()X_test = torch.tensor(X_test.toarray()).float()y_train = torch.tensor(y_train).long()y_test = torch.tensor(y_test).long()
神经网络训练
数据格式转换后，就可以使用神经网络来进行训练，参考代码如下：
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.target0输出：
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.target1最后，用神经网络来训练的结果好像与svm得到的结果差不多,无论是集成算法还是支持向量机算法，还是神经网络，最后的结果都在66%左右。
4. 权重向量编码
对于文本分类来说，如果想要了解对于一个文本的分类哪个单词特征是最重要的，可以通过权重向量编码配合一些设计权重分配的分类器来查看。（比如SVM等）
4.1 获取列表中最大的N个数索引
这里用一节的内容来介绍一个trick，也就是一个库的使用方法，参考链接见参考资料3.
针对数组无重复数的
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.target2针对有重复数的
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.target34.2 查看息量最大的N个特征名字
np.argsort实现
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.target4输出：
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.target5代码来源，见参考资料4.
heapq实现
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.target6输出：
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.target7可以看见，这两种的方法的输出结果是一直的，只是第二种方法还会按权重的大小从大到小排列。而对于输出结果来说，大概可以知道权重越大的单词和文本类别越相关。
4.3 使用权重编码进行文本分类
参考代码如下：
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.target8输出：
from sklearn.datasets import fetch_20newsgroupscategories = [ "alt.atheism", "talk.religion.misc", "comp.graphics", "sci.space",]remove = ("headers", "footers", "quotes")data_train = fetch_20newsgroups( data_home='./dataset/', subset='train', categories=categories, remove=remove, random_state=42)data_test = fetch_20newsgroups( data_home='./dataset/', subset='test', categories=categories, remove=remove, random_state=42)data_train.target_names, data_test.target_names# 输出：# (['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'],# ['alt.atheism', 'comp.graphics', 'sci.space', 'talk.religion.misc'])y_train, y_test = data_train.target, data_test.target9分析：可以看见，使用权重编码相比哈希编码的效果要更好，这是因为哈希编码其实是一种降维的方法，其减少了训练的时长，但是同时也会丢失部分的信息。所以哈希编码的特征分类效果要差一点，但是训练的时间要短一点。
5. 使用稀疏特征对文本文档进行分类
这里贴上一个官方的提供代码，供自己学习，具体内容见参考资料5. 相关的内容这里翻译为中文。
这是一个示例，展示了如何使用 scikit-learn 使用词袋方法按主题对文档进行分类。此示例使用 scipy.sparse 矩阵来存储特征，并演示了可以有效处理稀疏矩阵的各种分类器。
此示例中使用的数据集是 20 个新闻组数据集。它将被自动下载，然后缓存。
5.1 参数设置
from sklearn.feature_extraction.text import HashingVectorizerfrom sklearn.feature_extraction.text import TfidfVectorizervectorizer = HashingVectorizer(stop_words="english", alternate_sign=False, n_features=2 ** 10)X_train = vectorizer.fit_transform(data_train.data)X_test = vectorizer.transform(data_test.data)print("X_train.shape:{}, X_test.shape:{}".format(X_train.shape, X_test.shape))# 输出：# X_train.shape:(2034, 1024), X_test.shape:(1353, 1024)0Out：
from sklearn.feature_extraction.text import HashingVectorizerfrom sklearn.feature_extraction.text import TfidfVectorizervectorizer = HashingVectorizer(stop_words="english", alternate_sign=False, n_features=2 ** 10)X_train = vectorizer.fit_transform(data_train.data)X_test = vectorizer.transform(data_test.data)print("X_train.shape:{}, X_test.shape:{}".format(X_train.shape, X_test.shape))# 输出：# X_train.shape:(2034, 1024), X_test.shape:(1353, 1024)15.2 从训练集中加载数据
让我们从新闻组数据集中加载数据，该数据集包含 20 个主题的大约 18000 个新闻组帖子，分为两个子集：一个用于训练（或开发），另一个用于测试（或性能评估）。
from sklearn.feature_extraction.text import HashingVectorizerfrom sklearn.feature_extraction.text import TfidfVectorizervectorizer = HashingVectorizer(stop_words="english", alternate_sign=False, n_features=2 ** 10)X_train = vectorizer.fit_transform(data_train.data)X_test = vectorizer.transform(data_test.data)print("X_train.shape:{}, X_test.shape:{}".format(X_train.shape, X_test.shape))# 输出：# X_train.shape:(2034, 1024), X_test.shape:(1353, 1024)2Out:
from sklearn.feature_extraction.text import HashingVectorizerfrom sklearn.feature_extraction.text import TfidfVectorizervectorizer = HashingVectorizer(stop_words="english", alternate_sign=False, n_features=2 ** 10)X_train = vectorizer.fit_transform(data_train.data)X_test = vectorizer.transform(data_test.data)print("X_train.shape:{}, X_test.shape:{}".format(X_train.shape, X_test.shape))# 输出：# X_train.shape:(2034, 1024), X_test.shape:(1353, 1024)35.3 分类构建
用 15 种不同的分类模型训练和测试数据集，并获得每个模型的性能结果。
from sklearn.feature_extraction.text import HashingVectorizerfrom sklearn.feature_extraction.text import TfidfVectorizervectorizer = HashingVectorizer(stop_words="english", alternate_sign=False, n_features=2 ** 10)X_train = vectorizer.fit_transform(data_train.data)X_test = vectorizer.transform(data_test.data)print("X_train.shape:{}, X_test.shape:{}".format(X_train.shape, X_test.shape))# 输出：# X_train.shape:(2034, 1024), X_test.shape:(1353, 1024)4Out：
from sklearn.feature_extraction.text import HashingVectorizerfrom sklearn.feature_extraction.text import TfidfVectorizervectorizer = HashingVectorizer(stop_words="english", alternate_sign=False, n_features=2 ** 10)X_train = vectorizer.fit_transform(data_train.data)X_test = vectorizer.transform(data_test.data)print("X_train.shape:{}, X_test.shape:{}".format(X_train.shape, X_test.shape))# 输出：# X_train.shape:(2034, 1024), X_test.shape:(1353, 1024)55.4 可视化处理
条形图表示每个分类器的准确度、训练时间（标准化）和测试时间（标准化）。
from sklearn.feature_extraction.text import HashingVectorizerfrom sklearn.feature_extraction.text import TfidfVectorizervectorizer = HashingVectorizer(stop_words="english", alternate_sign=False, n_features=2 ** 10)X_train = vectorizer.fit_transform(data_train.data)X_test = vectorizer.transform(data_test.data)print("X_train.shape:{}, X_test.shape:{}".format(X_train.shape, X_test.shape))# 输出：# X_train.shape:(2034, 1024), X_test.shape:(1353, 1024)6Out：
参考资料：
1. sklearn特征提取方法汇总（包含字典、文本、图像的特征提取）
2. klearn特征降维方法汇总（方差过滤，卡方，F过滤，互信息，嵌入法）
3. python获取列表中最大的N个数索引
4. The 20 newsgroups text dataset
5. Classification of text documents using sparse features
原文：https://juejin.cn/post/7095005618074812424