首页 > 代码库 > 【学习笔记】Scikit-learn文本聚类实例
【学习笔记】Scikit-learn文本聚类实例
# -*- coding=utf-8 -*- """ text category """ from sklearn.datasets import fetch_20newsgroups from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.naive_bayes import MultinomialNB categories = [‘alt.atheism‘, ‘soc.religion.christian‘, ‘comp.graphics‘, ‘sci.med‘] twenty_train = fetch_20newsgroups(subset=‘train‘, categories=categories, shuffle=True, random_state=42) print len(twenty_train.data) len(twenty_train.filenames) count_vect = CountVectorizer() X_train_counts = count_vect.fit_transform(twenty_train.data) print X_train_counts.shape print count_vect.vocabulary_.get(‘algorithm‘) tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts) X_train_tf = tf_transformer.transform(X_train_counts) print X_train_tf.shape tfidf_transformer = TfidfTransformer() X_train_tfidf = tf_transformer.fit_transform(X_train_counts) print X_train_tfidf.shape clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target) docs_new = [‘God is love‘, ‘OpenGl on the Gpu is fast‘] X_new_counts = count_vect.transform(docs_new) X_new_tfidf = tfidf_transformer.fit_transform(X_new_counts) predicted = clf.predict(X_new_tfidf) for doc, category in zip(docs_new, predicted): print ‘%r=>%s‘ % (doc, twenty_train.target_names[category]
对fetch_20newsgroups中的2257条文档进行分类
统计每个词出现的次数
用tf-idf统计词频,tf是在一个文档里每个单词出现的次数除以文档的单词总数,idf是总的文档数除以包含该单词的文档数,再取对数;tf * idf就是这里用到的值,值越大表明单词越重要,或越相关。
例子具体做法:
先计算了每个单词出现的次数
然后计算了tf-idf值
然后带入模型进行训练
最后预测了两个新文档的类型
结果:
‘God is love‘=> soc.religion.christian ‘OpenGL on the GPU is fast‘=> comp.graphics
【学习笔记】Scikit-learn文本聚类实例
声明:以上内容来自用户投稿及互联网公开渠道收集整理发布,本网站不拥有所有权,未作人工编辑处理,也不承担相关法律责任,若内容有误或涉及侵权可进行投诉: 投诉/举报 工作人员会在5个工作日内联系你,一经查实,本站将立刻删除涉嫌侵权内容。