首页 > 代码库 > python生成职业要求词云

python生成职业要求词云

接着上篇的说的,爬取了大数据相关的职位信息,http://www.17bigdata.com/jobs/。

# -*- coding: utf-8 -*-
"""
Created on Thu Aug 10 07:57:56 2017

@author: lenovo
"""

from wordcloud import WordCloud
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import jieba

def cloud(root,name,stopwords):
    filepath = root +\\ + name
    f = open(filepath,r,encoding=utf-8)
    txt = f.read()
    f.close()
    cut = jieba.cut(txt)
    words = []
    for i in cut:
        words.append(i)
    df = pd.DataFrame({words:words})
    s= df.groupby(df[words])[words].agg([(size,np.size)]).sort_values(by=size,ascending=False)
    s = s[~s.index.isin(stopwords[stopword])].to_dict()
    wordcloud = WordCloud(font_path =rE:\Python\machine learning\simhei.ttf,background_color=black)
    wordcloud.fit_words(s[size])
    plt.imshow(wordcloud)
    pngfile = root +\\ + name.split(.)[0] + .png
    wordcloud.to_file(pngfile)
    
import os 
jieba.load_userdict(rE:\Python\machine learning\NLPstopwords.txt)
stopwords = pd.read_csv(rE:\Python\machine learning\StopwordsCN.txt,encoding=utf-8,index_col=False)
for root,dirs,file in os.walk(rE:\职位信息):
    for name in file:
        if name.split(.)[-1]==txt:
            print(name)
            cloud(root,name,stopwords)        

词云如图所示:

技术分享

可以看出有些噪声词没能被去除,比如相关、以上学历等无效词汇。本想通过DF判断停用词,但是我爬的时候没顾及到这个问题,外加本身记录数也不高,就没再找职位信息的停用词。当然也可看出算法和经验是很重要的。加油

python生成职业要求词云