首页 > 代码库 > 自然语言交流系统 phxnet团队 创新实训 个人博客 （十四）
自然语言交流系统 phxnet团队 创新实训 个人博客 （十四）
关于WikiExtractor的学习笔记：
WikiExtractor是一个Python 脚本，专门用于提取和清洗Wikipedia的dump数据，支持Python 2.7 或者 Python 3.3+，无额外依赖，安装和使用都非常方便：
安装：
git clone https://github.com/attardi/wikiextractor.git
cd wikiextractor/
sudo python setup.py install
cd wikiextractor/
sudo python setup.py install
使用：
WikiExtractor.py -o enwiki enwiki-latest-pages-articles.xml.bz2
...... INFO: 53665431 Pampapaul INFO: 53665433 Charles Frederick Zimpel INFO: Finished 11-process extraction of 5375019 articles in 8363.5s (642.7 art/s) |
这个过程总计花了2个多小时，提取了大概537万多篇文章。关于我的机器配置，可参考：《深度学习主机攒机小记》
提取后的文件按一定顺序切分存储在多个子目录下：
每个子目录下的又存放若干个以wiki_num命名的文件，每个大小在1M左右，这个大小可以通过参数 -b 控制:
-b n[KMG], --bytes n[KMG] maximum bytes per output file (default 1M)
我们看一下wiki_00里的具体内容：
<doc id="12" url="https://en.wikipedia.org/wiki?curid=12" title="Anarchism">
Anarchism
Anarchism is a political philosophy that advocates self-governed societies based on voluntary institutions. These are often described as stateless societies, although several authors have defined them more specifically as institutions based on non-hierarchical free associations. Anarchism holds the state to be undesirable, unnecessary, and harmful.
...
Criticisms of anarchism include moral criticisms and pragmatic criticisms. Anarchism is often evaluated as unfeasible or utopian by its critics.
</doc>
<doc id="25" url="https://en.wikipedia.org/wiki?curid=25" title="Autism">
Autism
Autism is a neurodevelopmental disorder characterized by impaired social interaction, verbal and non-verbal communication, and restricted and repetitive behavior. Parents usually notice signs in the first two years of their child‘s life. These signs often develop gradually, though some children with autism reach their developmental milestones at a normal pace and then regress. The diagnostic criteria require that symptoms become apparent in early childhood, typically before age three.
...
</doc>
...
Anarchism
Anarchism is a political philosophy that advocates self-governed societies based on voluntary institutions. These are often described as stateless societies, although several authors have defined them more specifically as institutions based on non-hierarchical free associations. Anarchism holds the state to be undesirable, unnecessary, and harmful.
...
Criticisms of anarchism include moral criticisms and pragmatic criticisms. Anarchism is often evaluated as unfeasible or utopian by its critics.
</doc>
<doc id="25" url="https://en.wikipedia.org/wiki?curid=25" title="Autism">
Autism
Autism is a neurodevelopmental disorder characterized by impaired social interaction, verbal and non-verbal communication, and restricted and repetitive behavior. Parents usually notice signs in the first two years of their child‘s life. These signs often develop gradually, though some children with autism reach their developmental milestones at a normal pace and then regress. The diagnostic criteria require that symptoms become apparent in early childhood, typically before age three.
...
</doc>
...
每个wiki_num文件里又存放若干个doc，每个doc都有相关的tag标记，包括id, url, title等，很好区分。
自然语言交流系统 phxnet团队 创新实训 个人博客 （十四）
声明：以上内容来自用户投稿及互联网公开渠道收集整理发布，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任，若内容有误或涉及侵权可进行投诉： 投诉/举报 工作人员会在5个工作日内联系你，一经查实，本站将立刻删除涉嫌侵权内容。