1 DESCRIPTION 2     The Natural Language Toolkit (NLTK) is an open source Python library 3     for Natural Language Processing.  A free online book is available. 4     (If you use the library for academic research, please cite the book.) 5      6     Steven Bird, Ewan Klein, and Edward Loper (2009). 7     Natural Language Processing with Python.  O‘Reilly Media Inc. 8     http://nltk.org/book 9     10     @version: 2.0.4

1 192:chapter2 rcf$ python2 Python 2.7.5 (default, Mar  9 2014, 22:15:05) 3 [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin4 Type "help", "copyright", "credits" or "license" for more information.5 >>> import nltk6 >>> nltk.download()

 1 from nltk.book import * 2 *** Introductory Examples for the NLTK Book *** 3 Loading text1, ..., text9 and sent1, ..., sent9 4 Type the name of the text or sentence to view it. 5 Type: ‘texts()‘ or ‘sents()‘ to list the materials. 6 text1: Moby Dick by Herman Melville 1851 7 text2: Sense and Sensibility by Jane Austen 1811 8 text3: The Book of Genesis 9 text4: Inaugural Address Corpus10 text5: Chat Corpus11 text6: Monty Python and the Holy Grail12 text7: Wall Street Journal13 text8: Personals Corpus14 text9: The Man Who Was Thursday by G . K . Chesterton 1908

 1 >>> text1.concordance("monstrous") 2 Building index... 3 Displaying 11 of 11 matches: 4 ong the former , one was of a most monstrous size . ... This came towards us ,  5 ON OF THE PSALMS . " Touching that monstrous bulk of the whale or ork we have r 6 ll over with a heathenish array of monstrous clubs and spears . Some were thick 7 d as you gazed , and wondered what monstrous cannibal and savage could ever hav 8 that has survived the flood ; most monstrous and most mountainous ! That Himmal 9 they might scout at Moby Dick as a monstrous fable , or still worse and more de10 th of Radney .‘" CHAPTER 55 Of the Monstrous Pictures of Whales . I shall ere l11 ing Scenes . In connexion with the monstrous pictures of whales , I am strongly12 ere to enter upon those still more monstrous stories of them which are to be fo13 ght have been rummaged out of this monstrous cabinet there is no telling . But 14 of Whale - Bones ; for Whales of a monstrous size are oftentimes cast up dead u

1 >>> text1.similar("monstrous")2 Building word-context index...3 abundant candid careful christian contemptible curious delightfully4 determined doleful domineering exasperate fearless few gamesome5 horrible impalpable imperial lamentable lazy loving

1 >>> text4.dispersion_plot(["citizens","democracy","freedom","duties","America"])

1 >>> len(text1)   #计算text1的词汇个数2 2608193 >>> len(set(text1)) #计算text1 不同的词汇个数4 193175 >>> len(text1[0])   #计算text1 第一个词的长度6 1

1 >>> sent12 [‘Call‘, ‘me‘, ‘Ishmael‘, ‘.‘]3 >>> sorted(sent1)4 [‘.‘, ‘Call‘, ‘Ishmael‘, ‘me‘]

1 >>> fdist1=FreqDist(text1)    #获取text1的频率分布情况2 >>> fdist1      　　　　　　　　#text1具有19317个样本,但是总体有260819个值3 <FreqDist with 19317 samples and 260819 outcomes> 4 >>> keys=fdist1.keys()       5 >>> keys[:50]                 #获取text1的前50个样本
6 [‘,‘, ‘the‘, ‘.‘, ‘of‘, ‘and‘, ‘a‘, ‘to‘, ‘;‘, ‘in‘, ‘that‘, "‘", ‘-‘, ‘his‘, ‘it‘, ‘I‘, ‘s‘, ‘is‘, ‘he‘, ‘with‘, ‘was‘, ‘as‘, ‘"‘, ‘all‘, ‘for‘, ‘this‘, ‘!‘, ‘at‘, ‘by‘, ‘but‘, ‘not‘, ‘--‘, ‘him‘, ‘from‘, ‘be‘, ‘on‘, ‘so‘, ‘whale‘, ‘one‘, ‘you‘, ‘had‘, ‘have‘, ‘there‘, ‘But‘, ‘or‘, ‘were‘, ‘now‘, ‘which‘, ‘?‘, ‘me‘, ‘like‘]

1 >>> fdist1.items()[:50]      #text1的样本分布情况，比如‘,‘出现了18713次，总共的词为2608192 [(‘,‘, 18713), (‘the‘, 13721), (‘.‘, 6862), (‘of‘, 6536), (‘and‘, 6024), (‘a‘, 4569), (‘to‘, 4542), (‘;‘, 4072), (‘in‘, 3916), (‘that‘, 2982), ("‘", 2684), (‘-‘, 2552), (‘his‘, 2459), (‘it‘, 2209), (‘I‘, 2124), (‘s‘, 1739), (‘is‘, 1695), (‘he‘, 1661), (‘with‘, 1659), (‘was‘, 1632), (‘as‘, 1620), (‘"‘, 1478), (‘all‘, 1462), (‘for‘, 1414), (‘this‘, 1280), (‘!‘, 1269), (‘at‘, 1231), (‘by‘, 1137), (‘but‘, 1113), (‘not‘, 1103), (‘--‘, 1070), (‘him‘, 1058), (‘from‘, 1052), (‘be‘, 1030), (‘on‘, 1005), (‘so‘, 918), (‘whale‘, 906), (‘one‘, 889), (‘you‘, 841), (‘had‘, 767), (‘have‘, 760), (‘there‘, 715), (‘But‘, 705), (‘or‘, 697), (‘were‘, 680), (‘now‘, 646), (‘which‘, 640), (‘?‘, 637), (‘me‘, 627), (‘like‘, 624)]

1 >>> fdist1.hapaxes()[:50]   #text1的样本只出现一次的词2 [‘!\‘"‘, ‘!)"‘, ‘!*‘, ‘!--"‘, ‘"...‘, "‘,--", "‘;", ‘):‘, ‘);--‘, ‘,)‘, ‘--\‘"‘, ‘---"‘, ‘---,‘, ‘."*‘, ‘."--‘, ‘.*--‘, ‘.--"‘, ‘100‘, ‘101‘, ‘102‘, ‘103‘, ‘104‘, ‘105‘, ‘106‘, ‘107‘, ‘108‘, ‘109‘, ‘11‘, ‘110‘, ‘111‘, ‘112‘, ‘113‘, ‘114‘, ‘115‘, ‘116‘, ‘117‘, ‘118‘, ‘119‘, ‘12‘, ‘120‘, ‘121‘, ‘122‘, ‘123‘, ‘124‘, ‘125‘, ‘126‘, ‘127‘, ‘128‘, ‘129‘, ‘130‘]
3 >>> fdist1[‘!\‘"‘]
4 1

1 >>> long_words=[w for w in set(text1) if len(w) > 15]  #获取text1内样本词汇长度大于15的词并按字典序排序2 >>> sorted(long_words)        3 [‘CIRCUMNAVIGATION‘, ‘Physiognomically‘, ‘apprehensiveness‘, ‘cannibalistically‘, ‘characteristically‘, ‘circumnavigating‘, ‘circumnavigation‘, ‘circumnavigations‘, ‘comprehensiveness‘, ‘hermaphroditical‘, ‘indiscriminately‘, ‘indispensableness‘, ‘irresistibleness‘, ‘physiognomically‘, ‘preternaturalness‘, ‘responsibilities‘, ‘simultaneousness‘, ‘subterraneousness‘, ‘supernaturalness‘, ‘superstitiousness‘, ‘uncomfortableness‘, ‘uncompromisedness‘, ‘undiscriminating‘, ‘uninterpenetratingly‘]4 >>> fdist1=FreqDist(text1)    #获取text1内样本词汇长度大于7且出现次数大于7的词并按字典序排序
5 >>> sorted([wforwin set(text5) if len(w) > 7 and fdist1[w] > 7]) 6 [‘American‘, ‘actually‘, ‘afternoon‘, ‘anything‘, ‘attention‘, ‘beautiful‘, ‘carefully‘, ‘carrying‘, ‘children‘, ‘commanded‘, ‘concerning‘, ‘considered‘, ‘considering‘, ‘difference‘, ‘different‘, ‘distance‘, ‘elsewhere‘, ‘employed‘, ‘entitled‘, ‘especially‘, ‘everything‘, ‘excellent‘, ‘experience‘, ‘expression‘, ‘floating‘, ‘following‘, ‘forgotten‘, ‘gentlemen‘, ‘gigantic‘, ‘happened‘, ‘horrible‘, ‘important‘, ‘impossible‘, ‘included‘, ‘individual‘, ‘interesting‘, ‘invisible‘, ‘involved‘, ‘monsters‘, ‘mountain‘, ‘occasional‘, ‘opposite‘, ‘original‘, ‘originally‘, ‘particular‘, ‘pictures‘, ‘pointing‘, ‘position‘, ‘possibly‘, ‘probably‘, ‘question‘, ‘regularly‘, ‘remember‘, ‘revolving‘, ‘shoulders‘, ‘sleeping‘, ‘something‘, ‘sometimes‘, ‘somewhere‘, ‘speaking‘, ‘specially‘, ‘standing‘, ‘starting‘, ‘straight‘, ‘stranger‘, ‘superior‘, ‘supposed‘, ‘surprise‘, ‘terrible‘, ‘themselves‘, ‘thinking‘, ‘thoughts‘, ‘together‘, ‘understand‘, ‘watching‘, ‘whatever‘, ‘whenever‘, ‘wonderful‘, ‘yesterday‘, ‘yourself‘]

1 >>> bigrams([‘more‘,‘is‘,‘said‘,‘than‘,‘done‘])2 [(‘more‘, ‘is‘), (‘is‘, ‘said‘), (‘said‘, ‘than‘), (‘than‘, ‘done‘)]3 >>> text1.collocations()4 Building collocations list5 Sperm Whale; Moby Dick; White Whale; old man; Captain Ahab; sperm6 whale; Right Whale; Captain Peleg; New Bedford; Cape Horn; cried Ahab;7 years ago; lower jaw; never mind; Father Mapple; cried Stubb; chief8 mate; white whale; ivory leg; one hand

 1 >>> type(text1) 2 <class ‘nltk.text.Text‘> 3 >>> help(‘nltk.text.Text‘) 4 Help on class Text in nltk.text: 5  6 nltk.text.Text = class Text(__builtin__.object) 7  |  A wrapper around a sequence of simple (string) tokens, which is 8  |  intended to support initial exploration of texts (via the 9  |  interactive console).  Its methods perform a variety of analyses10  |  on the text‘s contexts (e.g., counting, concordancing, collocation11  |  discovery), and display the results.  If you wish to write a12  |  program which makes use of these analyses, then you should bypass13  |  the ``Text`` class, and use the appropriate analysis function or14  |  class directly instead.15  |  16  |  A ``Text`` is typically initialized from a given document or17  |  corpus.  E.g.:18  |  19  |  >>> import nltk.corpus20  |  >>> from nltk.text import Text21  |  >>> moby = Text(nltk.corpus.gutenberg.words(‘melville-moby_dick.txt‘))22  |  23  |  Methods defined here:24  |  25  |  __getitem__(self, i)26  |  27  |  __init__(self, tokens, name=None)28  |      Create a Text object.29  |      30  |      :param tokens: The source text.31  |      :type tokens: sequence of str32  |  33  |  __len__(self)34  |  35  |  __repr__(self)36  |      :return: A string representation of this FreqDist.37  |      :rtype: string38  |  39  |  collocations(self, num=20, window_size=2)40  |      Print collocations derived from the text, ignoring stopwords.41  |      42  |      :seealso: find_collocations43  |      :param num: The maximum number of collocations to print.44  |      :type num: int45  |      :param window_size: The number of tokens spanned by a collocation (default=2)46  |      :type window_size: int47  |  48  |  common_contexts(self, words, num=20)49  |      Find contexts where the specified words appear; list50  |      most frequent common contexts first.51  |      52  |      :param word: The word used to seed the similarity search53  |      :type word: str54  |      :param num: The number of words to generate (default=20)55  |      :type num: int56  |      :seealso: ContextIndex.common_contexts()