Как я могу получить доступ к необработанным документам из корпуса Брауна?

Question

Для всех остальных корпусов НЛТК, звоните corpus.raw() возвращает исходный текст из файлов. Например:

>>> from nltk.corpus import webtext
>>> webtext.raw()[:10]
'Cookie Man'

Однако при звонке brown.raw() Вы получаете помеченный текст.

>>> from nltk.corpus import brown
>>> brown.raw()[:10]
'\n\n\tThe/at '

Я прочитал всю документацию, которую могу найти, но не могу найти очевидного объяснения или способа получить версию без тега. Есть ли причина, по которой этот корпус помечен, а другие нет?

Answer 1

TL;DR

import nltk
nltk.download('brown')
nltk.download('nonbreaking_prefixes')
nltk.download('perluniprops')

from nltk.corpus import brown
from nltk.tokenize.moses import MosesDetokenizer

mdetok = MosesDetokenizer()

brown_natural = [mdetok.detokenize(' '.join(sent).replace('``', '"').replace("''", '"').replace('`', "'").split(), return_str=True)  for sent in brown.sents()]

for sent in brown_natural:
    print(sent)

В длинном

Это потому, что "сырая" версия корпуса Брауна маркирована и помечена, т. Е. Корпус приходит помечен как исходная форма корпуса =)

Вы можете посмотреть отдельные файлы в вашем nltk_data каталог:

$ head -n10 nltk_data/corpora/brown/ca01


    The/at Fulton/np-tl County/nn-tl Grand/jj-tl Jury/nn-tl said/vbd Friday/nr an/at investigation/nn of/in Atlanta's/np$ recent/jj primary/nn election/nn produced/vbd ``/`` no/at evidence/nn ''/'' that/cs any/dti irregularities/nns took/vbd place/nn ./.


    The/at jury/nn further/rbr said/vbd in/in term-end/nn presentments/nns that/cs the/at City/nn-tl Executive/jj-tl Committee/nn-tl ,/, which/wdt had/hvd over-all/jj charge/nn of/in the/at election/nn ,/, ``/`` deserves/vbz the/at praise/nn and/cc thanks/nns of/in the/at City/nn-tl of/in-tl Atlanta/np-tl ''/'' for/in the/at manner/nn in/in which/wdt the/at election/nn was/bedz conducted/vbn ./.


    The/at September-October/np term/nn jury/nn had/hvd been/ben charged/vbn by/in Fulton/np-tl Superior/jj-tl Court/nn-tl Judge/nn-tl Durwood/np Pye/np to/to investigate/vb reports/nns of/in possible/jj ``/`` irregularities/nns ''/'' in/in the/at hard-fought/jj primary/nn which/wdt was/bedz won/vbn by/in Mayor-nominate/nn-tl Ivan/np Allen/np Jr./np ./.

Если вы хотите, чтобы слова из корпуса, вы можете использовать brown.words()например,

>>> from nltk.corpus import brown

>>> brown.words()
[u'The', u'Fulton', u'County', u'Grand', u'Jury', ...]

>>> ' '.join(brown.words()[:30])
u"The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced `` no evidence '' that any irregularities took place . The jury further said in"

Если вы хотите получить слова из определенного файла:

>>> brown.fileids()[:10] # The first 10 fileids from brown.
[u'ca01', u'ca02', u'ca03', u'ca04', u'ca05', u'ca06', u'ca07', u'ca08', u'ca09', u'ca10']

>>> ' '.join(brown.words('ca01')[:30]) # First 30 words from the 'ca01' file.
u"The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced `` no evidence '' that any irregularities took place . The jury further said in"

И предложения из конкретного файла:

>>> brown.sents('ca01')
[[u'The', u'Fulton', u'County', u'Grand', u'Jury', u'said', u'Friday', u'an', u'investigation', u'of', u"Atlanta's", u'recent', u'primary', u'election', u'produced', u'``', u'no', u'evidence', u"''", u'that', u'any', u'irregularities', u'took', u'place', u'.'], [u'The', u'jury', u'further', u'said', u'in', u'term-end', u'presentments', u'that', u'the', u'City', u'Executive', u'Committee', u',', u'which', u'had', u'over-all', u'charge', u'of', u'the', u'election', u',', u'``', u'deserves', u'the', u'praise', u'and', u'thanks', u'of', u'the', u'City', u'of', u'Atlanta', u"''", u'for', u'the', u'manner', u'in', u'which', u'the', u'election', u'was', u'conducted', u'.'], ...]

Чтобы распечатать отдельные предложения:

>>> for sent in brown.sents('ca01')[:5]: # First 5 sentences.
...     print(' '.join(sent))
... 
The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced `` no evidence '' that any irregularities took place .
The jury further said in term-end presentments that the City Executive Committee , which had over-all charge of the election , `` deserves the praise and thanks of the City of Atlanta '' for the manner in which the election was conducted .
The September-October term jury had been charged by Fulton Superior Court Judge Durwood Pye to investigate reports of possible `` irregularities '' in the hard-fought primary which was won by Mayor-nominate Ivan Allen Jr. .
`` Only a relative handful of such reports was received '' , the jury said , `` considering the widespread interest in the election , the number of voters and the size of this city '' .
The jury said it did find that many of Georgia's registration and election laws `` are outmoded or inadequate and often ambiguous '' .

Попытка детокенизации токенизированного корпуса довольно грязная и может или не может работать, но вы можете попробовать MosesDetokenizer:

Сначала загрузите данные, необходимые для MosesDetokenizer:

>>> import nltk
>>> nltk.download('perluniprops')
[nltk_data] Downloading package perluniprops to
[nltk_data]     /home/ltan/nltk_data...
[nltk_data]   Unzipping misc/perluniprops.zip.
True
>>> nltk.download('nonbreaking_prefixes')
[nltk_data] Downloading package nonbreaking_prefixes to
[nltk_data]     /home/ltan/nltk_data...
[nltk_data]   Package nonbreaking_prefixes is already up-to-date!
True

Затем инициализируйте MosesDetokenizer:

>>> from nltk.tokenize.moses import MosesDetokenizer
>>> mdetok = MosesDetokenizer()

И использовать MosesDetokenizer.detokenize():

>>> for sent in brown.sents('ca01')[:5]: # First 5 sentences.
...     # Join the words in sentences and convert the `` -> "
...     # also convert '' -> " and ` -> '
...     munged_sentence = ' '.join(sent).replace('``', '"').replace("''", '"').replace('`', "'")
...     print(mdetok.detokenize(munged_sentence.split(), return_str=True)) # MosesDetokenizer expects a list of strings as input.
... 
The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced "no evidence" that any irregularities took place.
The jury further said in term-end presentments that the City Executive Committee, which had over-all charge of the election, "deserves the praise and thanks of the City of Atlanta" for the manner in which the election was conducted.
The September-October term jury had been charged by Fulton Superior Court Judge Durwood Pye to investigate reports of possible "irregularities" in the hard-fought primary which was won by Mayor-nominate Ivan Allen Jr..
"Only a relative handful of such reports was received", the jury said, "considering the widespread interest in the election, the number of voters and the size of this city".
The jury said it did find that many of Georgia's registration and election laws "are outmoded or inadequate and often ambiguous".

Чтобы преобразовать каждое предложение в brown в естественное чтение текста:

from nltk.tokenize.moses import MosesDetokenizer
mdetok = MosesDetokenizer()
brown_natural = [mdetok.detokenize(' '.join(sent).replace('``', '"').replace("''", '"').replace('`', "'").split(), return_str=True)  for sent in brown.sents()]

[из]:

>>> for sent in brown_natural:
...     print(sent)
...     break
... 
The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced "no evidence" that any irregularities took place.

Answer 2

Помеченный текст является необработанным документом, фактическим содержимым файлов Brown corpus. raw() метод показывает, что именно хранится в файлах; он возвращает только чистый текст для "простого текста", а не для "всех других корпусов", как вы предполагаете. Пытаться nltk.corpus.treebank.raw('wsj_0001.mrg') или же nltk.corpus.conll2000.raw("train.txt")Например, вы увидите деревья и текст в формате IOB соответственно.

Теперь, если ваша цель состоит в том, чтобы восстановить читаемый текст, объединение пробелов обычно достаточно для меня:

for sent in brown.sents():
    print(" ".join(sent))

Вы получите вывод, как это:

`` Only a relative handful of such reports was received '' , the jury said , `` considering
the widespread interest in the election , the number of voters and the size of this 
city '' .

Если вам не нравится, как это выглядит, посмотрите ответ alvas для более амбициозной реконструкции.