add StanfordSentimentTreeBankDatasetReader.apply_token_indexers() (#…

…273) * add `StanfordSentimentTreeBankDatasetReader.apply_token_indexers()`
allenai · Jun 8, 2021 · d721487 · d721487
1 parent 996adff
commit d721487
Show file tree

Hide file tree

Showing 2 changed files with 6 additions and 1 deletion.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 - Added support for NLVR2 visual entailment, including a data loader, two models, and training configs.
+- Added `StanfordSentimentTreeBankDatasetReader.apply_token_indexers()` to add token_indexers rather than in `text_to_instance` 
 
 ### Fixed
 

diff --git a/allennlp_models/classification/dataset_readers/stanford_sentiment_tree_bank.py b/allennlp_models/classification/dataset_readers/stanford_sentiment_tree_bank.py
@@ -128,7 +128,7 @@ def make_token(t: Union[str, Token]):
  tokens = [make_token(x) for x in tokens]
  else:
  tokens = self._tokenizer.tokenize(" ".join(tokens))
- text_field = TextField(tokens, token_indexers=self._token_indexers)
+ text_field = TextField(tokens)
  fields: Dict[str, Field] = {"tokens": text_field}
  if sentiment is not None:
  # 0 and 1 are negative sentiment, 2 is neutral, and 3 and 4 are positive sentiment
@@ -153,3 +153,7 @@ def make_token(t: Union[str, Token]):
  sentiment = "1"
  fields["label"] = LabelField(sentiment)
  return Instance(fields)
+
+ @overrides
+ def apply_token_indexers(self, instance: Instance) -> None:
+ instance.fields["tokens"].token_indexers = self._token_indexers