Expose DelimitedTermFrequencyTokenFilter

[russcam]

Is your feature request related to a problem? Please describe.
Lucene provides DelimitedTermFrequencyTokenFilter to be able to provide tokens along with their term frequency. For example, a document with a text field with repeated terms like

{
   "text": "dog dog dog cat cat"
}

can use DelimitedTermFrequencyTokenFilter as part of analysis, and provide term frequencies along with terms

{
   "text": "dog|3 cat|2"
}

DelimitedTermFrequencyTokenFilter is not exposed in OpenSearch, even though it can be useful. Care needs to be taken when using it, per the lucene docs:

1. the field must be indexed with IndexOptions.DOCS_AND_FREQS but no positions or offsets.
2. make sure your Tokenizer doesn't split on the delimiter, or this won't work

Describe the solution you'd like
Expose DelimitedTermFrequencyTokenFilter as a token filter in OpenSearch.

Describe alternatives you've considered
The alternatives are

• to send documents with repeated terms. For many terms with large term frequencies, this can mean much larger payloads going over the wire.

or

• include and maintain an analysis plugin to provide the functionality.

Additional context

Exposing DelimitedTermFrequencyTokenFilter would be complimentary to exposing termFreq in Painless scripting: #9081

[noCharger]

While including DelimitedTermFrequencyTokenFilter into the OpenSearch codebase is great, I am wondering if using two separate fields in documents to store terms and their corresponding frequencies could be an alternative. This approach involves using a script query to calculate scores based on the term frequencies. I wanted to confirm if this is the actual use case you are aiming for.

PUT /myindex/_doc/1
{
  "text": "dog,cat",
  "freq": "3,2"
}

PUT /myindex/_doc/2
{
  "text": "dog,cat",
  "freq": "2,2"
}

GET /myindex/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "text": "dog"
          }
        }
      ],
      "should": [
        {
          "script_score": {
            "query": {
              "match_all": {}
            },
            "script": {
              "source": """
                String[] terms = doc['text'].value.split(',');
                String[] freqs = doc['freq'].value.split(',');
                double score = 0;
                for (int i = 0; i < terms.length; i++) {
                  if (terms[i].equals(params.value)) {
                    score += Double.parseDouble(freqs[i]);
                  }
                }
                return score;
              """,
              "params": {
                   "value": "dog"
                }
            }
          }
        }
      ]
    }
  }
}

[russcam]

@noCharger the alternative approach looks very suboptimal:

1. Need two fields to carry information that can be readily provided in one
2. Doesn't leverage the postings list, and instead requires use of doc values to make values available to scripting
3. Every script execution needs to process (split) doc values to make use of them, as opposed to indexing in a form requiring little to no processing
4. Can't make use of term statistics for multiple purposes i.e. scripting, boosting, etc. because individual terms and their freqs are not indexed in a form that persists this information

[macohen]

Russ, if you have an idea for implementation, feel free to submit as a PR.

cc: @msfroh, @rishabhmaurya, @jainankitk

[russcam]

I've opened #9479