Tokenizer Class

Package:
com.microsoft.bot.dialogs.choices
Maven Artifact:
com.microsoft.bot:bot-dialogs:4.14.3
  • java.lang.Object
    • com.microsoft.bot.dialogs.choices.Tokenizer

Implements

TokenizerFunction

public class Tokenizer
implements TokenizerFunction

Provides the default Tokenizer implementation.

Constructor Summary

Constructor Description
Tokenizer()

Method Summary

Modifier and Type Method and Description
java.util.List<Token> tokenize(String text, String locale)

Simple tokenizer that breaks on spaces and punctuation.

Methods inherited from java.lang.Object

java.lang.Object.clone java.lang.Object.equals java.lang.Object.finalize java.lang.Object.getClass java.lang.Object.hashCode java.lang.Object.notify java.lang.Object.notifyAll java.lang.Object.toString java.lang.Object.wait java.lang.Object.wait java.lang.Object.wait

Constructor Details

Tokenizer

public Tokenizer()

Method Details

tokenize

public List tokenize(String text, String locale)

Simple tokenizer that breaks on spaces and punctuation. The only normalization done is to lowercase.

Parameters:

text - The input text.
locale - Optional, identifies the locale of the input text.

Returns:

The list of the found Token objects.

Applies to