TokenizerFunction Interface

public interface TokenizerFunction

Represents a callback method that can break a string into its component tokens.

Method Summary

Modifier and Type	Method and Description
abstract java.util.List<Token>	tokenize(String text, String locale) The callback method that can break a string into its component tokens.

public abstract List tokenize(String text, String locale)

The callback method that can break a string into its component tokens.

Parameters:

text - The input text.

locale - Optional, identifies the locale of the input text.

Returns:

The list of the found Token objects.