TextChunker Class

Package:
com.microsoft.semantickernel.text
Maven Artifact:
com.microsoft.semantic-kernel:semantickernel-api:1.4.0
  • java.lang.Object
    • com.microsoft.semantickernel.text.TextChunker

public class TextChunker

Split text in chunks, attempting to leave meaning intact. For plain text, split looking at new lines first, then periods, and so on. For markdown, split looking at punctuation first, and so on.

Constructor Summary

Constructor Description
TextChunker()

Method Summary

Modifier and Type Method and Description
static java.util.List<java.lang.String> splitMarkDownLines(String text, int maxTokensPerLine)

Split markdown text into lines
static java.util.List<java.lang.String> splitMarkdownParagraphs(List<String> lines, int maxTokensPerParagraph)

Split markdown text into paragraphs
static java.util.List<java.lang.String> splitPlainTextLines(String text, int maxTokensPerLine)

Split plain text into lines
static java.util.List<java.lang.String> splitPlainTextParagraphs(List<String> lines, int maxTokensPerParagraph)

Split plain text into paragraphs

Methods inherited from java.lang.Object

java.lang.Object.clone java.lang.Object.equals java.lang.Object.finalize java.lang.Object.getClass java.lang.Object.hashCode java.lang.Object.notify java.lang.Object.notifyAll java.lang.Object.toString java.lang.Object.wait java.lang.Object.wait java.lang.Object.wait

Constructor Details

TextChunker

public TextChunker()

Method Details

splitMarkDownLines

public static List splitMarkDownLines(String text, int maxTokensPerLine)

Split markdown text into lines

Parameters:

text - Text to split
maxTokensPerLine - Maximum number of tokens per line

Returns:

List of lines

splitMarkdownParagraphs

public static List splitMarkdownParagraphs(List lines, int maxTokensPerParagraph)

Split markdown text into paragraphs

Parameters:

lines - Lines of text
maxTokensPerParagraph - Maximum number of tokens per paragraph

Returns:

List of paragraphs

splitPlainTextLines

public static List splitPlainTextLines(String text, int maxTokensPerLine)

Split plain text into lines

Parameters:

text - Text to split
maxTokensPerLine - Maximum number of tokens per line

Returns:

List of lines

splitPlainTextParagraphs

public static List splitPlainTextParagraphs(List lines, int maxTokensPerParagraph)

Split plain text into paragraphs

Parameters:

lines - Lines of text
maxTokensPerParagraph - Maximum number of tokens per paragraph.

Returns:

List of paragraphs

Applies to