Claude Tokenizer

Tools

• claude_tokenizer_compare_dataset Tokenize the first rows of a dataset column with every selected tokenizer. Returns: A totals table over all rows, a summary of what was read, and a characters-per-token, bar chart.
• claude_tokenizer_compare_text Tokenize one text with every selected tokenizer. Returns: A counts table (tokenizer, tokens, ratio against the fewest, characters and bytes per, token, vocabulary size), a one-line size summary of the input, and the segmentation as, HTML. Counts exclude the fixed frame a one-message API request adds.
• claude_tokenizer_compare_text_1 Tokenize one text with every selected tokenizer. Returns: A counts table (tokenizer, tokens, ratio against the fewest, characters and bytes per, token, vocabulary size), a one-line size summary of the input, and the segmentation as, HTML. Counts exclude the fixed frame a one-message API request adds.
• claude_tokenizer_compare_text_2 Tokenize one text with every selected tokenizer. Returns: A counts table (tokenizer, tokens, ratio against the fewest, characters and bytes per, token, vocabulary size), a one-line size summary of the input, and the segmentation as, HTML. Counts exclude the fixed frame a one-message API request adds.
• claude_tokenizer_compare_text_3 Tokenize one text with every selected tokenizer. Returns: A counts table (tokenizer, tokens, ratio against the fewest, characters and bytes per, token, vocabulary size), a one-line size summary of the input, and the segmentation as, HTML. Counts exclude the fixed frame a one-message API request adds.
• claude_tokenizer_on_dataset_change
• claude_tokenizer_on_dataset_change_1
• claude_tokenizer_peek_columns Fill the column picker, ranked by mean word count over the first few rows. Not the first row's longest string: row 0 is often a header or canary whose fields are all the same length, and not length either — an id like `test-en-news.3585` is long and is not text. Words per row separates prose from identifiers whatever their length.
• claude_tokenizer_peek_columns_1 Fill the column picker, ranked by mean word count over the first few rows. Not the first row's longest string: row 0 is often a header or canary whose fields are all the same length, and not length either — an id like `test-en-news.3585` is long and is not text. Words per row separates prose from identifiers whatever their length.

No tool matches.

As the server's own listing gives them. We never call a tool.