Nothing
tests/testthat/test-pre_tokenizers.R
In tok: Fast Text Tokenization
test_that("can set a pre_tokenizer for a tokenizer", {
tok <- tokenizer$new(model_bpe$new())
tok$pre_tokenizer <- pre_tokenizer_whitespace$new()
expect_true(inherits(tok$pre_tokenizer, "tok_pre_tokenizer"))
})
Try the tok package in your browser
Any scripts or data that you put into this service are public.
tok documentation built on Sept. 11, 2024, 5:21 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
test_that("can set a pre_tokenizer for a tokenizer", {
tok <- tokenizer$new(model_bpe$new())
tok$pre_tokenizer <- pre_tokenizer_whitespace$new()
expect_true(inherits(tok$pre_tokenizer, "tok_pre_tokenizer"))
})
Try the tok package in your browser
Any scripts or data that you put into this service are public.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.