Files
llama.cpp/tests/peg-parser/simple-tokenize.h
admin 8e5a449007
Some checks are pending
Copilot Setup Steps / copilot-setup-steps (push) Waiting to run
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Waiting to run
Python check requirements.txt / check-requirements (push) Waiting to run
Python Type-Check / python type-check (push) Waiting to run
Update Operations Documentation / update-ops-docs (push) Waiting to run
llama.cpp verification source 2026-05-22
2026-05-22 16:44:08 +08:00

7 lines
114 B
C++

#pragma once
#include <string>
#include <vector>
std::vector<std::string> simple_tokenize(const std::string &);