Skip to content

Conversation

@syedshazli
Copy link

@syedshazli syedshazli commented Dec 22, 2025

This change aims to address the TODO in tokenize.cpp, which asks to use log.h instead of printf and fprintf in some cases.

I decided to start off by replacing the usage information and error logging with LOG and LOG_ERR respectively. Once this gets approved, I aim to add more logging to tokenize.cpp using other log.h based features.

I have confirmed the errors and usage information still shows up in the command line after trying to input incorrect arguments and calling the --help flag.

@syedshazli
Copy link
Author

cc @ggerganov, codeowner of tokenizer and log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant