fix(tokenizer): honor add_bos in HF tokenization - #2389

fix(tokenizer): honor add_bos in HF tokenization#2389

dajiaohuang wants to merge 4 commits into

abetlen:mainabetlen/llama-cpp-python:mainfrom

dajiaohuang:fix/hf-tokenizer-add-bosdajiaohuang/llama-cpp-python:fix/hf-tokenizer-add-bosCopy head branch name to clipboard