You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Reason for not applying remove_non_prining_characters normalization #416
We are much inspired by this great work and are in the process of cleaning our data. However, if we understand correctly, the remove_non_prining_characters normalization step is not used for the final cleaning. Do you have any thoughts on why this should not be used?
This is the main reason why this function is present in the code. If we didn't use it for the normalization of the documents, it was probably because there was \n or \t in the list as you mentioned, but it would make sense to use this function without these characters for the normalization of the documents (but not for before the tokenization, because the tokenizer did not see any \n or \t during its training).
I don't really know why \n or \t are in the list, it's mostly a code from Facebook CCNet which was used for the training of the tokenizers and KenLM models. I think the only thing we modified from them was not converting the characters to lower case. @edugp did this part.
So if you want to use the same tokenizers or KenLM models as us, you should check the parameters of the normalization applied before and use the same ones.
Activity
HugoLaurencon commentedon May 20, 2022
Hi, thank you for your comment!
The
remove_non_printing_charactersfunction was not used during the normalization of the documents:data_tooling/ac_dc/filtering.py
Line 357 in e28064e
However, it was used just before the tokenization step:
data_tooling/ac_dc/filtering.py
Line 213 in e28064e
and
data_tooling/ac_dc/filtering.py
Line 688 in e28064e
Because we trained our tokenizers and KenLM models (https://huggingface.co/edugp/kenlm/tree/main/wikipedia) on data after removing these non-printing characters, to be sure the new data we pass to the tokenizer is the same form as the data it was trained on, we added this function as it was.
This is the main reason why this function is present in the code. If we didn't use it for the normalization of the documents, it was probably because there was
\nor\tin the list as you mentioned, but it would make sense to use this function without these characters for the normalization of the documents (but not for before the tokenization, because the tokenizer did not see any\nor\tduring its training).I don't really know why
\nor\tare in the list, it's mostly a code from Facebook CCNet which was used for the training of the tokenizers and KenLM models. I think the only thing we modified from them was not converting the characters to lower case. @edugp did this part.So if you want to use the same tokenizers or KenLM models as us, you should check the parameters of the normalization applied before and use the same ones.
Don't hesitate if you have more questions!