I am attempting to tokenize markdown files using python, but I am not sure on the best way to do this. I know python has a built-in tokenize library, but that is for python source code. I also know there are lots of great libraries like pandoc and python-markdown for converting markdown into other formats, but I’m not very familiar with these or how to use them to create tokens from markdown files.