converting Python bytearray with hexadecimal to a bytearray with bytes efficiently

I would like to convert a bytearray with ASCII hexadecimal encoded into straight bytes