My ABS light is always on as well. I take out the fuse when off road but keep it in when on pavement.
It eventually throws a CEL. One of the ABS fuses under the dash also powers the diagnostic port for OBD. So you will throw a code and then be confused why you can't check what it is.
So I leave it in.
I also thought I read somewhere that if you are in a bad accident and they find out you took out the ABS fuse/altered the ABS system, you are screwed whether the ABS had anything to do with the accident or not.