I once briefly drove my DII without the front cross-member in place. I could immediately feel the difference. Wouldn't do it again.
You must be more sensitive than I am. I removed mine on my last 3 D2's, and haven't noticed any difference, even with significant off-road use.