Stu, I think you are right and to build on that, here's how PT explained it to me, more or less:
Unlike engine oil, which collects and filters crap produced during combustion, the AT is a closed system. In order to work properly (this is where I found things counterintuitive) the AT needs some element of friction, and this is achieved in part by particulates suspended in the fluid. If you replace the fluid and the filter the transmission is now bathed in a medium that doesn't provide sufficient friction because it doesn't have the suspended particulates of the old fluid. Something like that.
There was more to it, and I'm probably butchering PT's explanation with my paraphrasing, but that was the gist of it. Bottom line: if my transmission has made it to 167,000 miles without any problems, don't mess with it now. Seems reasonable.