I'm waiting for this to be an issue where I am. We are surrounded by wind and farm fields, so everything here gets covered under a layer of dust pretty quickly (I gave up on washing our cars more than once or twice a year or I would be doing it every weekend). We're also in California so we're probably never gonna get rain again(I kid of course, we'll probably get some in 2027), so I'm waiting to see at what point I have to go up on our roof with a hose. So far I haven't noticed a drop in output.
If you choose to rinse them, do it when it's dark or at least cloudy. The shock of a big stream of cold water on hot panels can and does cause microfractures. Even a heavy rain doesn't put the kind of mass of water that is easy to accidently do with a hose. Never use a rag/shammy/etc.