To be honest one of my daughters asked me and I couldn't think of a good reason other then it matches the playersYou don’t?
Rugby refs wear shorts, American football wear pants, baseball wear pants, cricket wear pants, ice hockey wear pads and helmet like players etc.
Assuming you’ve just had a few whiskeys before posting that.
Rugby aside - even though refs dont wear scrum caps - most of what you've cited have a lot of differences to the players.
Basketball refs dont wear shorts, cricketers wear tracksuit type trousers now but umpries wear proper trousers.
Football refs kit is more or less the same as the players kit nowadays.