That key problem isn't actually true when For / Against VAR calls are tallied up. Unless the six clubs you're talking about are Villa, Chelsea, Everton and 3 from Brighton, Burnley, Fulham or Spurs.
The biggest issue with VAR for me is the stop /start nature and the way it kills the moment...