One of the biggest issues is that the crowd and the TV audience at home don't get to hear the discussion between the ref, the linos and the VAR crew. This secrecy allows for speculation of bias. Let's give the entire officiating team microphones and let's hear what's going on throughout the game.
Look at what gets done in sports where similar technology is used, and look at the procedures they follow when video replays are used to assist the red on making their decision. Rugby Union has got a great model IMHO, and I'd say field hockey have worked it out too. NFL and NBA have used it for ages and periodically make improvements.
Football's application of VAR to date is so disjointed and flawed at the moment that it's hard to conclude that it isn't all deliberate.