Rugby Union is a very fluid game, and is not a stop/start game like Cricket and Tennis where this an 'end of current play'.
The big difference between Rugby Union and Football is the way it is perceived to be used. The crowd/TV audience know exactly what is being looked at, they know exactly what is being said (TV audience), they know exactly the thought process. There have been times when a try was not given because of something that happened a minute before the try was scored. The TMO is in the ear of the ref to tell him anything that has been missed, you will hear him say 'offside', 'high tackle', etc. and the on pitch ref will straight away signal a penalty.
It is not perfect (and never will be) and not everything is checked, it is still human based but there has always been a respect between refs an players in Rugby which, undoubtedly, helps. Football fans and players don't trust refs to be good at their jobs and this is one of the main reasons why fans want it scrapped because there is no knowledge of how it is being used.
I don't think it should be dumped but I doubt that any wholesale changes, in regards to the accountability of their decisions and process, will be made so will not be sad to see it go, in that respect.