My answer to this question is usually that it persists because politicians -- the ones who can change the laws affecting water allocations -- like it that way. Why do they not reform the system to put it on a sustainable basis? Because that action would reduce their power.
This logic clarifies why it's so rare to find information of who has water rights, how much they total, and how they compare to actual water flows. These data are easy to gather in theory, but they cannot or will not be gathered without support or funding from the public administration that can order the data collected.
What we get instead is typical: a politician promises more water than there is to constituents seeking favors. In many instances, overallocations are not noticed because they are not publicized or compared to actual supply (the data problem) but also because it may take years or extreme events to reveal that the sum of allocations over time is too large.
The Colorado River Compact of 1922 is a famous example of an allocation that suffered two flaws. First, the river was overallocated according to flows -- 16.4 million acre feet (MAF) of allocations were given out when average flows were 13.2 MAF. Second, the allocations were in terms of quantities instead of percentages. Why was Colorado water allocated that way? Politicians sought to please as many people as possible with water rights, so they gave away 100 percent of what they thought was there (no safety margin, let alone environmental flows). Politicians also wanted to give people firm property rights in fixed quantities of water, which would make it easier to farm, build cities, etc. Percentage shares would have been MUCH better in terms of sustainability and reality, but water users don't like hearing they are going to get 80 percent of some number they prefer.*
So the Colorado River was broken from the start, and it's not getting any better... because politicians do not want to reconfigure rights to users (not states) on a percentage basis. They want to keep rights at the state level -- to manipulate and reward their fans -- and negotiate adjustments in
I'm hopeful for data collection projects that will point out the gap between water rights and water reality, but I'm sad that the powers that be -- those who could speed those projects up by a factor of 10 -- are not giving logistical, legal or financial help. The powers that be prefer failure that delivers job security to them to reform and success that sidelines them.
Bottom Line: Water scarcity is rising in many places. It will cause the most harm where politicians try to hide or control the problem.
* I see that SWP and CVP allocations in California are below 50 percent for irrigators. I wonder