Bugs in the System in the California Technology System

This is a long story.  But, important for you to understand.  Fro instance, thanks to the total incompetence of Arnold, Jerry and Gavin, the EDD will not allow you to even file an unemployment claim for two weeks.  Over the years due to payoffs, special interests and sweetheart deals, we have spent billions of tax dollars and are still in the 1980’s of technology in California.

Thank you for reading this post, don't forget to subscribe!

Bugs in the System

California Government Technology Failures and Recommended Solutions

By Adam B. Summers, Independent Institute,  9/17/20 

Introduction

It is the juxtaposition that has confounded public officials and taxpayers alike for decades now: How can California be home to the most innovative and successful technology companies in the world, and yet the state government seems incapable of maintaining halfway up-to-date information technology (IT) infrastructure or making new IT programs and updates work properly, much less be completed on time and under budget?

“We live in the tech capital of the world, yet we don’t have anybody driving the modernization of how we as government operate and provide services to the people we represent,” California Assembly Majority Leader Ian Calderon (D−Whittier), who also serves as co-chair for the Legislature’s technology caucus, said last year.

From Apple and Hewlett-Packard to Yahoo, Intel, Cisco Systems, Oracle, Facebook, eBay, and Google (Alphabet Inc.), the greatest minds in technology are a mere two-to-three-hour drive in decent traffic from the state Capitol. Yet, taxpayers and lawmakers alike continue to be baffled by the inability to modernize the state’s IT systems.

From numerous IT project failures at the Department of Motor Vehicles, going back to the 1980s, to the copious issues with today’s Financial Information System for California (FI$Cal) budgeting, accounting, cash management, and procurement system—a more than $1 billion project that has been in the works for 15 years and still is not ready for prime time—California state government has repeatedly doomed itself to greater inefficiency and years of project delays while saddling taxpayers with enormous costs for technology that does not work—and sometimes is so woefully inept that it must be scrapped altogether.

For these continual failings, the Independent Institute awards its tenth California Golden Fleece^® Award—a dishonor given quarterly to California state or local agencies or government projects that swindle taxpayers or break the public trust—to the agencies responsible for overseeing and administering the state’s various failed IT projects. These agencies include the California Department of Technology, Department of General Services, Department of Finance, State Controller’s Office, State Treasurer’s Office, Judicial Council of California, Administrative Office of the Courts, Department of Motor Vehicles, and Department of Consumer Affairs.

Background

California is hardly the only state or local government to suffer costly IT project failures, but it seems to struggle more than most, and the fact that it is the most populous state in the nation means that the size of its failures is also all the more massive.

“[B]etween 1994 and 2013, the state terminated or suspended seven IT projects after spending almost $1 billion,” the State Auditor’s Office reported in 2015. “In addition, during that time, the state paid $1 billion more in federal penalties for its delay in implementing the California Department of Social Services’ Child Support Automation System.”

As a result, the state auditor has repeatedly designated California’s IT projects as one of the “high-risk” issues facing the state. “The high costs of certain projects and the failure of others continues to make the state’s oversight of information technology projects an area of high risk,” it concluded in 2013. The state auditor also found that the California Department of Technology (CDT, also referred to as CalTech by the state auditor) did little to verify state agencies’ compliance with the State Administrative Manual and described its strategic planning efforts as “insufficient.”

Two years later, things were not appreciably better, and then state auditor Elaine M. Howle detailed a number of CDT’s oversight deficiencies:

CalTech’s independent project oversight (IPO) analysts are unclear when to recommend corrective actions to their managers, or when CalTech management should suspend or terminate a project. Furthermore, CalTech does not formally set expectations with agencies that are implementing IT projects. On a broader level, there is a potential conflict between IPO analysts’ role to oversee IT projects and their role to provide advice to agencies. Finally, high turnover, an insufficient state job classification, constrained resources, and inconsistent training of staff impacts CalTech’s ability to oversee state IT projects.

The Rise of Ransomware and Other Cybersecurity Threats

Poor technology practices heighten other risks as well. Ransomware attacks, malware, and other cybersecurity incidents targeted at federal, state, and local governments, as well as universities, have been on the rise in recent years. These can lead to data loss, systems failure (including, for example, the shutdown of 911 call centers or the inability to pay utility bills, property taxes, or fines), the theft of residents’ personal information, and expensive and time-consuming remediation efforts to remove malicious code, recover data, and restore functionality.

In 2016, a single hacker known as Rasputin was blamed for breaching the systems at more than 60 federal, state, and local government agencies and prominent universities in the United States and United Kingdom. These included the U.S. Department of Housing and Urban Development; Child Welfare Information Gateway (maintained by the U.S. Department of Health and Human Services); Postal Regulatory Commission; State of Oklahoma; Rhode Island Department of Education; South Carolina Public Employee Benefit Authority; District of Columbia Office of Contracting and Procurement; City of Pittsburgh; Cornell University; University of Washington; University of California, Los Angeles; University of Cambridge; and University of Oxford. Rasputin stole data from these systems and then attempted to sell it on the dark web.

Ransomware attacks, in particular, have led to some high-profile cybersecurity incidents. In these kinds of attacks, hackers infiltrate the target’s systems, encrypt them so they can no longer be accessed by the target entity, and demand a ransom—typically denominated in a cryptocurrency like bitcoin that is difficult to trace—in exchange for the digital keys to unlock the system. According to StateScoop, which maintains an interactive map of public-sector ransomware attacks, there have been 361 such incidents—including 23 attacks in California—documented since January 2013.

While targets are oftentimes smaller local governments—like the 23 Texas cities hit during a coordinated attack in 2019—because these are more likely to have limited budgets and less sophisticated technological defenses, large cities like Atlanta and Baltimore have also been victimized.

For those governments and institutions that are not adequately protected and find themselves victims of a ransomware attack, the decision whether to pay the extortionists is difficult and painful. Atlanta chose not to pay approximately $52,000 in ransom in March 2018 after an attack disrupted its Police Department records system, judicial system, and infrastructure maintenance requests, and prevented residents from paying their water bills for several days. The damage ultimately took weeks to fix and cost the city more than $2.6 million.

Baltimore similarly rejected ransom demands of roughly $76,000 following a May 2019 attack—the second such attack in a little more than a year—but ended up paying much more to fix things. According to the Baltimore Sun, “The attack left city employees without access to their email, halted real estate sales in the city, and held up water billing for months.” In all, it cost the city more than $18.2 million in direct costs and lost or delayed revenue. After the incident, Mayor Bernard Young placed a nonbinding resolution before the United States Conference of Mayors, which was adopted unanimously, opposing the payment of ransomware attackers by local governments.

Federal law enforcement agencies such as the FBI and Secret Service generally counsel against paying such ransom, under the theory that it encourages more attacks. Moreover, as Mayor Young noted, paying ransom does not guarantee that the attackers will live up to their end of the bargain and fully unlock systems, and there is always the chance that they leave behind other malware that will allow them to make additional demands in the future. Others may simply refuse to reward criminal behavior, even if sticking to this principle costs them more in the long run.

But it is difficult to fault a desperate government or institution from making a simple cost-benefit decision to pay the ransom in hopes of recovering their systems and data. The Los Angeles Community College District elected to pay a $28,000 ransom in bitcoin after hackers took control of a Los Angeles Valley College email and computer network in December 2016, only days before the start of the school’s winter session. And in June 2020, the University of California, San Francisco, paid more than $1 million in ransom after an attack made servers inaccessible at its School of Medicine, noting, “The data that was encrypted is important to some of the academic work we pursue as a university serving the public good.” Regardless of whether the larger trend is to pay the ransom or decide to deal with losses and try to fix things on one’s own, circumstances differ for various government agencies, institutions of higher learning, and private companies, and the threat of cyberattacks will not be going away anytime soon.

California IT Projects and Administration

CDT’s IT Project Tracking website lists 22 projects currently underway, with a total estimated cost of nearly $2.7 billion. Of these, under CDT’s color-coded system, four projects have been rated red—demanding immediate corrective action due to a significant risk to the health of the project—by independent project oversight reports, eight are rated yellow, indicating caution due to risks and issues identified, nine were rated green/satisfactory, and one has not been rated because there are not yet any reports available.

Among the big-ticket items rated red are the Department of Finance’s nearly $1.1 billion Financial Information System for California (FI$Cal), discussed in this report, and the Department of Corrections and Rehabilitation’s Statewide Correctional Video Surveillance Project ($386 million). The Secretary for California Health and Human Services Agency’s $421 million Child Welfare Services–California Automated Response and Engagement System (CWS-CARES) project (formerly known as the Child Welfare Services–New System) was recently upgraded from red to yellow.

California has tried a number of ways to oversee and implement its IT projects over the years. A single agency, which became the California Department of Information Technology, was responsible for IT project oversight from 1983 to 2002. It was determined that the agency was not successful, however, so its authorizing legislation was allowed to sunset and the oversight role was divided between the Department of Finance and the Department of General Services. The Office of the State Chief Information Officer (State CIO) was established in 2006, and IT project approval and oversight duties were transferred to the agency the following year.

Under the Arnold Schwarzenegger administration’s reorganization plan in 2009, the Department of Technology Services, Office of Information Security and Privacy Protection, and Telecommunications Division of the Department of General Services were merged into the California Technology Agency. During another reorganization in 2012 under the Jerry Brown administration, the California Technology Agency was renamed the California Department of Technology, which was under the direction of the State CIO, and the State CIO was removed from the governor’s cabinet (see Figure 1).

The latest development is California Gov. Gavin Newsom’s establishment of a new technology agency, the Office of Digital Innovation, with an initial budget of more than $40 million. (An agency within the CDT with the same name, which is devoted to developing technology solutions across state government, was subsequently rebranded as the Office of Enterprise Technology.) The new agency, which is charged with improving service delivery to members of the public using modern technology, will be exempt from many government procedures, and this added flexibility could prove to be a good thing if it can maintain transparency and accountability. Newsom does genuinely seem to be motivated to reform the state’s technology practices, and he has some expertise in this area. He coauthored the 2013 book Citizenville: How to Take the Town Square Digital and Reinvent Government. And he seems to grasp the scope of the problem, asserting in a Google interview that “California . . . when it comes to technology and governing, is on the leading and cutting edge of 1973.”

In yet another example of typical bureaucratic government inertia, however, Newsom’s tech agency will function in addition to—rather than replacing—the Schwarzenegger-era Office of Technology Services (formerly the Department of Technology Services), which operates with the same objective.

It is too soon to determine if the new Office of Digital Innovation will succeed in modernizing the state’s technology and improving government services, but if the many failed efforts of the past are any indication, residents and taxpayers are right to be skeptical.

Public-Sector vs. Private-Sector IT Management

Despite the state government’s IT project troubles, it would be unreasonable to expect all projects to go perfectly smoothly. With complex data projects serving hundreds of thousands of state workers or millions of citizens, there are no off-the-shelf products that will be able to handle that kind of scope or the unique demands of state agencies. Some trial and error—and, yes, even failure—is to be expected.

Though estimates vary widely, even in the private sector, large IT projects fail anywhere from about half the time to as much as 85 percent of cases (see, for example, related articles from CIO.com and Digital Journal and this study on IT project success and failure factors). The 2019 version of the Standish Group’s annual CHAOS report found that only about 16 percent of projects were deemed successful, defined as being completed on time, on budget, and with all the promised functionality. The majority (53 percent) were over cost, delayed, or did not deliver on all of the promised features, while 31 percent were total failures and had to be canceled or abandoned.

The private sector has some big advantages over the public sector, however—from the taxpayer’s or economist’s view, anyway. First, private firms are much more constrained by cost considerations. As with their other costs, IT spending will be limited based on the company’s profitability and lenders’ willingness to extend credit (with the expectation of being repaid with interest). They cannot simply rely on the faceless taxpayer or the strength of their lobbying to shoulder other government agencies out of the way for larger budgets. As a result, failing projects are more likely to be jettisoned sooner, and new solutions sought. Moreover, private companies’ and employees’ incentives to succeed are much stronger. A successful big data project may make a huge difference in a company’s bottom line—or even the difference between the business surviving or failing—and those responsible for the projects that fail are much more likely to find themselves out of a job. In the public sector, by contrast, financial decisions are based on politics instead of economic realities, taxes can always be raised, money can be shuffled from other places, the money is not yours but someone else’s, and employees enjoy such job protections that they rarely fear losing their jobs for poor performance.

Finally, contracting with a private company makes it easier to cut losses when things do not turn out well. Failed projects are not always the government’s fault. The private contractors responsible for creating and implementing the state’s IT systems often bear a share of the blame. Contractors may be negligent, disingenuously make low-ball bids, or be well-intentioned but find out that they have bitten off more than they can chew, or fail to live up to the contract due to unforeseen losses in key personnel or other factors beyond their control. Regardless of the reason, however, it is much easier to get rid of a bad contractor (and then find a good one to do the job right) than to put an end to an in-house project run by a monopolistic government bureaucracy.

As numerous independent oversight and audit reports have concluded over the years, taxpayers have had to endure more than their fair share of failed state IT projects, and there are a number of persistent problems with project planning, contracting, transparency, oversight, accountability, and execution that must be addressed in order to prevent future IT project boondoggles.

Case Studies

In telling the story of California’s string of IT project failures, there is no small list of projects from which to choose. A number of these problematic projects are described in Appendix A. In this section, I will begin with a few brief examples and then delve into some more in-depth case studies.

Department of Social Services – Statewide Automated Child Support System

One of the state’s early IT failures was the Statewide Automated Child Support System, a project of the Department of Social Services. The project was terminated in 1997 after five years and $111 million spent. In addition, the state was forced to pay $1 billion in fines to the federal government because of the delays in implementing the system.to finally begin later this year and take four years to complete, if all goes well.

TO SEE COMPLETE STORY CLICK ON HERE