Phoenix Pay: Government got conflicting advice before launching ill-fated system

Don’t think I would recommend S.i. Systems given their candy coating compared to Gartner.

But Bagnall’s conclusion is right: really hard to put on the brakes on a major initiative at a late stage, given bureaucratic inertia and that people are vested in it going forward:

In the wake of last week’s damning report by auditor general Michael Ferguson — who concluded the pay system is at risk of chewing up $540 million more than its budgeted $310 million by 2019, with no end in sight — it’s worth re-examining some of the independent advice government agencies were getting in early 2016.

Treasury Board, along with Phoenix-sponsor Public Services and Procurement Canada, commissioned at least two reviews that were delivered just days before the February 2016 launch of the new pay system.

One review, by Gartner Inc., offered a number of important warnings, but the second report, by S.i. Systems, was surprisingly upbeat about the Phoenix project’s chances for success.

“The (Phoenix) initiative is very likely to achieve its goals and desired outcomes within the first year or two of full operations,” S.i. Systems noted in its draft final report dated Jan. 18, 2016. “All in-scope work has been completed, a (software) code freeze has been imposed on Phoenix and the Miramichi pay centre is fully operational.”

Ferguson last week gave short shrift to such sentiment, pointing out that roughly one in two federal government employees was experiencing a significant pay issue as of last June — fully 16 months after the launch of Phoenix.

S.i. Systems couched some of its conclusions with caveats, noting that the system was not yet fully automated, with the result some pay transactions were being dealt with manually. However, the consultants viewed this as a “temporary” issue during the transition from dozens of older pay systems to the consolidated Phoenix system.

S.i. Systems nevertheless was clear that Public Services and Procurement Canada — the department in charge of the project — should move ahead with Phoenix. Such a move “will be challenging,” the S.i. Systems report noted, “but it is likely that the problems and difficulties will be manageable.”

The consultants concluded “The (Phoenix) project team is to be commended for bringing this complex initiative to its current stage.”

The Gartner report, dated Feb. 11, 2016, offered a much different view. Not only did Gartner identify a dozen significant risks facing the impending rollout of Phoenix, it offered strategies for minimizing them. Many of the risks proved all too real, while the tips for reducing them were ignored.

Consider this item, offered in a discussion of potential problems associated with testing the new pay system: “End to end testing has not been performed by any department that Gartner has interviewed,” Gartner noted, “Best practice would dictate multiple end-to-end cycles be tested prior to go-live (in February 2016).”

The Gartner document added that its consultants were never provided with “a clearly documented testing strategy and plan.”

Gartner was hired on Dec. 21, 2015, leaving it just enough time to interview eight federal departments. Nevertheless, the sample included some of the largest ones (Health Canada, Employment and Social Development Canada and Public Services).

Other key risks identified by Gartner included training, support and transition.

For instance, Gartner notes that federal departments hadn’t yet implemented their training programs. This meant that if any gaps in training emerged it would be impossible to address these through revised or remedial courses before Phoenix went live. Gartner concluded the training shortfall could result in “unanticipated consequences such as an incorrect pay calculation.”

Gartner also brought attention to what has proved one of Phoenix’s most intractable problems — technical support for government employees using the system, a problem exacerbated by the reduction in the number of pay administrators starting in 2014.

Gartner correctly predicted there would be a very large number of queries facing pay administrators at the central location in Miramichi, N.B. — not least because employees across government had little opportunity beforehand to become familiar with Phoenix’s many quirks.

The consultants offered a number of suggestions for reducing the risks of the Phoenix rollout, including trying a more piecemeal approach. Divide the two main waves of employees into multiple waves, for instance, and start with the least difficult departments — those with relatively few seasonal employees, shift workers and other complicating features when it comes to pay.

Critically, Gartner also suggested running Phoenix in tandem with the older pay system as a contingency in case the new system didn’t perform as advertised.

These and other recommendations were ignored, with the result now all too plain to see. Nearly 350,000 pay transactions are today choking a system designed to accommodate 80,000.

To be fair, S.i. Systems also took note of the potential risks involved in abandoning the old pay system before making sure Phoenix actually worked. “(We) did not see evidence of a fallback or test strategy to mitigate this potentially risky event,” the S.i. Systems report noted in an Annex.

But the consultants downplayed the risk in its summary assessment that declared the Phoenix project was using an “excellent testing strategy” and that “when problems were encountered, appropriate and timely action was taken.”

But no matter the consultants’ advice, the final call about moving ahead with a project this big belonged to government. After nearly a decade in development, Phoenix suffered the flaw of unstoppable bureaucratic momentum. The directors of the project seemed not inclined to pay much attention to last-minute advice unless it happened to line up with where they were going anyway.

via Phoenix Pay: Government got conflicting advice before launching ill-fated system | Ottawa Citizen

U.S. consultants slam Shared Services Canada for failing projects

To the current government’s credit, it engaged Gartner to review the implementation of the shared services initiative.

The question remains whether officials who promoted and supported the previous government’s strategy provided sound advice on the risks and mitigation strategies, and whether or not Ministers and the government accepted it or not.

Complex IT projects are hard, and government by its very nature is not agile, further exacerbating risk:

Ottawa is in way over its head by attempting a massive transformation of its information-technology (IT) systems under Shared Services Canada, says a scathing indictment of the agency’s failings since 2011.

The government of Canada “has vastly underestimated the size, scale and complexity of this effort. … They are attempting the largest and most complex public-sector shared-service implementation ever considered,” concludes a $1.35-million report by international consultants.

“We … lack confidence in the ability of SSC (Shared Services Canada) and the GC (Government of Canada) to successfully execute the plan.”

The Jan. 12, 2017, report by consultant Gartner Inc. was ordered by the federal government last August, after repeated failures of the Phoenix payroll system and complaints from departments about Shared Services Canada’s inability to deliver technology upgrades, including new email systems.

U.S.-based Gartner brought together a five-person expert panel to examine the agency and its projects, a group that included executives experienced in public-sector digital transformations in California, Massachusetts and Northern Ireland, as well as the former IBM executive who handled big projects within that firm.

Shared Services Canada outsourcing

A $1.35-million consultants’ report, obtained by CBC News under the Access to Information Act, says Shared Services Canada is in way over its head trying to manage a massive transformation of technology. (Shutterstock)

The report lauds the project of consolidating the federal government’s information technology, including creation of a single email system, but says “very little progress” has been made in the last six years because of persistent management failures.

“Decision making cannot follow current approaches,” said the document, obtained by CBC News under the Access to Information Act.

“Execution must be based on agile, effective decision making, with clear and singular accountabilities. This is the antithesis of governance today.”

The report repeatedly underscores the enormous scale of the consolidation project, likening it to combining the infrastructure of between 30 and 40 large banks.

Slow-footed

The consultants say Shared Services Canada is slow-footed, partly hobbled by complex procurement rules, so that an email solution it chose in 2011 and still has not completed has since been outmoded by new cloud services.

“The world in 2016 is much different from how it was in 2011, and the expert panel and Gartner believe developments such as cloud services should be given much more prominence in SSC’s future,” said the 198-page report.

Some of the document is redacted, including key financial information. The authors make a series of recommendations, chief of which is the appointment of a deputy minister for IT for all of government, to whom the head of Shared Services Canada would report.

In April 2011, then-prime minister Stephen Harper lauded the project to consolidate the government’s IT systems and data centres, saying on the election campaign trail that year that “we know we can save all kinds of money there.”

‘The project was set up to fail through underfunding, lack of service standards, and poor planning from the previous government.’– Jean-Luc Ferland, spokesperson for Treasury Board President Scott Brison

The new agency charged with carrying out the transformation, Shared Services Canada, was announced on Aug. 4, 2011, after Harper won a majority.

But two projects in particular went off the rails in the early going, one to consolidate cell and telephone services, the other to consolidate email services. Both have been plagued by delays, among other problems.

And the new agency was immediately required to cut costs as part of a government-wide effort to wipe out the federal deficit by 2015.

Shared Services Canada data centre

Shared Services Canada is the department responsible for the federal government’s IT services, including its data centres. A new report says the federal government must create a new deputy minister of IT, to help get the troubled agency back on track. (Shared Services Canada)

Jean-Luc Ferland, a spokesperson for Treasury Board President Scott Brison, welcomed the consultants’ conclusions and recommendations, pinning much of the blame for the bad results on the former Conservative government.

“As the report makes clear, the motivation and objectives behind the creation of Shared Services Canada are even more relevant today than they were when it was conceived in 2011,” said Ferland.

“The report is equally clear that the former Conservative government failed to put in place the basic fundamentals for success at the time SSC was created. The project was set up to fail through underfunding, lack of service standards, and poor planning from the previous government.”

No timeline

Ferland said the government is still reviewing the recommendations, alongside those of the auditor general, House of Commons committees and other consultations. He did not provide a timeline for solutions.

“Our government’s ambition is to provide exemplary service to Canadians while making a seamless transformation to the age of digital government — not booking false savings, arbitrarily hobbling the public service, or cutting corners.”

Source: U.S. consultants slam Shared Services Canada for failing projects – Politics – CBC News

Wayne Smith, former Chief Statistician, continues his critique of Shared Services: Questionable transfers from Statistics Canada to Shared Services Canada