May: Federal paymasters struggle to rise from Phoenix

Ongoing so many years lager:

Thousands of workers are still owed money when they leave or retire from Canada’s public service, and some are still waiting years to get paid. Thousands more move to jobs in other departments and wait months, if not years, for their paperwork to catch up with them.

It’s been six years – and more than $1.4 billion in fixes – since the disastrous Phoenix launch. The government wanted the pay system to be stabilized this year, meaning the backlog of erroneous pay cases would be eliminated permanently. It’s not there yet.

The spotlight on the Phoenix fiasco all but disappeared when the pandemic took the national stage and the number of pay hardship cases fell. The errors that remain aren’t as egregious as they were in the early days when people were getting underpaid or overpaid massively, or not paid at all. However, despite improvements, a backlog of trouble remains.

The Office of the Auditor General found an overall error rate of 47 per cent in the pay of employees it sampled in 2020-21. That’s slightly lower than the 51 per cent error rate the prior year. It also flagged that 41 per cent still had pay errors waiting to be fixed at the end of the year – compared to 31 per cent the year before.

With $910 million in funding in 2020, Public Services and Procurement Canada (PSPC) put together a three-year plan to get rid of the backlog by December 2022.

The backlog-reduction strategy was built around using technology, boosting the productivity of pay advisers and reducing the volume of cases coming in. The idea is to help pay advisers plow through the flow of transactions every pay cycle and also have enough time to work on the backlog.

The size of the backlog steadily decreased for more than two years, from the peak in January 2018 of 384,000 transactions, to a low of 98,000 in April 2021. The number of people affected dropped. The pay centre also improved its wait times, meeting the service standard of dealing with most pay requests within 20 days at least 80 per cent of the time.

But then it started to backslide.

Phoenix’s processing started to hiccup and sputter for a couple of months before it inched back up throughout the summer and fall of 2021. The active backlog of transactions settled at 141,000 cases by late January 2022.

“When it comes to Phoenix, it is two steps forward, one step back. The backlog of outstanding transactions has reduced since 2016, but whenever a new collective agreement or MOU is signed, it slows down the progress,” said Dany Richard, president of the Association of Canadian Financial Officers.

PSPC officials argued the backslide was nothing more than the normal flux in the volume of work.

They said the number of transactions processed each month varies depending on the complexity of transactions, the pay centre’s capacity, any new collective agreements being implemented, and the normal seasonal spikes in demand, such as hiring summer students.

The size of the public service has also grown 24 per cent since 2015 and there’s been a considerable amount of churn – with people moving in and out of jobs, creating more transactions.

“Although we have made significant progress this past year in reducing the number of transactions in the backlog and queue from 2016-2020, we have seen an increase in new transactions received at the pay centre in 2021, which challenged our ability to reduce the backlog,” said PSPC spokesperson Michèle LaRose in an email.

Some say public servants, many safely working from home, aren’t complaining as loudly about pay errors when so many Canadians lost their jobs during the COVID-19 crisis. Some argue that unions backed off on Phoenix after winning $560 million in damages to compensate workers for pay gaffes. Others say public servants are now so used to pay errors that they have simply become inured to them.

“As the backlog of Phoenix pay issues continues to grow, it seems the government has quietly accepted that Phoenix can’t be fixed, “said Chris Aylward, president of the Public Service Alliance of Canada.”

“We’re nearing the six-year anniversary of the Phoenix fiasco, and thousands of workers are still having pay problems while waiting years for their previous issues to be resolved…Over the past few months, we’re seeing more and more members turn to us for help.”

The auditor general found that 141,100 employees had at least one outstanding “pay action request” last June, down from more than 182,000 in 2018.

The improvements made to Phoenix came partly through technology, such as automating retroactive payments like backpay for 2018 collective agreements. Processes were streamlined, programs introduced to speed up processing of routine work. This was all aimed at freeing up pay advisers to be deployed on the backlog.

The next focus was the culture change around timeliness and accuracy, which should have happened before Phoenix was launched. Managers, human resource and employees had to learn how to manage pay differently. The old pay system could handle late transactions – largely because pay advisers manually fixed things. But Phoenix runs in real time and can’t digest late or after-the-fact transactions.

That cultural shift from late to on-time transactions is slowly taking hold but problems remain around the timeliness and accuracy of requests for terminations and transfers.

At last count, 30,000 terminations were waiting to be processed, either in the queue or dumped into the backlog after the 45-day service standard is missed. Terminations are issued for anyone who leaves the public service.

About 34 per cent of the cases in backlog are less than a year old, but 12 per cent are more than four years old and the rest are between one and four years old. Employees could be owed vacation pay, overtime, allowances and/or severance pay.

About 10,000 transfers are waiting to catch up with employees in their new departments. Those who move to new jobs with raises or promotions could wait months or years for that extra money to kick in – and then discover they have other pay adjustments such as higher pension premiums.

Overpayments are another headache. Departments are racing to recover salary overpayments to employees in 2016, when Phoenix went live. By law, the government has six years to collect salary overpayments from employees – after which they become unrecoverable and are written off as debt. The government has collected about $2.3 billion so far but has another $552 million to collect.

Sources familiar with Phoenix say the problems dogging the system can be fixed with more pay advisers. PSPC has already quadrupled its pay workforce— to about 2,500 people—since Phoenix was launched.

Source: Federal paymasters struggle to rise from Phoenix

Ottawa pay mess shows how hard it is to fire anyone in this town: McKenna

Valid critique over the lack of accountability. Surprised that no one has been leaking the names:

It is one of the lamest whodunits in Canadian history.

We know that three senior bureaucrats badly botched the creation and roll-out of a new pay system for nearly 300,000 federal workers. The trio left behind a trail of misery, including a relentless stream of pay errors, disrupted lives and hundreds of millions of dollars in cost overruns.

But Ottawa won’t say who the three are or how they were disciplined. We do know that no one was fired. And two of the three officials still work for the department that runs the pay system, Public Services and Procurement Canada; the other retired.

The identity of the trio − including an assistant deputy minister and an associate assistant deputy minister − must be the worst-kept secret inside the bureaucracy. Auditor-General Michael Ferguson laid out exactly what happened in a scathing report, calling the new Phoenix payroll system an “incomprehensible failure of project management and oversight.” Anxious to stick to a schedule and stay on budget, the three officials launched the system in early 2016 even though they knew it was not working properly and had dangerous security holes, the report found.

The department of Public Services isn’t naming names, citing “internal matters.”

Nor is Privy Council Clerk Michael Wernick, the government’s top civil servant and the de facto chief operating officer of the bureaucracy. Appearing last week before a parliamentary committee that is investigating the failed system, Mr. Wernick was more interested in taking swipes at Mr. Ferguson for smearing the integrity of the public service than laying blame.

Surely, the buck-passing has to stop somewhere. How can Canadians have faith in their government if no one is ever held accountable for the biggest mess ups?

Absent that, Mr. Ferguson is right. The culture inside the federal government is broken.

The mysterious Phoenix trio ignored dire warnings from outside consultants and officials in other departments that they were hurtling toward disaster. They failed to do a pilot test and had no back-up plan in place if anything went wrong. And they stripped the system of 100 key functions, such as paying employees who switch jobs or who file for back pay. They knew the system would not work properly and then apparently kept their own bosses in the dark about these problems as the system went live. All of this was done in the name of expediency and saving money, according to the Auditor-General.

Paying its employees is one of the most basic functions of any organization, and the federal government failed miserably. A system that was to have cost $310-million has soared to more than $1-billion, and the price is still rising as the government struggles to make the system do what it’s supposed to do – pay people what they are owed. Virtually every government worker has been touched in some way by the resulting mess, including tens of thousands of workers not paid for months and others paid too much (and then were overtaxed).

Given all that, you might expect heads to roll. Not in Ottawa. Public Services and Procurement Canada says only that the performance of senior officials was “assessed and appropriate measures taken.” The department did confirm that the officials involved did not get their bonuses in the year Phoenix was launched.

But what about the years before, when they made a series of fatal errors that condemned the launch to failure?

In the private sector, these would almost certainly be firing offences. Federal government leaders want to be paid on par with executives in the private sector, and many are. But they also enjoy a lot more job and pension security.

Mr. Wernick, the Clerk of the Privy Council, acknowledged that it’s extremely hard to fire anyone below the deputy minister rank for poor performance, citing protections provided by Public Service Employment Act. And he suggested that the government’s bonus and incentive system for senior managers may cause some to focus too much on cost over function.

That’s hardly confidence-inspiring for most of us who work in the private sector, where bonuses are rare and people frequently get fired when they mess up.

via Ottawa pay mess shows how hard it is to fire anyone in this town – The Globe and Mail

Top bureaucrat rejects auditor general’s ‘opinion piece’ on broken government culture

Fairly combative appearance, perhaps reflecting ongoing frustration with auditor general reports. Phoenix, however one looks at it, is a classic large-scale bureaucratic failure.
My memory is long enough (as it the Clerk’s!) to remember the universal classification system (UCS) initiative in the 1990s that consumed an enormous amount of time in reviewing job descriptions and criteria only to be abandoned. IMO there are cultural factors that make such large-scale transformations high risk, and there is enough collective experience to be more wary about proposing these kinds of initiatives given the implementation challenges:
Top bureaucrat Michael Wernick has rejected the auditor general’s assertion last month of a broken culture in the federal government that enabled the Phoenix pay system disaster. While there’s room for improvement, and Phoenix was a failure, the kind of deep malaise that Michael Ferguson described in his message accompanying his spring reports to Parliament does not reflect the reality of the public service, the Clerk of the Privy Council said during a House of Commons committee meeting Tuesday,.
“I believe it contains sweeping generalizations, it’s not supported by evidence, and it does not provide you any particular guidance on what to do to move forward,” Wernick said in his opening comments to the committee, calling the auditor general’s message “an opinion piece which I take issue with.”
Wernick told the parliamentarians who comprise the public accounts committee, many of whom greeted his words with skepticism, that he saw Phoenix as a “perfect storm,” the culmination of multiple factors that have already been laid out in two auditor general reports and an independent study by consulting firm Goss Gilroy Inc.
David Christopherson, a New Democrat MP and committee vice-chair, challenged Wernick on his conclusions.
“With all due respect … either we have a (Clerk) of the Privy Council who has his head buried in the sand and is in complete denial with what the cultural problems are, or we’ve got an auditor general that is off the rails. “Where does that leave us?”
But Wernick said that contrary to what the auditor general observed, the public service does not have a pervasive problem with deputy minister turnover. Of the 33 deputy ministers over which Wernick said he has some influence and the last three terms they’ve each completed, 49 of 99 were more than three years, 27 more than four years, and 16 more than five years.
And Public Services and Procurement Canada, the same department that oversaw the botched Phoenix rollout, delivered parliamentary precinct construction projects on time, on budget, and fully-functioning, he said. “I’m not saying the public service culture is perfect … We are risk-averse, we are process and rules-driven, we need to be more nimble, we need to be more creative, we need to be more assertive,” Wernick later concluded.
“What I take issue with is the insinuation that it is a generalized broken culture, which implies a generalized broken public service, and I have to contest that.”
He registered his belief that the public service needs structural reform. It has too many layers, he said, having climbed 15 of them to get to the position he holds today. The hundreds of classification groups and thousands of special pay groups and allowances make building an effective pay system extremely challenging, he said.
He also recommended the committee consider the incentive structure under which public servants operate. There are numerous layers of oversight and feedback around the senior bureaucrats, Wernick said, and almost all are negative. The exceptions are performance pay and promotion.
“Culture is shaped by incentives and disincentives,” he said, and there are opportunities to create those “which reward innovation, creativity, or that stifle it.”
But MPs continued to raise the question of a larger cultural crisis throughout the bureaucracy. Conservative MP Lisa Raitt pointed out that Ferguson isn’t the first to come to this conclusion.
Last year, public service integrity commissioner Joe Friday flagged a culture of fear silencing public servants from speaking out about wrongdoing. And Kevin Sorenson, committee chair and Conservative MP, cited letters his office had received from public servants “saying this culture has to be fixed.”
Wernick pointed out these letters and emails come from those motivated enough to write, and “officers of parliament have their role and have their opinion, but they are outside observers.”
For the most part, Canada’s public service is free of nepotism, corruption and partisanship, he said.
“It’s important in this day and age that Canadians have some confidence in their public institutions, and I am committed to making them better as we go along.” But, he cautioned, “be very careful on the diagnosis before you start prescribing remedies. There are a lot of governance quacks out there, and I think it’s important to listen carefully to people with some expertise.”
Wernick also extended some cultural advice of his own to the committee: create a space in which questioning the auditor general is possible. For a decade or more, he said, government was taught the only way to respond to auditor general recommendations was with agreement. “It should be OK to challenge the analysis and the findings of the auditor general. It will make for a healthier, richer debate.”

Source: Top bureaucrat rejects auditor general’s ‘opinion piece’ on broken government culture

Fixing a broken culture: public service in the wake of Phoenix

Good representative set of comments. Minister Brison has a valid point: service delivery is a very poor cousin compared to policy and communications, with most senior execs having moved up the ladder largely on their ability to deliver on policy:

In no uncertain terms, auditor general Michael Ferguson laid bare last month his belief that a “broken government culture” enabled the Phoenix pay system fiasco to play out, despite bureaucratic safeguards that should have been enough to prevent the failure.

However, after explaining the “why” of Phoenix — “an obedient public service that fears mistakes and risk,” unwilling or unable to hear and convey “hard truths” — in a message accompanying the audit of its building and implementation, Ferguson left it up to the federal government to puzzle out a remedy.

That solution depends on who you ask.

THE FORMER PBO: New leadership needed Kevin Page, president and chief executive of the Institute of Fiscal Studies and Democracy at the University of Ottawa, proposed a stark fork in the bureaucracy’s road forward. “You either get rid of the top echelon of the public service — all the deputy ministers in the central agencies go, and replace them with a new group that we feel confident have the competency and the values and make sure that this stuff won’t repeat over and over again, or we just wait for the next generation.” Page, who served as Canada’s first parliamentary budget officer, and at various central agencies during a 27-year federal public service career, called Phoenix a “failure of values.” While they might not always be emphasized, he said, qualities such as integrity, stewardship and excellence comprise a public service “code.”

In his eyes, the negligence of executives charged with the Phoenix project — cancelling a pilot run, pressing forward without an adequate contingency plan — speaks to a deeper disregard for the values that should underscore public service. “It’s not just a few project managers. It cuts right through all the work that’s done by senior managers and central agencies who are responsible to support decision-making and oversight for the executive,” Page said. “There’s going to have to be new leadership in the public service.” While the current cadre of senior executives could ostensibly have a “come-to-Jesus moment” in which a post-Phoenix reckoning leads to commitment to examine these underlying values, Page said he’s not hopeful. Despite the AG’s self-professed “bleak” assessment of its workplace culture, Page said he works with many students who continue to aspire to public service. In fact, he said, it’s possible that this recent bit of bad press could actually spur on those who believe they can be part of a culture and values shift in the bureaucracy. “The time to buy is when the market is low,” Page quipped.

THE EXECUTIVES: It’s time to move forward The head of the association that represents the more than 6,000 executives — typically directors and above — in the federal public service, sees things very differently. “The executive community is as frustrated as any other group with this, and we’re being tarred with a brush with respect to Phoenix,” said Michel Vermette, CEO of the Association of Professional Executives of the Public Service of Canada.

The auditor general made some important recommendations, Vermette said, including the need for project oversight and independent review mechanisms for government-wide IT projects — to which Public Services and Procurement Canada and the Treasury Board of Canada Secretariat have agreed for all such projects, moving forward. Ferguson’s comments about public service culture have also provided a “good reminder” to executives about the nature of their job — to provide advice to ministers, and implement the decisions those ministers ultimately make, according to Vermette. But after two AG reports and an independent study by the Goss Gilroy Inc. consulting firm, Vermette said, it’s time to move forward, implement the audits’ recommendations, and work on fixing the pay problems Phoenix has given rise to. He rejected calls from the Public Service Alliance of Canada for a public inquiry into the project’s failure, and a freeze on executive performance bonuses until employee pay issues are resolved. “That’s damning an entire community of people who are working hard to make sure their staff get paid, but have little control over the system.”

THE PUBLIC SERVANTS: A public inquiry Meanwhile, PSAC national president Chris Aylward says a national public inquiry into Phoenix is the public service’s only hope for changing the culture of “incomprehensible failures” that Ferguson cited. PSAC, the largest union representing federal public-sector workers, will submit a formal request for an inquiry in the coming weeks.

“We need people to be compelled to come and testify under oath to say exactly what had happened,“ Aylward said, describing a culture that often discourages public servants from speaking out about perceived or possible flaws. “I think the broader issue is that workers don’t want to come forward and say, ‘Hey there might be an issue here,’ because they’re afraid of reprisal. “Then, a lot of times, the senior bureaucrats at the deputy level or the assistant deputy level, they don’t want to hear it because they’ve got the political pressure coming down from their ministers and from parliamentarians saying, ‘This has to get done,'” Aylward said.

THE EXPERT: Change the reward structure The subjugation of the bureaucracy by political priorities was among Ferguson’s grim observations about what ails today’s public service. It’s something Christopher Stoney, an associate professor at Carleton University’s School of Public Policy and Administration, has observed. “When it comes to culture, it seems to be very top-down as a hierarchy,” he said. “It cascades down from the political priorities and timing to the managerial, then it goes down to the public service themselves, who are then under pressure to try and meet these deadlines, which may be unrealistic.” With Phoenix, Ferguson pointed out that executives were “more focused on meeting the project budget and timeline than on what the system needed to do,” as evidenced by decisions such as removing pay processing functions from Phoenix, compressing the project schedule and reducing the number of employees assigned to it, rather than asking for more time or money. Stoney said this kind of perversion of priorities is enabled in part by the reward system in the public service. “I would do away with performance bonuses, I think it gets too much tied into what we call goal displacement, so the people start trying to achieve things for the wrong reasons and the public interest gets lost because of self-interest and short-term thinking.” In 2015-16 — the most recent period for which figures are posted by the Treasury Board of Canada Secretariat — more than $75 million was spent on performance pay for nearly 5,500 executives. The value of the Phoenix experience, Stoney said, lies in the insight it has offered into the public service culture. “There are some projects that are so important, so big, and so time-sensitive vis-à-vis elections,” that the cracks start to show, and what lies beneath is laid bare and talked about. “Otherwise how the heck do we know what it is?”

TREASURY BOARD PRESIDENT: Tend to the plumbingThis underlying “plumbing of government,” said Scott Brison, president of the Treasury Board, is what his government has been working to improve since taking office in 2015. “The two shiny objects to which people are attracted in government are usually policy and communications,” the longtime parliamentarian said. The Liberals, according to Brison, are delving deeper.

“We have taken and continue to take concrete actions to strengthen the culture of the public service, and to encourage a culture of experimentation and innovation,” evidenced, he said, by such policies as the “unmuzzling” of government scientists and new standards for government digital projects. According to Brison, the public service needs to look less hierarchical, more agile and innovative in its approach to problem-solving and citizen engagement, and more enticing to millennials. As for why the Phoenix failure was not averted under his government’s tenure — the auditor general held accountable both the previous Harper Conservative government under which Phoenix was first approved, and the current Liberal government for the decision to green light the pay system’s launch — Brison said culture change doesn’t happen overnight. “We have made significant changes in the last couple of years, but we have a lot of work to do.” “We feel actually the auditor general’s report helps reaffirm that we are heading in the right direction.”

Source: Fixing a broken culture: public service in the wake of Phoenix

The Phoenix disaster was predictable – and preventable: Hutton

Problems go deeper than ineffective whistle blower legislation:

Federal Auditor-General Michael Ferguson’s scathing report on the Phoenix pay project reveals that, far from being an accident, this disaster was manufactured: The project was set up for failure from the start.

This process began in 2012 with a big lie – that the project could be completed successfully with about half the budget that the contractor estimated to be necessary. This magic was to be achieved by removing 100 functions from the product (including many that proved later to be vital), slashing the development staff, reducing the testing and compressing the schedule. IT professionals have a saying for this approach: “If the product doesn’t have to work, we can meet any budget and any schedule.”

Once told, this big lie had to be sustained, and the executives in charge of Phoenix boldly and successfully kept this up for years – until the truth couldn’t be hidden any more. They achieved this by seizing control of all the supposedly mandatory oversight mechanisms, such as independent reviews and audits, and keeping their superiors in the dark. Mr. Ferguson’s report states categorically that there was no oversight – zero.

In 2015, shortly before the fateful decision to launch the rollouts, the Phoenix executives received an independent report commissioned by the Treasury Board, which raised a number of serious concerns, including incomplete testing and uncorrected errors, the likelihood of high call volumes, the short timelines and lack of contingency plans.

This report was not even shown to the minister responsible. So the massive rollout went ahead – with only partial function, incomplete test results, no fallback and no contingency plan. The result was entirely predictable.

Although Phoenix executives managed to hobble the normal oversight mechanisms, there was one more mechanism outside of their control that could have prevented this disaster, even though everything else had failed – the federal whistle-blowing system.

In other countries, this type of system has proven highly effective at exposing even massive frauds that powerful vested interests were desperate to conceal.

But the Phoenix executives didn’t have to hobble the whistle-blowing system – the government had already done this for them.

Created in 2007, the system is supposed to work as follows: Honest employees who see evidence of misconduct go to the Integrity Commissioner (who is an officer of Parliament, like the Auditor-General), confident that they will be protected from reprisals; the Integrity Commissioner conducts an independent investigation into these allegations, with all the considerable powers of the Inquiries Act at his disposal; if wrongdoing is found, the Commissioner reports his findings – which are public – directly to Parliament. One can only imagine what such a report would have done to the big lie, and how quickly the axe would have fallen on this troubled project and its perpetrators.

Unfortunately, this could not happen because our federal whistle-blower-protection system simply doesn’t work – and everybody in Ottawa knows it. We have a deeply flawed law, unchanged for 11 years, and a succession of integrity commissioners who, far from being protectors, have proven to be a whistle-blower’s worst nightmare since they rarely investigate anything. And no whistle-blower has ever prevailed at the tribunal that is supposed to compensate them for reprisals they have suffered.

In 2017, the government asked a parliamentary committee to review this system. Shocked at the dysfunction they found, the members unanimously recommended sweeping changes. But government’s response was to give the committee the brush-off – none of the recommendations were implemented.

In general, there’s no lack of honest public servants willing to come forward and expose fiascos such as we just experienced with Phoenix. But under the current system, they would never have a chance.

Since the government seems determined not to help whistle-blowers – even after this debacle – Ryerson’s Centre for Free Expression is doing what civil society has done in other countries: We are going help them ourselves. We are setting up a free confidential advice centre that they can go to for help, which we expect to be staffed and operational by the fall.

We have also decided to conduct our own investigation of the Phoenix project – to find out what happened to those honest employees who tried to blow the whistle, and to learn how others were deterred from speaking out at all.

via The Phoenix disaster was predictable – and preventable – The Globe and Mail

Phoenix Pay: Government got conflicting advice before launching ill-fated system

Don’t think I would recommend S.i. Systems given their candy coating compared to Gartner.

But Bagnall’s conclusion is right: really hard to put on the brakes on a major initiative at a late stage, given bureaucratic inertia and that people are vested in it going forward:

In the wake of last week’s damning report by auditor general Michael Ferguson — who concluded the pay system is at risk of chewing up $540 million more than its budgeted $310 million by 2019, with no end in sight — it’s worth re-examining some of the independent advice government agencies were getting in early 2016.

Treasury Board, along with Phoenix-sponsor Public Services and Procurement Canada, commissioned at least two reviews that were delivered just days before the February 2016 launch of the new pay system.

One review, by Gartner Inc., offered a number of important warnings, but the second report, by S.i. Systems, was surprisingly upbeat about the Phoenix project’s chances for success.

“The (Phoenix) initiative is very likely to achieve its goals and desired outcomes within the first year or two of full operations,” S.i. Systems noted in its draft final report dated Jan. 18, 2016. “All in-scope work has been completed, a (software) code freeze has been imposed on Phoenix and the Miramichi pay centre is fully operational.”

Ferguson last week gave short shrift to such sentiment, pointing out that roughly one in two federal government employees was experiencing a significant pay issue as of last June — fully 16 months after the launch of Phoenix.

S.i. Systems couched some of its conclusions with caveats, noting that the system was not yet fully automated, with the result some pay transactions were being dealt with manually. However, the consultants viewed this as a “temporary” issue during the transition from dozens of older pay systems to the consolidated Phoenix system.

S.i. Systems nevertheless was clear that Public Services and Procurement Canada — the department in charge of the project — should move ahead with Phoenix. Such a move “will be challenging,” the S.i. Systems report noted, “but it is likely that the problems and difficulties will be manageable.”

The consultants concluded “The (Phoenix) project team is to be commended for bringing this complex initiative to its current stage.”

The Gartner report, dated Feb. 11, 2016, offered a much different view. Not only did Gartner identify a dozen significant risks facing the impending rollout of Phoenix, it offered strategies for minimizing them. Many of the risks proved all too real, while the tips for reducing them were ignored.

Consider this item, offered in a discussion of potential problems associated with testing the new pay system: “End to end testing has not been performed by any department that Gartner has interviewed,” Gartner noted, “Best practice would dictate multiple end-to-end cycles be tested prior to go-live (in February 2016).”

The Gartner document added that its consultants were never provided with “a clearly documented testing strategy and plan.”

Gartner was hired on Dec. 21, 2015, leaving it just enough time to interview eight federal departments. Nevertheless, the sample included some of the largest ones (Health Canada, Employment and Social Development Canada and Public Services).

Other key risks identified by Gartner included training, support and transition.

For instance, Gartner notes that federal departments hadn’t yet implemented their training programs. This meant that if any gaps in training emerged it would be impossible to address these through revised or remedial courses before Phoenix went live. Gartner concluded the training shortfall could result in “unanticipated consequences such as an incorrect pay calculation.”

Gartner also brought attention to what has proved one of Phoenix’s most intractable problems — technical support for government employees using the system, a problem exacerbated by the reduction in the number of pay administrators starting in 2014.

Gartner correctly predicted there would be a very large number of queries facing pay administrators at the central location in Miramichi, N.B. — not least because employees across government had little opportunity beforehand to become familiar with Phoenix’s many quirks.

The consultants offered a number of suggestions for reducing the risks of the Phoenix rollout, including trying a more piecemeal approach. Divide the two main waves of employees into multiple waves, for instance, and start with the least difficult departments — those with relatively few seasonal employees, shift workers and other complicating features when it comes to pay.

Critically, Gartner also suggested running Phoenix in tandem with the older pay system as a contingency in case the new system didn’t perform as advertised.

These and other recommendations were ignored, with the result now all too plain to see. Nearly 350,000 pay transactions are today choking a system designed to accommodate 80,000.

To be fair, S.i. Systems also took note of the potential risks involved in abandoning the old pay system before making sure Phoenix actually worked. “(We) did not see evidence of a fallback or test strategy to mitigate this potentially risky event,” the S.i. Systems report noted in an Annex.

But the consultants downplayed the risk in its summary assessment that declared the Phoenix project was using an “excellent testing strategy” and that “when problems were encountered, appropriate and timely action was taken.”

But no matter the consultants’ advice, the final call about moving ahead with a project this big belonged to government. After nearly a decade in development, Phoenix suffered the flaw of unstoppable bureaucratic momentum. The directors of the project seemed not inclined to pay much attention to last-minute advice unless it happened to line up with where they were going anyway.

via Phoenix Pay: Government got conflicting advice before launching ill-fated system | Ottawa Citizen

Bagnall: Phoenix — a disaster so bad, it just might spark real change

Good analysis by Bagnall:

While software experts sort through the technical problems, the government has gone on a hiring binge so Public Services and other federal departments can begin to make a dent in pay requests that require manual processing. The irony, of course, is that the Conservatives, starting in 2014, had reduced the number of compensation advisers across government from 2,000 to 1,350 in anticipation of a more efficient system.

The government today now employs more compensation advisers and support staff than it did prior to the launch of Phoenix — counting the employees within departments who have recently been reassigned — temporarily, one hopes — to handle pay issues.

Such is the “all-hands-on-deck” sentiment throughout government that even the public service unions are contemplating agreeing to simpler language in dozens of collective agreements.

They have a number of incentives to make changes in contract language, even if it potentially eliminates certain types of overtime pay or complex provisions relating to paid leave. Labour leaders have been inundated with demands from members to help get Phoenix fixed. Few blame the unions for the broken system but blame is no longer the point.

Phoenix has been in crisis so long that government employees are now making career decisions stemming from their fear of payroll consequences. If they transfer to another department, retire, go on paternity leave or accept a raise, will they be able to weather a temporary loss of pay?

There are other knock-on effects of Phoenix, most notably in other departments that are also managing complex information technology projects. Earlier this year, the Department of National Defence cancelled a procurement that would have linked part of its pay system to Phoenix. Departments are also evaluating whether they should manage IT projects in a different manner altogether.

The federal government has a long history of top-down management — intricate, massive designs that try to anticipate every contingency. But by the time all aspects are locked in, the world of technology has moved on.

In the case of Phoenix, project managers seemed to understand the risks and potential complications of the system they were proposing — but at a theoretical level. The auditor general made it clear that very little was ready when the system was launched at a practical level — not the software, the processes or the oversight. When the system early on began sounding warning sirens, those managing Phoenix didn’t know how to ask the right questions to establish a fix.

This knowledge gap existed because Public Services and Procurement Canada tried to do everything itself.

Ferguson concluded his report by urging Public Services and Treasury Board — the federal government’s main employer — to develop a sustainable repair for Phoenix based on a fuller understanding of the system’s underlying flaws.

Ideally such a fix would address the culture that produces such IT disasters. There’s too little direct experience in IT, too much fear of making errors that embarrass cabinet members and top brass in the department, too little feedback from the rest of government.

Getting all this right might mean we’d never have to read another report as damning as the one Ferguson delivered Tuesday.

via Bagnall: Phoenix — a disaster so bad, it just might spark real change | Ottawa Citizen

Feds underestimated complexity, ignored concerns about Phoenix pay system, review finds

Pretty damning account of senior management failures:

An independent report on the problem-plagued Phoenix pay system rollout released Thursday says the government underestimated its complexity.

Public Services and Procurement Minister Carla Qualtrough (Delta, B.C.) apologized to affected public servants in an Oct. 5 press conference, and said the review confirmed many of the thoughts she already had. She noted fixing the pay system is the first priority in her mandate letter from the prime minister, given to her after he moved her into the job in August.

“The implementation of such a complex business transformation across the entire government of Canada was a massive undertaking that I believe history will record was set up to fail,” she said, noting there was poor planning on almost every key aspect. “There’s no easy or quick fix for the problems in the pay system.”

The report’s “lessons learned” included: assigning accountability and authority to a single office, properly defining the scope of the project, fully testing the software before launch, and not expecting savings until well after implementation.

Overall, the independent consultants hired to do the review found the officials working on the project were most concerned with being on time and on budget. Briefings were usually positive and the culture of the department responsible prevented people from speaking negatively about the project. The report said bad news was usually buried, with concerns mostly ignored. Even at the pay system’s launch in February 2016, testing was incomplete, with no planned fix date and no contingency plan.

Other lessons included making sure to have sufficient employee knowledge capacity, and communicating effectively. The report noted workplace culture was also important, and that the 17 lessons it outlined “are yet to be learned” by the government.

Though work on the new pay system began under the previous Conservative government, the Liberals launched it in February 2016. It was supposed to consolidate the payroll of over 300,000 public service employees, but instead it has left many of them overpaid, underpaid, or not paid at all. Radio-Canada recently reported that as of Aug. 8 nearly half of the 313,734 federal public servants paid through Phoenix had been waiting at least a month to have their complaints dealt with. The government contracted IBM to configure the software for the government payroll system.

The program was supposed to save the government $70-million per year, but so far the Liberals have sunk in about $400-million to fix it.

Source: Feds underestimated complexity, ignored concerns about Phoenix pay system, review finds – The Hill Times – The Hill Times

And good background to some of the potential conflicts of interest with some of companies doing background studies and recommendations, and working on implementation:

Two companies that were awarded tens of millions of dollars in contracts to help create the new federal pay system played a part in recommending the Phoenix project in the first place, CBC News has learned.

This appearance of a conflict of interest in the very early days of the project is raising flags for those who monitor federal procurement and accountability.

The internal government report that recommended a new pay system in 2009 relied heavily on two studies — one from IBM and another from PricewaterhouseCoopers. These were two of the companies that went on to help develop Phoenix for a combined price tag of more than $200 million — and counting.

“That’s a cause for concern,” said Christopher Stoney, who follows procurement and accountability issues as a professor at the School of Public Policy and Administration at Carleton University. “[It’s] clearly getting into areas here of conflict of interest.”

The recommendation

In May 2009, top government bureaucrats delivered a document called Initiative to Fix the Pay System Business Plan, which convinced decision makers to bring in a new, modern pay system. That system would later be called Phoenix.

A year earlier, international professional services firm PricewaterhouseCoopers published a report called Analysis of Industry and Government of Canada Pay Administration Services Delivery Model Options. The PricewaterhouseCooper’s study has not been made public, but it is quoted liberally throughout the Phoenix business plan.

The PwC study suggested the most cost effective choice for the Government of Canada would be to create a consolidated pay centre and use customized off-the-shelf pay software. It was advice the government embraced.

PricewaterhouseCoopers also reviewed the business case as, in the government’s words, “an independent Third Party to ensure unbiased and accurate content.”

The company has earned $17.4 million through its work on the Phoenix system since 2009, according to the federal government database

“I think anybody looking at this would be concerned that there was this possible undermining of independence,” said Stoney, who co-edits the annual publication How Ottawa Spends. “At some point it seems as though that independence was eroded and [PwC] became increasingly key players in this.”

IBM’s role

The government’s Initiative to Fix the Pay System Business Case also depended on research provided by IBM in a February 2007 report called Pay Benchmarking Study for the Government of Canada.

In September, CBC reported details about IBM Canada’s extensive responsibilities to design, implement, operate and fix the Phoenix system, for a price tag that has so far reached $185 million.

The 2007 IBM study pointed out that custom, off-the-shelf software systems are “consistently more cost effective and enable higher quality and efficiency, when implemented and sustained properly.”

This was exactly the kind of system that IBM was hired to implement for Phoenix.

IBM also noted the government’s old system was at risk of failure. Its report warned that these “payroll errors can have significant consequences for both the financial picture of the organization and talent retention.”

‘Brought into the tent’

“It’s clear that two of the contractors that played a role on the build and operations played a role in the business case,” said Kevin Page, president of the Institute of Fiscal Studies and Democracy at the University of Ottawa.

“So we literally brought them into the tent, asked them to see whether or not the business case was strong, asked them to do benchmarking, which is critical to performance of a business case.”

Page was appointed Canada’s first Parliamentary Budget Officer in 2008 and was in that role when this business plan was developed. But he had never seen the business case until CBC sent it to him recently.

Someone should take the fall for Ottawa’s botched Phoenix pay system

Hard to disagree with Barrie McKenna: the lack of accountability at both the political and bureaucratic level, the inability of government to manage large-scale IT projects and the miss match between those who “sold” the project and those responsible for delivery are of broad concern, not just in the case of Phoenix.

IT in government is complex given the myriad of requirements and groups involved.

My experience with IT in government is a mixed bag. My most successful project, done with a small group of PCO policy types, was the creation of an Access database to manage the then Chrétien government annual priority setting exercise. Delivered on time, it worked  and ensured consistent tracking rather than the previous time-consuming and error prone Word-based process.

My second experience, also at PCO, but on a larger scale (more data, more users) and thus involving IT folks, was replacing the previous Cabinet meeting planning system with a more up-to-date platform with more flexibility. The IT folks and the consultant never got it to work during my time there.

Lastly, at Service Canada, we partnered with Service New Brunswick to deliver, on Transport Canada’s behalf, a system for pleasure craft licensing. When it went live, it crashed but we were able to identify and fix the problem within a few days (the data link capacity was too small, something missed in all the preparations, by all involved). After that it worked well (Service Canada eventually decided to end its involvement and focus on core services to ESDC):

The mess that is Phoenix is a story of misguided political objectives, bungled management of a major technology project and a complete failure by anyone in charge to take responsibility for mistakes.

The fiasco raises troubling questions about the government’s ability to perform one of its most basic functions – paying its bills and taking care of employees. The Phoenix system is just one of the major information-technology projects, totalling billions of dollars, now under way in the government.

Centralizing the multitude of separate payroll systems was the brainchild of Stephen Harper’s Conservative government, which was convinced it could wring huge savings out of the bureaucracy. In charge were then-public works minister Rona Ambrose (now interim Conservative leader) and Tony Clement, former president of the treasury board. Neither has expressed any remorse for the fiasco.

The Conservatives eliminated 700 payroll jobs in dozens of departments, mainly in Ottawa, and created a new centralized pay centre in Miramichi, N.B. – political compensation for the shuttered gun registry. Most of those offered positions there refused to move, leaving the running of Phoenix in the hands of hundreds of untrained new hires.

The problem now belongs to the Liberal government, which could have delayed deployment of the system to work out the inevitable bugs. To his credit, Prime Minister Justin Trudeau has acknowledged his government initially didn’t take the problem seriously.

“I’ll admit it,” Mr. Trudeau told a frustrated civil servant at a town hall in Kingston, Ont., earlier this year. “This government … didn’t pay enough attention to the challenges and the warning signs on the transition we were overseeing.”

But the mea culpa came three months after the government had promised to resolve the payroll mess. Now, it’s not even offering a target.

Just as troubling is the lack of accountability within the upper ranks of civil servants. Many of those responsible have retired or moved to other jobs in the government. No one has been fired.

Nor has there been a thorough investigation by Parliament of what went wrong. Deputy Minister of Public Works Marie Lemay, who inherited the payroll problem, appeared before the House of Commons government operations committee last year. But none of the original architects of the system have had to answer for their roles.

And then there is IBM Canada, which Ottawa hired to design and implement the system. It appears the government, not IBM, is on the hook for fixing the problems. So why, one wonders, would the government sign a contract that left it so dangerously exposed to financial and technical risk?

Phoenix was supposed to save Ottawa $70-million a year. Instead, the government has spent $50-million fixing the problem, including an extra $6-million paid to IBM, and there is no end in sight.

This isn’t just a story of a botched payroll system. It’s about the chronic inability of governments to manage major purchases, including technology projects.

Unless Ottawa gets to the bottom of what went wrong on Phoenix, it will keep making the same mistakes elsewhere in the government.

That should worry all taxpayers, not just government workers.

Source: Someone should take the fall for Ottawa’s botched Phoenix pay system – The Globe and Mail

Shared Services Canada, already having resulted in the resignation of the Chief Statistician (INSERT), now comes under fire from the RCMP:

CBC News has obtained a blistering Jan. 20, 2017, memo to Public Safety Minister Ralph Goodale in which Commissioner Bob Paulson details how critical IT failures have increased by 129 per cent since the beleaguered department took over tech support for the entire government five years ago.

Not only that, the memo says, the duration of each outage has increased by 98 per cent.

“Its ‘one size fits all’ IT shared services model has negatively impacted police operations, public and officer safety and the integrity of the criminal justice system,” reads the memo.

The document appears to respond to a request for more information after a series of CBC News reports on the RCMP’s long-standing dissatisfaction with Shared Services Canada (SSC).

Despite the agency’s creation of special teams and committees to address shoddy service and repeated computer outages, Paulson said minimal progress has been made.

The commissioner bolstered his arguments by enclosing an appendix of recent critical incidents to show just how little appreciation or understanding there is for operational law enforcement requirements.

RCMP commissioner warns continued IT failures will have ‘catastrophic’ consequences

No excuse for Ottawa’s bungled technology: Barrie McKenna

One of the rare commentaries that connects the dots between Shared Services Canada, the Phoenix pay system, and the inability of government to manage complex IT projects (admittedly, some of the most complex around).

It does beg the question, as posed by Donald Savoie in his book, What Is Government Good At?: A Canadian Answer.

One also has to ask the question, in all the decks, analyses, MCs and TB submissions, were the risks clearly stated and assessed? Did the public servants provide ‘fearless advice’ or not?

Maybe you don’t think it’s a big deal that tens of thousands of federal government workers are going unpaid because of the botched roll out of a new pay system.

Most civil servants are overpaid and underworked anyway, right?

Many Canadians may feel similarly untroubled that government data centres are frequently crashing, downing websites and leaving key agencies, such as Statistics Canada, unable to get timely economic information to financial markets.

But it does matter. Canada isn’t some tin-pot country that can’t pay its workers, run a computer or produce timely data. It’s a G7 country, a modern, advanced economy that should be a model of good governance.

There is a disturbing back story to these embarrassing headlines.

Turn back the clock to 2010. Stephen Harper’s Conservative government was eager to demonstrate it could wring billions of dollars in savings out of a fat government bureaucracy it neither liked nor trusted.

Two of the signature initiatives that emerged from this effort was the centralized Phoenix pay system and the birth of Shared Services Canada, a $1.9-billion super agency that would consolidate all of the government technology systems.

And for years afterward it would point to these efforts to bolster its reputation as a sound manager of the machinery of government.

Both have been unmitigated disasters.

The fallout from these moves continues to reverberate through the government. Not only have the promised savings never materialized, but Ottawa is now spending tens of millions more to fix the problems.

On Friday, Statscan’s chief statistician, Wayne Smith, abruptly resigned, complaining that the agency’s independence has been compromised by “disruptive, ineffective, slow and unaffordable” technology supplied by Shared Services Canada.

Mr. Smith’s frustrations boiled over July 8 when the agency’s main website was down for nearly eight hours due to a power switch failure, snarling the release of June’s jobs numbers, one of the country’s most important economic indicators. Statscan staff resorted to snapping the document on a smartphone and faxing pages to data users at financial institutions and media outlets.

Statscan’s website routinely goes down on busy data-release days.

The problems at Shared Services, which consolidated the information technology of 43 departments, go way beyond Statscan. The federal Auditor-General concluded in a report this year that Shared Services’ operations are so dogged by hidden costs, delays, security problems and poor accounting that potential savings remain “largely unknown.”

The Liberals quietly boosted Shared Services’ budget by $384-million over two years in its March budget, in part to keep creaky old computer systems from crashing. Critics worry that much more will be needed to fully modernize systems.

In late July, smoke inside a federal data centre in Ottawa forced the temporary shutdown of government e-mail and some websites.

Meanwhile, Ottawa says the estimated bill to fix IBM’s Phoenix pay system has reached $45-million to $50-million, and could climb higher. The government has promised to cover any out-of-pocket expenses of workers who couldn’t pay bills or were forced to borrow money when they weren’t paid.

Just like Shared Services, Phoenix was supposed to save the government money – $70-million a year – by consolidating a myriad of pay systems spanning 300,000 workers in more than 100 departments. Most of the first-year savings have now been wiped out.

A big part of the problem can be traced to a decision by the Conservatives to create a new payroll-processing centre in Miramichi, N.B. Roughly 500 – mostly inexperienced – new hires, would replace more than 2,000 payroll staff from across the country.

Ottawa has since been forced to add pay specialists in Gatineau, Que., and at temporary offices in Winnipeg, Montreal, Toronto and Sherbrooke, Que. – all to help fix the problem of workers getting paid too much, not enough or not at all.

Efficiency was never the main reason for choosing Miramichi. Putting the payroll centre in the city was political compensation for the closing of the long-gun registry, which had been located there.

The Conservatives fed the country a narrative about making government leaner and more efficient.

They delivered something quite different.

Source: No excuse for Ottawa’s bungled technology – The Globe and Mail