ICYMI – Bagnall: Could cloud services signal the end of ‘big zombie IT projects’ in government?

Still will face the same management challenges as conventional IT and the particular difficulties governments have in fixing and sticking to specifications. But certainly worth exploring:

As federal government announcements go, this one could have been a real snooze-fest.

Treasury Board President Scott Brison and Carla Qualtrough on Wednesday jointly unveiled a new information technology policy that had been in force for months, involving a once-obscure branch of technology called cloud services.

But their short show-and-tell, delivered on Facebook Live, hinted at something more profound taking place.

Brison in particular has accepted the idea that big government has to change the way it builds and manages its IT infrastructure — and that cloud services, which allow departments to lease computer capacity a bit at a time from private sector firms such as Amazon Web Services or Microsoft Azure — offer the way to do it.

“We can’t be a Blockbuster government when we’re serving a Netflix citizenry,” he said in comparing a defunct video rental business with a video-streaming company.

Brison appeared to be taking aim in part at Shared Services Canada, the government’s central computer services agency, which reports to Qualtrough. Since its formation in 2011, Shared Services has invested hundreds of millions of dollars in new data centres to house information and software that underpins programs ranging from the Canada Pension Plan to Statistics Canada’s census.

While the data centres are fresh and modern, Shared Services hasn’t impressed many federal departments because it’s been slow and inflexible in setting up new online services and ordering the new hardware.

Over the past year, smaller departments have forced the issue by quietly running their own pilot projects using Azure and other cloud providers. Private contractors now have dozens of cloud-based IT procurements in the works, such as applications designed to make scientific or business data available to the public.

Brison, whose department sets the overall policy for government IT, has reportedly been impressed with what cloud technology can do. On Wednesday he enumerated key benefits, such as how departments using cloud services can experiment with software applications a bit at a time, learning from the inevitable mistakes along the way.

“It’s better to learn the lessons early,” he said in an apparent reference to IT disasters such as the botched rollout of the Phoenix Pay system, “than to have big zombie IT projects rumbling on, trapped under the tyranny of sunk costs.”

How would Phoenix have developed in a cloud-based world? We could actually have an opportunity to find out if Qualtrough opts to restart the entire project. Nothing on that prospect Wednesday, though.

Of course, it’s very early days in the cloud services revolution. Qualtrough, who also seemed very much on board, noted her department had negotiated 22 contracts to date with companies that are selling cloud services to seven government departments and agencies, including Correctional Service Canada.

However, the value of these contracts — which are brokered by Shared Services in exchange for a fee — barely tops $2 million. This is a tiny fraction of Shared Services’ annual budget of more than $1.5 billion.

And there’s the other matter of security. Most of the government’s data is secret (a Protected B or higher classification), and Shared Services still has a monopoly over storing this information. Wednesday’s announcement was for unclassified stuff such as government websites that are to be viewed by the public.

Private contractors are suspicious that Shared Services is relying on security designations to retain its share of the government’s IT business.

It’s not clear how long its monopoly will last. Qualtrough noted the government is mulling further changes that would allow departments “in the future” to store even secret data in the cloud — one of the reasons tech giants such as Google and Amazon have been adding data centres in Canada.

This much is clear: providers of cloud services have secured their foothold in government. If they deliver as promised, this could be the beginning of the end of monster IT failures. There’ll be many small ones, to be sure. But the egregious example of Phoenix Pay has taught us that’s a much better way to run.

via Bagnall: Could cloud services signal the end of ‘big zombie IT projects’ in government? | Ottawa Citizen

Phoenix Pay: Government got conflicting advice before launching ill-fated system

Don’t think I would recommend S.i. Systems given their candy coating compared to Gartner.

But Bagnall’s conclusion is right: really hard to put on the brakes on a major initiative at a late stage, given bureaucratic inertia and that people are vested in it going forward:

In the wake of last week’s damning report by auditor general Michael Ferguson — who concluded the pay system is at risk of chewing up $540 million more than its budgeted $310 million by 2019, with no end in sight — it’s worth re-examining some of the independent advice government agencies were getting in early 2016.

Treasury Board, along with Phoenix-sponsor Public Services and Procurement Canada, commissioned at least two reviews that were delivered just days before the February 2016 launch of the new pay system.

One review, by Gartner Inc., offered a number of important warnings, but the second report, by S.i. Systems, was surprisingly upbeat about the Phoenix project’s chances for success.

“The (Phoenix) initiative is very likely to achieve its goals and desired outcomes within the first year or two of full operations,” S.i. Systems noted in its draft final report dated Jan. 18, 2016. “All in-scope work has been completed, a (software) code freeze has been imposed on Phoenix and the Miramichi pay centre is fully operational.”

Ferguson last week gave short shrift to such sentiment, pointing out that roughly one in two federal government employees was experiencing a significant pay issue as of last June — fully 16 months after the launch of Phoenix.

S.i. Systems couched some of its conclusions with caveats, noting that the system was not yet fully automated, with the result some pay transactions were being dealt with manually. However, the consultants viewed this as a “temporary” issue during the transition from dozens of older pay systems to the consolidated Phoenix system.

S.i. Systems nevertheless was clear that Public Services and Procurement Canada — the department in charge of the project — should move ahead with Phoenix. Such a move “will be challenging,” the S.i. Systems report noted, “but it is likely that the problems and difficulties will be manageable.”

The consultants concluded “The (Phoenix) project team is to be commended for bringing this complex initiative to its current stage.”

The Gartner report, dated Feb. 11, 2016, offered a much different view. Not only did Gartner identify a dozen significant risks facing the impending rollout of Phoenix, it offered strategies for minimizing them. Many of the risks proved all too real, while the tips for reducing them were ignored.

Consider this item, offered in a discussion of potential problems associated with testing the new pay system: “End to end testing has not been performed by any department that Gartner has interviewed,” Gartner noted, “Best practice would dictate multiple end-to-end cycles be tested prior to go-live (in February 2016).”

The Gartner document added that its consultants were never provided with “a clearly documented testing strategy and plan.”

Gartner was hired on Dec. 21, 2015, leaving it just enough time to interview eight federal departments. Nevertheless, the sample included some of the largest ones (Health Canada, Employment and Social Development Canada and Public Services).

Other key risks identified by Gartner included training, support and transition.

For instance, Gartner notes that federal departments hadn’t yet implemented their training programs. This meant that if any gaps in training emerged it would be impossible to address these through revised or remedial courses before Phoenix went live. Gartner concluded the training shortfall could result in “unanticipated consequences such as an incorrect pay calculation.”

Gartner also brought attention to what has proved one of Phoenix’s most intractable problems — technical support for government employees using the system, a problem exacerbated by the reduction in the number of pay administrators starting in 2014.

Gartner correctly predicted there would be a very large number of queries facing pay administrators at the central location in Miramichi, N.B. — not least because employees across government had little opportunity beforehand to become familiar with Phoenix’s many quirks.

The consultants offered a number of suggestions for reducing the risks of the Phoenix rollout, including trying a more piecemeal approach. Divide the two main waves of employees into multiple waves, for instance, and start with the least difficult departments — those with relatively few seasonal employees, shift workers and other complicating features when it comes to pay.

Critically, Gartner also suggested running Phoenix in tandem with the older pay system as a contingency in case the new system didn’t perform as advertised.

These and other recommendations were ignored, with the result now all too plain to see. Nearly 350,000 pay transactions are today choking a system designed to accommodate 80,000.

To be fair, S.i. Systems also took note of the potential risks involved in abandoning the old pay system before making sure Phoenix actually worked. “(We) did not see evidence of a fallback or test strategy to mitigate this potentially risky event,” the S.i. Systems report noted in an Annex.

But the consultants downplayed the risk in its summary assessment that declared the Phoenix project was using an “excellent testing strategy” and that “when problems were encountered, appropriate and timely action was taken.”

But no matter the consultants’ advice, the final call about moving ahead with a project this big belonged to government. After nearly a decade in development, Phoenix suffered the flaw of unstoppable bureaucratic momentum. The directors of the project seemed not inclined to pay much attention to last-minute advice unless it happened to line up with where they were going anyway.

via Phoenix Pay: Government got conflicting advice before launching ill-fated system | Ottawa Citizen

Bagnall: Phoenix — a disaster so bad, it just might spark real change

Good analysis by Bagnall:

While software experts sort through the technical problems, the government has gone on a hiring binge so Public Services and other federal departments can begin to make a dent in pay requests that require manual processing. The irony, of course, is that the Conservatives, starting in 2014, had reduced the number of compensation advisers across government from 2,000 to 1,350 in anticipation of a more efficient system.

The government today now employs more compensation advisers and support staff than it did prior to the launch of Phoenix — counting the employees within departments who have recently been reassigned — temporarily, one hopes — to handle pay issues.

Such is the “all-hands-on-deck” sentiment throughout government that even the public service unions are contemplating agreeing to simpler language in dozens of collective agreements.

They have a number of incentives to make changes in contract language, even if it potentially eliminates certain types of overtime pay or complex provisions relating to paid leave. Labour leaders have been inundated with demands from members to help get Phoenix fixed. Few blame the unions for the broken system but blame is no longer the point.

Phoenix has been in crisis so long that government employees are now making career decisions stemming from their fear of payroll consequences. If they transfer to another department, retire, go on paternity leave or accept a raise, will they be able to weather a temporary loss of pay?

There are other knock-on effects of Phoenix, most notably in other departments that are also managing complex information technology projects. Earlier this year, the Department of National Defence cancelled a procurement that would have linked part of its pay system to Phoenix. Departments are also evaluating whether they should manage IT projects in a different manner altogether.

The federal government has a long history of top-down management — intricate, massive designs that try to anticipate every contingency. But by the time all aspects are locked in, the world of technology has moved on.

In the case of Phoenix, project managers seemed to understand the risks and potential complications of the system they were proposing — but at a theoretical level. The auditor general made it clear that very little was ready when the system was launched at a practical level — not the software, the processes or the oversight. When the system early on began sounding warning sirens, those managing Phoenix didn’t know how to ask the right questions to establish a fix.

This knowledge gap existed because Public Services and Procurement Canada tried to do everything itself.

Ferguson concluded his report by urging Public Services and Treasury Board — the federal government’s main employer — to develop a sustainable repair for Phoenix based on a fuller understanding of the system’s underlying flaws.

Ideally such a fix would address the culture that produces such IT disasters. There’s too little direct experience in IT, too much fear of making errors that embarrass cabinet members and top brass in the department, too little feedback from the rest of government.

Getting all this right might mean we’d never have to read another report as damning as the one Ferguson delivered Tuesday.

via Bagnall: Phoenix — a disaster so bad, it just might spark real change | Ottawa Citizen

Sweating the details at Shared Services: What it will take to reset it

Good article capturing some of the major differences between the public and private sector, and why large-scale IT projects are so hard to do well in the former:

It’s not just that its top mandarins lack knowledge and interest in IT. It’s that the entire procurement system and its political overseers suffocate rather than expedite the rollout of large IT projects.

This is messy stuff — software underpinning data centres and telecommunications networks evolves constantly. Upgrading applications across dozens of federal departments inevitably produces conflicts. Programmers and their managers must be free to resolve them — and to drop approaches that aren’t working. The job demands constant testing and feedback at a very micro level.

Shared Services’ first chief operating officer, Grant Westcott, had nearly four decades of experience in government and the private sector — where he was instrumental in consolidating IT systems at the Canadian Imperial Bank of Commerce. But at Shared Services, nearly every move he made was constrained.

At CIBC, Westcott would have been given a budget, a mandate and left alone to get on with it. Had his projects been late and run over budget, it’s unlikely he would have lasted there nearly a decade. In the event, Westcott and his team streamlined the bank’s telecommunications systems and collapsed 22 data centres into just two, trimming CIBC operating costs significantly.

However, the federal government doesn’t allow for this sort of flexibility. Procurement documents contain page after page of technical requirements for programmers and IT consultants. The projects are over-engineered, in other words, in a usually forlorn effort to mitigate most conceivable risks.

Budgets and timelines are spelled out in meticulous detail — even though relatively little is known during the earliest stages about how projects will actually progress. And, of course, there is often extensive cabinet oversight of projects that are costly, late or affect government websites. Which is to say, most of them.

According to experts hired to do these projects, what is needed are wins — IT projects that succeed. And the best way to make these happen is to start with small steps — manageable projects or parts of projects that work. The more of these that Shared Services can string together, the more other federal departments will be willing to let it handle.

This would also make things much easier for Shared Services president Ron Parker — instead of continually revising deadlines for his agency’s main projects, he would be able to point to services actually being performed. Far more satisfying — assuming his people can get things done.

Source: Sweating the details at Shared Services: What it will take to reset it | Ottawa Citizen

Shared Services Canada: How politics sabotaged the government’s grand IT plans | Ottawa Citizen

A good long read by James Bagnall regarding Shared Services Canada and the failure of officials and politicians to anticipate, understand and manage the risks involved. Sobering read:

But Shared Services and Phoenix have something in common — a botched introduction caused, it appears, by deep flaws in how government operates. In both cases, cabinet ministers and bureaucrats underestimated complexity and risk. In this, they were hardly unique — it was the scale of the misjudgment that set the federal IT agenda apart.

Standish Group, a Boston-based consulting firm, has been tracking the performance of IT projects since the mid-1990s — with surprisingly little variation in results. The consultants last year examined 50,000 projects worldwide, including government and private sector. About 30 per cent of these efforts succeeded — that is, they were on time, on budget and produced a payoff. Roughly half the projects ran into difficulty and nearly 20 per cent failed outright. The larger and more complex the project, the higher the rate of failure.

Carol Bellringer, the auditor general for the B.C. government, last month offered three key reasons why IT projects fail: Government departments, she said, lack in-house expertise; they attempt “overly ambitious” programs; they justify the latter through “incomplete” business cases.

All three elements were present at the launch of Shared Services. Most of the responsible bureaucrats were not trained in IT, yet were tasked with remaking on the country’s electronic infrastructure. Many also lacked experience in project management with a heavy IT component.

“I don’t know how many times I heard from deputy ministers that they didn’t understand information technology,” said a senior Shared Services official, “They didn’t like IT and they hoped never to see anything to do with IT for the rest of their career.”

Yet it is a group of deputy ministers — the ones in charge of the most IT-intensive departments — who determine the shape and scope of large IT projects. And when it came to launching Shared Services — the centrepiece of the government’s online renewal — the already high risks were exacerbated by a political agenda that stripped it of the capital necessary to get the job done.

It will likely end up costing taxpayers a fortune to set things right again.

A government data centre in Ottawa.
Detail from one of the many legacy data centres in the National Capital Region.JULIE OLIVER /  POSTMEDIA

It had seemed so simple in the beginning. The idea for Shared Services emerged from the Conservatives’ fifth budget, tabled March 4, 2010. The themes were clear: The economic recession was over; it was time to regain control of government spending.

One aspect of the strategy — little noticed at the time — was the launch of a “comprehensive review” of government spending on administration and overhead expenses. This should have offered easy pickings: Federal government employment was near high tide; and most departments and agencies had expanded rapidly.

Daniel Jean, the deputy secretary to the cabinet of the Privy Council Office, was picked to run the review, making it a big deal. The PCO is home to 950 bureaucrats who provide advice to cabinet and the Prime Minister’s Office, and oversee the development of the civil service.

Jean reported directly to Wayne Wouters (pronounced “Waters”) — the clerk of the privy council and the government’s top bureaucrat. Among the members of the review helping out Jean were Benoit Long, a senior manager seconded from the office of the government’s chief technology officer, and Liseanne Forand, then chief operating officer of Service Canada. The latter department offers Canadians online access to pensions and employment insurance.

The administrative services review was carried out in secret, typical PCO modus operandi. Its members roamed the bureaucracy, collecting information and searching for ways to consolidate or standardize how things were done. Some departments were already moving down this path.

…Information technology offered an even richer vein of potential savings. For half a century, computer networks and software applications had multiplied willy-nilly as individual departments and agencies looked after their own needs. The result was a patchwork of incompatible, higher-cost systems. Standardizing common, basic technologies such as email, data storage and telecommunications seemed logical.

It had been tried before. But attempts to centralize the buying of high-tech gear and services had failed, largely because federal departments were allowed to opt out. Most did so. They did want to give up control of their IT networks to a central agency.

The PCO determined this time would be different. The prime minister had the authority to create a new federal department through a simple cabinet approval known as an order-in-council. Most departments, including a reluctant Canada Revenue Agency and Department of National Defence, would be forced to carve out a significant portion of their IT groups and budgets — about 40 per cent on average — and hand them over to Shared Services.

Crucially, the move would not be subject to scrutiny by Parliament. And so Shared Services was born on Aug. 3, 2011.

Speed was demanded of the agency from the start. Minutes of meetings involving senior Shared Services staff are studded with references to “tight schedules” and the “urgency” of getting projects done.

Part of that had to do with the sheer age of the government’s infrastructure. The hardware was in danger of breaking down and the underlying software for many applications was so old that suppliers such as Microsoft, PeopleSoft and Adobe had stopped supporting it.

The faster Shared Services could install new networks, the less money it would be forced to throw at solving the problems caused by older technology.

But that wasn’t the only reason Shared Services was pressed for time. Senior Shared Services officials said the Conservatives were eager to see cost savings, and impressed upon them the importance of securing an early win.

The PCO framed the upgrade in simple terms: Consolidate, modernize and reap the savings. And Shared Services would have nearly a decade to get it done. By 2020, the thinking went, the government of Canada would be able to offer its citizens secure, online services that would be the envy of the world; and Shared Services would be a magnet for attracting the best and the brightest employees in government.

But cabinet — and to some extent the PCO — failed to account for the complexity. They were proposing to create a new organization using bits and pieces from other departments and loading it up with a series of mandates on a tight schedule. The entire production was fraught with risk.

“They had articulated the problem and come up with an organizational response (in the form of Shared Services),” an independent adviser to the PCO said in an interview, “but they completely underestimated the scale.”

Source: Shared Services Canada: How politics sabotaged the government’s grand IT plans | Ottawa Citizen

Back to the beginning: the Conservatives burst a hiring bubble of their own making

Back_to_the_beginning__the_Conservatives_burst_a_hiring_bubble_of_their_own_making___Ottawa_CitizenGood analysis by James Bagnall on public service employee number swings. Most interesting figure for me was shift from the regions to Ottawa/Gatineau (from 33.9 to 39.4 percent), reflecting in part that the decisions are made in the capital, not the regions, and likely disproportionate cuts to service delivery. The controversy over the closing of Veterans Affairs example being the most public example, with cuts to CIC’s regional network being partly responsible for the dramatic decline in the number of new citizens in 2012 and 2013 :

The initial rapid rise in the size of the federal workforce was a response to the onset of the 2008 financial crisis. The thinking was that if the private sector stopped spending, government had to pick up the slack to prevent economic collapse.

When it became apparent a couple of years later that the world hadn’t ended, the Conservatives reasserted a party imperative: the budget must balance. The late finance minister Jim Flaherty began signalling restraint in 2010, then accelerated things with his March 2012 budget. An important catalyst was the introduction of executive bonus programs that rewarded managers who trimmed their budgets.

Huge swings in government employment aren’t unique to Conservatives. The Liberals under prime minister Jean Chrétien implemented equally drastic cuts in percentage terms during the mid- to late-1990s. Chrétien and his finance minister, Paul Martin, had little choice. Interest payments on the federal government’s debt consumed 31 per cent of total revenues and were growing.

Even after adding more than $150 billion to taxpayers’ debt burden, the Conservatives budget is still much healthier. Last year, debt interest represented little more than 10 per cent of revenues, thanks in large part to substantially lower interest rates than were faced by Chrétien.

An unexpected result of the Conservative government’s recent retrenchment has been a sharp rise in the percentage of public sector employees based in the National Capital Region. According to data compiled by Statistics Canada, 39.4 per cent of the federal government’s workforce in June lived in Ottawa or Gatineau – compared to just 33.9 per cent when the Conservatives were sworn in almost nine-and-a-half years ago.

Indeed, had it not been for this centralization, the economy of the National Capital Region might have dipped perilously close to recession. Another way to look at it: From early 2006 to mid-2015, the Conservatives added 18,700 government jobs in Ottawa and Gatineau – and took away 15,200 from the rest of the country. Among the federal departments disproportionately hurt by the job losses were Veterans Affairs, Agriculture, Defence, Employment and Environment – organizations with a strong presence nationally.

Whoever wins the federal election will find much within the government’s workforce in need of repair – and many employees who would like to see an end to the wild swings of the past 20 years.

Back to the beginning: the Conservatives burst a hiring bubble of their own making | Ottawa Citizen.