An Excel error could delay Japan’s massive immigration overhaul

As someone who works a lot with data, this can happen. But it shouldn’t, given government validation and checking processes:

Japan’s government seems to be in need of some tech support.

Its plans to pass a crucial immigration bill that could open the country’s doors further to as many as 340,000 foreign workers from next year might be stymied due to data input errors.

Japan’s government had given lawmakers an analysis of why foreign workers in the country are dropping out of an existing work training program, as it argues for the country to create create two new categories of work visas. The justice ministry admitted last week that the data on those workers was incorrect, and blamed the problems on the handling of an Excel spreadsheet, Japan Times reported yesterday (Nov. 19). For example, the analysis exaggerated the number of foreign workers who left their jobs because they wanted higher-paying positions, rather than to escape poor wages or working conditions.

There are some 1.3 million foreign workers in Japan as of 2017, a 17% increase from the previous year, as businesses try and fill positions in industries ranging from construction to food preparation to nursing amid a shrinkage in Japan’s working population. Right now, foreign workers filling run-of-the-mill jobs are often in the country on temporary “trainee” visas that lock them into employers.

The proposed work-visa categories, approved this month by prime minister Shinzo Abe’s cabinet, would allow those with “specified skills” in the most labor-starved industries to live and work in Japan under for up to five years. The new visa status would also allow such workers more flexibility in changing jobs, which would make them less vulnerable, proponents of greater immigration to Japan say.

Though Japan’s justice ministry has said that the errors were the result of mistakes in data processing—the latest IT mishap after Japan’s newly appointed cybersecurity minister admitted that he had never used a computer—opposition lawmakers have accused the government of glossing over the problems with the current trainee program in order to rush the bill through.

In the revised data, the government said for example that 12.6% of trainees said that they left their jobs because of harsh working conditions, up from the previous 5.4% presented by the ministry. Opposition legislators boycotted a debate over the immigration overhaul in the Diet last week in protest, but deliberations could resume this week.Many of those currently working as trainees are expected to switch over to the new visa status once the bill becomes law.

Calling foreign workers technical trainees or interns was a workaround for the government to keep it from having to admit that more people from overseas are living in Japan—a country where many remain deeply apprehensive about immigration, even as it struggles with a severe labor shortage. But it’s also a workaround that has left thousands of workers vulnerable to exploitation by employers and the brokers who bring them over, as many of these trainees told lawmakers earlier this month.

Source: An Excel error could delay Japan’s massive immigration overhaul

And public opinion appears sceptical regarding the proposed changes:

Sixty-four percent of respondents said there is no rush to revise the immigration control law to expand the acceptance of foreign workers from next spring, according to an Asahi Shimbun poll released on Nov. 20.

They said it is not necessary to pass the revisions in the current extraordinary Diet session, while 22 percent of respondents believe it should be.

The nationwide poll was conducted Nov. 17 and 18.

The government and the ruling parties are seeking to pass the revisions to the immigration control law in the current Diet session.

However, even among supporters of the Liberal Democratic Party, the main force of the ruling coalition, 57 percent said that it is not necessary to do so. Only 31 percent replied that the revisions should be passed in the current session.

The survey also asked respondents about whether they support the expansion of acceptance of foreign workers. Forty-five percent, down from 49 percent in the previous survey in October, said they support it. Forty-three percent, up from 37 percent, expressed opposition.

Prime Minister Shinzo Abe has said that accepting more foreign workers into Japan is not a policy of accepting immigrants. As for Abe’s comment, 52 percent of respondents said that they don’t accept the explanation while 29 percent replied that they agree with it.

In the latest poll, the support rate for the Abe Cabinet stood at 43 percent, up from 40 percent of the previous survey, while the nonsupport rate was 34 percent, down from 40 percent.

The latest number means that the support rate for the Abe Cabinet recovered to the levels recorded in January and February polls, which were taken prior to the revelation of the alteration of Finance Ministry documents.

Respondents also were asked about the four islands off eastern Hokkaido that were occupied by the Soviet Union at the end of World War II in 1945. The islands, called the Northern Territories in Japan, are Etorofu, Kunashiri, Shikotan and Habomai.

Abe agreed with Russian President Vladimir Putin in their summit on Nov. 14 to accelerate peace treaty negotiations based on the 1956 Japan-Soviet joint declaration that stipulates the return of two islands, Shikotan and Habomai, to Japan after concluding a peace treaty.

The survey asked respondents if they expect an agreement to lead to a resolution of the long stalemate over the Northern Territories issue.

A total of 60 percent replied that they don’t expect that at all or very much. Thirty-eight percent responded that they very much expect it or at least to some degree.

The survey also asked about how Japan should deal with the Northern Territories issue.

Fifty-one percent replied that the government should first seek the return of Shikotan and Habomai and continue to hold negotiations on the return of the remaining two islands.

Meanwhile, 25 percent said Russia should return the four islands at the same time, and 11 percent said that Japan should conclude the Northern Territories issue with the return of the two islands. Six percent replied that Japan should not seek the return of any of the four islands.

The Asahi Shimbun conducted the survey through land-line telephones and mobile phones of eligible voters chosen randomly by computer.

Of 2,048 households contacted with land-line telephones, 991 people, or 48 percent, gave valid responses. As for mobile phone users, 949 of 2,022 people, or 47 percent, gave valid responses.

Land-line telephones do not include those located in a part of Fukushima Prefecture.

Source:  Poll: 64% say not necessary to rush revisions to immigration law November 20, 2018 Sixty-four percent of respondents said there is no rush to revise the 


‘Anything would be better:’ Critics warn Ottawa’s family-reunification lottery is flawed, open to manipulation – The Globe and Mail

Almost comical if it were not for the impact on people. And it should not be surprising, given our immigration system’s emphasis on high skilled economic immigrants, that some of them should have the mathematical and technical smarts to point out the lack of randomness:

Canada’s family-reunification program is using a common spreadsheet application to select candidates as part of a process critics say is flawed and open to manipulation.

As the first step in the program, the federal government uses Microsoft Excel to randomly pick applications in its lottery, The Globe and Mail has learned. Experts have warned that using Excel to conduct such a sensitive lottery could be problematic, and that the lottery process itself may make the system less fair over all.

The Parents and Grandparents Program allows Canadians to sponsor family members for permanent-resident status. The Liberals introduced a lottery in 2017 in an effort to make the system fairer – previously, applications were accepted on a first-come, first-served basis. The program receives roughly 100,000 applications each year and selects 10,000.

Details on the lottery, obtained through an Access to Information request shared with The Globe, show a procedure carried out in just a few steps: Immigration, Refugees and Citizenship Canada (IRCC) uses Excel to assign each application a random number, then takes the first 10,000 numbers.

Excel’s method for generating random numbers is “very bad,” according to Université de Montréal computer-science professor Pierre L’Ecuyer, an expert in random-number generation. “It’s a very old generator, and it’s really not state-of-the-art.” Prof. L’Ecuyer’s research has shown that Excel’s random-number generator doesn’t pass certain statistical tests, meaning it’s less random than it appears. Under the current system, “it may be that not everybody has exactly the same chance,” Prof. L’Ecuyer said.

Excel uses pseudo-random number generators, a class of algorithms that rely on formulas to generate numbers. These generators have a key flaw – they rely on a “seed” number to kick off the mathematical process. In the case of Excel, this seed is generated automatically by the application. “If you know one number at one step,” Prof. L’Ecuyer explained, “you can compute all the numbers that will follow.”

This means the process could be exploited by someone with the right skills. It’s happened before: In 1994, IT consultant Daniel Corriveau discovered a pattern in a keno game – which uses a random numbering system – at the Casino de Montréal and won $620,000 in a single evening. An investigation later determined the game was using the same seed number at the start of each day.

Using more robust generators, such as the ones used for cryptography, may not cost the government much, either. “Cryptographic generators are free. They are on the internet,” Prof. L’Ecuyer said. “Just pick one, you need to know about it and that’s all. It’s not complicated.

“Anything would be better.”

For its part, IRCC is satisfied with its use of Excel, spokeswoman Shannon Ker said in an e-mailed statement. “We stand by this randomized selection process as a sufficient means of equal opportunity for all who look to express an interest in sponsoring their parents and grandparents.”

Others would rather see the lottery scrapped altogether. For the past two years, Igor Wolford, a data-analytics manager at Loblaws, has applied to sponsor his parents in Russia. He hasn’t made it past the lottery stage, and recently started a website to petition the federal government to abandon the system.

Mr. Wolford has corresponded with members of Parliament about his concerns. “I actually prepared an Excel sheet showing how random processes work,” Mr. Wolford said. “After 10 years of selection, only half of people who were eligible 10 years ago would be selected.”

Number of people from an original pool of 95,000 applicants who haven‘t made it past the lottery stage
Assuming 20,000 new applicants each year and 10,000 applicants selected each year 

Although the lottery selects roughly one in 10 applications, the number of people who pass additional vetting and ultimately make it into the program is far lower.

“Last year, they selected the original 10,000 people [during the lottery], but only 6,000 people actually [made it into the program],” Mr. Wolford said. This is partly be cause the lottery is the first step in the process, meaning anyone can fill out the form.

IRCC responded to these complaints in 2018 by including a self-assessment screening for applicants. However, the questions are still optional, as one Twitter user noted.

When told the lottery was conducted in Excel, Mr. Wolford wasn’t surprised. “That’s a very sad process. It’s easily manipulatable,” he warned. According to IRCC, the process is double-blind, and to date there is no indication the system has been manipulated.

“The process has become unpredictable,” Mr. Wolford said. “Before, you knew that it would take seven years from start to finish, and you could plan your life. Right now, you don’t know if it will happen this year, in five years, in 15 years.”

“Because it’s a lottery, you might never be selected.”

via ‘Anything would be better:’ Critics warn Ottawa’s family-reunification lottery is flawed, open to manipulation – The Globe and Mail

Nine things everyone should know how to do with a spreadsheet | Macworld

As I am starting to use spreadsheets to analyze demographic and related data, my basic knowledge of spreadsheets is being challenged. Another primer from Macworld (but applies to Excel and Google’s Sheets as well).

As I have been only using sum and average functions, these examples of other functions caught my eye:

=MAXRANGE and =MINRANGE: Return the largest and smallest values in a range. Related to these two, I also often use =RANKCELL,RANGE, which returns the rank of a given cell within the specified range.

=NOW: Inserts the current date and time, which is then updated each time the spreadsheet recalculates. In both Excel and Sheets, you need to add a set of parentheses: =NOW.

=TRIMCELL: If you work with text that you copy and paste from other sources, there’s a good chance you’ll find extra spaces at the beginning or end of some lines of text. The TRIM function removes all those leading and trailing spaces but leaves the spaces between words.

Nine things everyone should know how to do with a spreadsheet | Macworld.