Plan B and Rutherford Rede commissioned the report written by Michael Mintrom, an expert in research methods and public policy analysis, after both financial advisory companies had their advice rejected in the Consumer study which described financial advice in New Zealand as "scandalously poor".

Consumer says the purpose of the study was to see if the standard of financial advice had improved and it received financial support from the Ministry of Economic Development and the Retirement Commission.

Overall it mystery shopped 33 financial advisers from large institutions with in-house advisers and agents, sharebrokers and nationwide adviser chains, to small standalone firms. An expert panel assessed the quality of advice and information in the 17 plans it received.

Only three out of 17 advisers produced plans that were rated "good" by the expert panel and the remaining 14 were rated as "disappointing" or were "rejected".

According to Mintrom the Consumer study was motivated by good intentions but the quality of the study was poor, especially when judged by the number of cases involved, the high rate of attrition of cases during the study and the lack of screening and training of the mystery shopper.

Mintrom says if this study had been done through the University of Auckland, the design would not have even made it through an ethics board.

One of the main concerns is the naming and shaming approach. Mintrom says this strategy should be avoided unless there is high quality and robust evidence, which is hard to obtain.

He says the sample size of the Consumer study was one of the problems. The Institute of Financial Advisers has 1,400 members, yet the decision was to include only 33 firms in the study.

Only 29 were assessed because four plans arrived too late for assessment by the panel. Of those 29, another eight firms were dropped because of a mismatch between the shoppers need and the adviser. Of the 21 remaining, another four only gave oral advice leaving just 17 responses to be judged.

"The attrition of cases from the original sample of 33 suggests very poor project management," says Mintrom.

He also says the mismatch between the shoppers and advisers is clear evidence that the people who managed the project had no understanding of how their own incompetence could bias the results they obtained.

Mintrom suggests a study of at least 100 firms would have been needed to be more representative of the financial adviser industry and at least two to three mystery shoppers would have needed to visit the same firm to be sure of the level of advice.

"If you want to engage in naming and shaming, you have to be dead sure you are correct in the claim that a company is not giving good advice."

He says it is especially troubling that the mystery shoppers got to choose their own scenario.

It is reported that "eleven mystery shoppers ranging in age from their mid-30s to just over 80 visited financial planners... These were real people with real financial questions".

Mintrom says there is a fundamental problem here because the diversity of the mystery shoppers is very likely to have affected the quality of the advisory sessions.

"Hardly any effort was made to control the character profiles and test scenarios, as a result, the quality of the advisory sessions would have been influenced in unknown ways by the knowledge and behaviours of the mystery shoppers, which Consumer did nothing to control."

Consumer said in eight of the 33 cases, "there was a mismatch between the shopper's needs and the adviser" which Mintrom believes points to a major flaw in the study that very likely corrupted the findings.

He says studies that make effective use of mystery shoppers devote time and energy to developing realistic character profiles and test scenarios.

Mintrom believes that while Consumer had good intentions to start with, the way they went about it could end up having legal implications if companies decide to go down that route.

He believes the report is detrimental to the investment climate in New Zealand especially when the government is trying to get people to diversify with investment.

"I do have concern that taxpayer money was spent on this when the quality of the study was so poor.

"We must question why a survey of this kind, with such potential influence, was left to Consumer rather than a professional research house with the experience and background to manage the entire process effectively."

Consumer was not available for comment.