NRTL, CB scheme and the 3rd edition

Over the past 10 years or so, standards for electrical medical devices have greatly expanded from the traditional areas of electrical safety - fire, shock, basic mechanical and thermal hazards. The new areas include special tests and requirements specific to individual medical devices (particular standards), as well as management system standards such as risk management, usability and programmable systems.

There has been a great deal of discussion among NRTLs  and members in the CB scheme about how to handle these new requirements, such as the methods and the extent to which manufacturer data can be accepted, and also methods to audit management systems. An underlying assumption has been that test laboratories must find a way to provide complete reports.

Surprisingly, though, there seems to be little thought given to the fundamentals of third party testing and certification: responsibility and impartiality

A test laboratory must take responsibility for the test results (ISO/IEC 17025 Clause 4.1.1). It must also make efforts to remain impartial and minimise the impact of conflicts of interest. There are similar requirements for certification schemes, including the rules of the NRTL and CB schemes via ISO/IEC Guide 65.

Responsibility and impartiality affect both the objective and subjective requirements, but in different ways.

With respect to the objective tests in particular standards, many laboratories lack the technical experience and equipment for these special tests, and so turn to the manufacturer for assistance. There is no problem for this, provided that the laboratory remains responsible and impartial. This requires the laboratory to carefully assess individuals within the manufacturer's laboratory for their expertise, and also make efforts to minimise the impact of conflicts of interest.

The mistake that most test laboratories make is to rely test data from the same persons that were responsible for the design of the equipment. While the designers are doing a fantastic job, they are highly partial to the results of tests for a number of reasons, and also rarely make efforts to provide formal, quality laboratory testing to support compliance. While some manufacturers do establish internal laboratories, this is the exception rather than the rule. When designers are involved, compliance with standards is a side issue, considered as an afterthought and a hindrance from their main and very difficult task of actually getting a product to work in the real world. In this context it is not unreasonable that designers give only a surface level check of compliance, using many assumptions and simplifications, and where mistakes are not unusual.

Consider the following scenario: a common hazard with X-ray systems is stray radiation. Standards can require measurements in a 3 dimensional volume around the X-ray source. This test is time consuming and complicated. One manufacturer's designers think their lead shielding is pretty good from experience, and they simplify the test using an X-ray film to detect where the maximum output is, and then measure only from that point.

Later, they ask an NRTL for certification. The NRTL agency sends an engineer to witness the tests. The agency engineer has limited experience in X-ray and accepts the modified test method. A reviewer checks and signs the report, and an NRTL certificate is issued.

The point of this case is not whether the simplified method is acceptable - it could be a reasonable simplification. The real question here is who takes responsibility for accepting the simplification. The designers offered the revised method on the expectation that the test agency takes responsibility. They assume that if the method is not acceptable, the agency will reject the test data and ask for full 3 dimensional tests according to the standard. The test agency on the other hand thinks the designers are taking responsibility. They assume that the designers would not have offered the data if the simplification was not reasonable, and lack the experience and confidence to challenge the test data offered.

The result is a responsibility gap, with the manufacturer and the test agency both looking at each other, and neither actually taking responsibility.

This responsibility gap explains why, when laboratories do properly perform full impartial tests, there are often many non-conformities with particular standards.

Of course, if a patient or operater were actually harmed, the NRTL may be held legally responsible. But stray radiation is not something that leaves a clear line between harm and the cause, that would allow OSHA or the FDA to point a finger at the NRTL. Rather, the requirement is there to protect the operator and patient against a hidden hazard.

Many requirements in particular standards are of this nature, protecting against lower probability events that are not easy to trace back to the device that caused it. Thus the absence of incidents or legal liability is no indication that NRTLs and CBTLs are doing their job properly, and the importance of checking compliance closely should not be understated. Also, better incident reporting systems in Europe and the US are finding more and more problems linked to equipment design, and it is only a matter of time before NRTLs and CBTLs are brought into the firing line for failing to verify compliance impartially.  

CB scheme rules do require responsibility and impartiality are maintained, including when the CBTL extends to include manufacturer test data. While this reasonably well enforced for basic (electrical) safety, it is a well known secret within the CB scheme that flexibility can be applied when it comes to particular standards in the medical series.

There are a few justifications for this approach:

First is that CB scheme requires the particulars to be applied, creating a potential deadlock situation if the rules were applied strictly. This is certainly true. Most CBTLs would have to stop work in the medical area if particulars are required and they must have full technical competence, impartiality and take full responsibility for the test results.

Second, CBTLs also assume that everyone knows they are primarily electrical safety laboratories. They assume that regulatory reviewers essentially ignore their reports and look at the manufacturer's raw data. Using this thinking, CBTLs consider themselves exempt from requirements for impartiality and technical competence, and most importantly they do not feel any sense of responsibility for any statements of compliance in test reports.

This part is false, and dangerously so. Manufacturers frequently use the reports prepared by NRTLs and CBTLs in their technical files to support regulatory compliance, in place of their own internal testing. The reason is simple: third party reports carry more weight than a internal report. Regulators also give more respect and trust to third party reports. This is partly because they assume the test laboratory is taking responsibility and ensuring impartiality, but in practice regulatory reviewers are also stretched for time and technical resources. The presence of a third party report relieves them of the responsibility to check in any further detail.

Finally, CBTLs also assume that the manufacturer is the expert, so their test data can be trusted. This is true if the manufacturer is making an effort to create an internal laboratory free from undue influence. It is also true that designers are usually experts, but it does not follow that their test data can be trusted. As described above, test data from the designers is often weak because of large conflicts of interest.

NRTLs operates under similar rules as the CB scheme, but there is a difference: the NRTL scheme itself is limited in scope to the protection of employees, not patients. Therefore many requirements in particular standards are out of the scope of the scheme, and NRTL accreditation is meaningless as evidence of competence for a test laboratory for these standards. Similarly, SCC accreditation in Canada is also limited in scope to basic safety (shock, fire) and does not cover many requirements in particular standards. So while some NRTL/SCC labs will insist on full compliance, in reality there is no high level reason for them to do so, nor anyone checking the quality of their work.

The result is that we get reports from NRTLs and CBTLs where the laboratory frequently ends up endorsing poor quality manufacturer testing, fails to take any responsibility, and where the regulators then blindly trust these reports. It is in effect a cosy relationship between test labs, manufacturers and regulators, where each one is looking to the other for responsibility, but no one is actually stepping up to the plate. Which is fine for everyone, except the patient and operator of the medical device.

Management systems also present a special problem for test laboratories: these actually specify requirements for the manufacturer rather than the product, and do not fit to ISO/IEC 17025 and Guide 65 which have been written for product certification based on impartial, objective laboratory test results. Moreover, assessment with any added value still requires expertise to understand the technology and clinical application of the device: expertise that is reasonably missing in test laboratories that have been established for electrical safety. It is not clear how regulators, who are more adept in auditing management system requirements, will consider efforts by CBTLs and NRTLs to evaluate these standards, but there remains the risk that CBTLs will end up unwittingly endorsing poor quality work. 

What does this mean for the future of NRTL and CB?

The CB scheme has a history of slowly getting more strict, as auditors get more experienced and start to ask better questions about the quality of manufacturer test data. We may find that the unwritten rules quietly disappear. Alternately, it may be that one or two laboratories come under legal attack for failing to take appropriate steps, for example an incident occurs, and the investigation finds the test lab accepted incorrect test data without any experience and from the same person responsible for design. This may lead to laboratories and their insurers taking a closer look at the medical business and the risk the laboratory is taking by accepting manufacturer data. Either way, the CB scheme may face a deadlock if it continues to enforce rules that require full compliance with medical standards to be established.

Thus we may see the CB scheme move to a formal system that allows partial assessment of a standards for medical equipment, possibly with a minimum requirement to cover "basic safety" according to the definition in the 3rd edition of IEC 60601-1. Only those laboratories that can show real technical competence, impartiality and be willing to take full legal responsibility should be allowed to perform tests against particulars and other standards. 

In this context, the CB scheme needs to acknowledge that it's primary role is supporting NRTL/SCC marks, which do not require testing outside of basic safety. While CB reports may also play a role in regulatory applications, this should only be where test labs can add value, which can only occur if they have the technical competence in the technologies involved. We may see in the future more test labs specializing in testing medical particular standards, and possibly a CB scheme structure that allows these labs to combine their test results with general requirements (basic electrical safety) to form a complete reports.

We may also see a separate system established for the auditing of management system standards, but this should stem from rules applicable to auditing (such as Guide 62) and possible run under a different accreditation system than the CB scheme. Again, technical competence is paramount if such a scheme is to have any added value.  

Regardless, even in the current environment, greater efforts should to be made to ensure the manufacturer is creating a impartial, laboratory environment to obtain that test data. Most importantly, test labs should simply refuse to accept data from the same people that designed the product. For sure, designers may assist to set up tests and explain test results, but they should be excluded from any decision making roles in determining compliance. This simple rule alone would greatly improve the quality of testing provided by CBTLs and NRTLs operating today.