HFE in China: A Roundtable Conversation on the NMPA Draft Guidance

Nick  0:20 

So, put simply, it means for products already on the market in China—there doesn’t need to be any remediation work done, or any additional or new human factors work to be conducted. It’s quite interesting, actually, when we think of the updates to the MDR requirements – which put special emphasis on no ‘grandfathering’ provisions – meaning a lot of companies are now looking at their existing products on the market in Europe, and conducting post market surveillance to see whether any new HF work or updates to their HF file are required. When we look at China’s draft guidance, they haven’t said anything specifically about conducting HF for products that are already on the market. So that is good news for manufacturers.

Serge  1:20 

I would agree with Nick. I think that there’s always a concern, right? You don’t want your product to be pulled from the market because of new guidance – but the same thing happened in the US in 2009. When the draft guidance came out, people were concerned. I think it’s fair to believe that China will take a similar position, existing products will stay on the market. The question is more ‘what about Gen2?’ and ‘can I use the data that I was able to collect from Gen1 and to apply it to Gen2?’ In China, you won’t be able to ‘grandfather’ Gen1 data for Gen2, it’s going to be considered a new device, and you’ll need to supply human factors data in that submission.

Nick  2:40 

So when we look at the differences to the new NMPA guidance, and other international standards, such as FDA guidance or IEC 62366-1&2, which all consultancies and manufacturers follow when they build their HF file. In general there aren’t significant differences from that guidance and standards. NMPA follows a very similar approach and methodologies that are outlined in existing human factors standards and guidance. Our team has compared them side by side and pulled out any differences. But in general, I think what people should be most aware of is considering where they’re testing. Similar to the FDA guidance that states testing should be done in the US with US residing participants; the China guidance does the same. It specifically calls out testing Chinese users. Whereas when we consider IEC 62366, there’s no specific requirement of who those users are, so we could be testing in Europe, London, Germany, France, etc. – which we’ve done many times. Another interesting point is the recommendation on the number of participants in a study. NMPA draft guidance recommends 20 participants, but a minimum of 15 is acceptable for a summative study – similar again to FDA’s minimum of 15 – but formally recommending 20 per user group in summative (validadtion) study. Formative studies have the same recommended sample sizes as FDA, so no differences there.

Another interesting thing is the guidance regarding the classification of devices. NMPA clearly states that only Class II and Class III devices need to go through the HF procedures. Serge, any other points you feel are worthy of noting?

Serge  4:54 

In general, if you step back and you look high level at NMPA’s draft guidance –  you need to generate human factors documentation to support your rationale for study protocol design and ultimately the claim that your device is safe and effective. So, you’ll have that documentation, you’ll conduct formative testing to help mitigate any risks as you develop your product, and you have to conduct a summative (validation) study, which will be the sum of all your human factors work. So, at a high level, it is exactly the same thing most companies and HF teams are used to. It’s the smaller details within the draft guidance that could have a very important impact, but overall the principles are the same. I think that NMPA was smart in aligning their guidance with international standards. A lot of it is aligned with FDA recommendations. So now it comes down to small details within the guidance that could apply to you (or not). And we believe the biggest one will not be not how you actually design a study, conduct a study, the principle of basing your protocol on a risk assessment and on tasks that could lead to harm – those principles are the same. But for example, when we are looking at users, when we’re looking at the training those users receive prior to working in those facilities, training on the job, or the different workloads each of those users may have – what you’ll have in China versus the US or Europe might be different. And if those things have an impact – then tasks or user abilities may also be different in China. That’s where you start to see where it could truly impact the way companies design their studies. But the overall principles stay the same.

Nick  7:30 

Serge, to add to that, The NMPA have said that when a manufacturer is considering those differences, a comparative study would equal more effort than conducting the actual study with end users in China. When we conduct studies in the U.S. for an FDA submission, we rarely try and justify no testing because a similar product is on the market. It’s likely the intended use and users will have differences—meaning we’d conduct a human factors study with the end users. It’s no different in China. As we know, once the guidance is implemented, if you’re releasing a product and the human factors data is weak or you can’t prove the minor differences between geographies—you should just conduct a study locally and gather that feedback. In my opinion, it’ll make your submission process a lot quicker and less burdensome. 

Nick  8:40 

A comparative study would be used with an existing device already on the market, say in the US or Europe. That comparative study would be run to show how the differences in China have no impact on the critical tasks or associated risk with that device. It can also be used for updating devices (next Gen) that are already on the Chinese market. For example, if a device is sold in the Chinese market and say a specific feature on the GUI is updated – a comparative study can be done with users to substantiate that update creates no additional risk or potential harm to users and, therefore, a final summative study shouldn’t be required. Again, if you find that your comparative study shows there’s new associated risk, you will need to conduct a validation study with Chinese users.

Matt  10:01 

I just want to go back to something you highlighted in your earlier answer. Essentially, that was about whether or not someone should be thinking about running a comparative study versus perhaps just a validation study in China, what would you say to someone considering a comparative study, in lieu of actually running a validation study?

Serge  10:18 

The first step would be to look at your documentation. We’d start by creating a table to compare users, use environment, training, tasks that each user is responsible for, and how this device is used in relation to other devices within a case. Take surgical equipment for example, you will look at all those variables, and from there you might decide that the differences that you see do not impact the critical tasks you’ve already identified and your risk assessment that you already conducted for US and EU. So, you may feel very comfortable to say, my product doesn’t need an additional study and a report showing there’s no differences or impact to the safety and efficacy of my device in China, even though there may be minor differences. Now, if you see there’s differences that do impact critical tasks, that’s when you start to consider executing formative or validation testing. At that point, it’s all about your risk tolerance. How willing are you to go and submit that to NMPA? What are the risks of being rejected and having to conduct it afterwards? We understand that some people may try at the beginning to propose that justification, but in the background, they will still conduct some studies, just to have that in their back pocket. Then if NMPA says it’s required, they can respond more quickly. So there’s a lot of different strategies that could be taken. But a step-by-step approach will be the safest and the most systematic way to assess if you need to conduct any HF studies on products you want to launch in the Chinese market.

Otmar  12:39 

To take that a step further, you can compare what the users are doing, but you’ll also want to compare who the actual users are. It’s not uncommon that the person doing a certain task in the US or in the EU is not the same person in China, which means you really have to go all the way back to defining your use specifications – and then everything after that is going to be quite different.

Otmar  13:17

In a lot of the the user research and human factors programs we’ve run in China, users or task owners are different in China compared to other geographies. What I’ve found especially interesting is that the role of the distributor is a lot larger, sometimes they even scrub in and play quite an active role in surgery. So there’s a very big difference in who the actual users are.

You have to understand, China is a developing country and there are certainly hospitals that are very top of the line with highly educated users. But there’s also  part of the population who’s really living in more developing areas, in more remote regions, or the lower class hospitals who have a very different experience. So that tremendous range has a huge impact for human factors studies where you want to make sure it’s safe for everyone using your product.

It’s different than other forms of user or marketing research that a lot of our clients are doing, where you look at the top tiers – the people who are buying the most. But you have to also conduct HF in the lower tier hospitals in the more remote regions. The guidance specifically points that out.

To provide some background, China has different city tiers based on their level of development, so in a tier one city like Shanghai versus a less developed tier two or tier three, you’re getting into very different kinds of hospitals. And running parallel to that you have the hospital classes, where a class one hospital—they don’t do a lot of surgeries, class two hospital – they can do more and in a class three hospital—they can do a lot. And within that you have A and B, where a class 3A hospital would be considered the best. But even within Shanghai, there are many class 3A hospitals where one is certainly not the same as the other. So within the city tier and hospital class system, there’s a big range to consider.

Serge  15:43 

Yeah Otmar, what you’re saying is that between Europe and U.S. for example, there’s a lot more similarities —I understand there’s some differences between different regional hospitals or outpatient clinics versus a University Hospital and a teaching hospital in the U.S and Europe. Yet, the differences that you may have in China’s is often far greater. The way a product may be used in one setting versus another might be completely different, versus the more minor differences we see across North America and Europe. Same thing with users.

From the many ethnographic and observational research studies we’ve conducted in China, we even see that the type of training is completely different. You might expect that a physician in the U.S. would be managing a critical task, when in China it could be done by a nurse – or the other way around. It’s very interesting to see the extent of the shifts in task ownership.

We also talked about distributor impact in China—and that they can be way more involved than a distributor or sales rep would be doing in the US or Europe. What they do in China might be very different, and that individual may have no medical training whatsoever even though they know their product perfectly well. So that becomes quite a challenge in terms of deciding which user does which tasks compared to Europe or the U.S.

Otmar  17:28  

You also find that the labor cost is a lot lower here, so you’ll see hospitals throwing more users at a problem instead of training them more. But when someone is trained to use a complex piece of equipment, then suddenly there are not that many people who can do it. Then you have one person managing a critical device in multiple operating rooms at the same time. It doesn’t matter how loud your alarm is then—they’re not going to hear it!

Serge  18:02 

Exactly. We’re talking a lot about training, but there’s also the synergy within a team that you’ll have in the U.S. or Europe, compared to China. For example, we saw that in the aviation world before where a co-pilot was not allowed to challenge a pilot’s decision about a critical situation. We shifted in the aviation world with more teamwork instead of Captain and co-captain – and in the OR in the U.S. and Europe, we see that shift, too – the surgeon is no longer running as the captain of the OR. But in China, it can still be the case where it may be harder for a user to challenge a decision from someone that is ‘above’ them in terms of hierarchy. So that also becomes, from a workflow perspective, very different. There’s cultural elements that you need to take into consideration.

Otmar  19:16

Yeah, and not just in, you know, the actual workflow, but especially when doing usability studies. You’re going to try to get to the root cause of what the problem was, but believe me—that nurse is not going to say it with the surgeon there. You have to approach that conversation very carefully.

Serge  19:35

Correct. So there’s even considerations you need to make when running usability studies to adapt to the reality of the cultural differences, right? In China, they’re not really differences, they’re just reality. If you just take your protocol that you generated for a study in Chicago, and just have it translated in Chinese and run it in China, it won’t work. You need to adapt it to the reality of your users, cultural nuances, etc. over there.

Nick  20:21

It hasn’t been formally released when the new guidance will come into play. But we currently believe, late 2021 or early 2022, is likely when companies will have to comply with the NMPA guidance.

In relation to your question about what companies can do to prepare – I think there’s two things: there’s preparing, and then there’s actually implementing. For companies that are looking to release devices onto the market after the guidance has been released, or updating devices that are already on the market (next Gen)—I recommend taking a stepwise approach. Begin by assessing the market you are looking to launch in, and if that product is already in the US market, conducting a gap analysis is always a really good place to start. Look at the differences between the users in each market, with an emphasis on China.

Worrell is really well placed because we’ve done that user research in China, and can already help consult with teams that may have less experience. We recently shared insights with a client regarding a product they wanted to release in a different city in China. And even between those different cities in China, there were significant differences of who the end users were. Once we pointed that out, that client realized their use specification changed—meaning that a new HF plan had to be created.

So in summary – conduct a gap analysis, identify the differences. And from there, take the next step and create a human factors test plan. By doing this earlier, rather than later, you can have your formal HF plan created prior to the guidance implementation and be ready to conduct those studies in China in time. Serge, any other points on that?

Serge 22:14

Yeah, I think that it’s a very good question. Every company planning or currently selling products in China—must ask themselves that question. The other question to add is, how long of a transition period will NMPA allow? If you provide your submission the day after implementation, will it be rejected if there is no HF data? Or will there be a transition period between the two? It’s anyone’s guess on what it will be truly, and the decision on how you approach this really depends on how risk averse your company is – or perhaps the degree of risk associated with your device if it’s not used properly. How much harm could your device lead to? If you have a product that is, for example, on the list of high priority in the U.S. – well, it’s a good guess that on day two, they will require HF data included in your submission. They might not reject your submission, but they may delay it. If you know that today, to Nick’s point, you should be starting to prepare for this. As mentioned earlier, there is a lot of sensible preparation work you can do to mitigate this timing issue. We’re talking about a year to a year and a half of lead time here. That gives you some time to make an assessment and prepare documentation that will inevitably need to be done at some point.

The interesting thing when we’re talking about aligning all the different Human Factors work that will need to happen – before we had FDA, now we have Europe, and now China. But if you truly look at the different requirements of each of those three regulatory bodies, if you’re designing your human factors activities to align with all of them, you should not need to do the same thing twice. You should be careful, plan and be very strategic in the way you design your HF plan to minimize the impact on costs, but also timeline to make sure that you are in sync with your product development process. Those are things that we can help you with – crafting a proper plan that aligns your global HF strategy and requirements with your product development activities and timelines.

Matt  25:00 

So just to summarize some of those insights for our listeners, essentially, the advice here in terms of preparing for this guidance is to start with a gap analysis, examine your use specification, understand who your actual users are in China, what their environment of use is, what their training is, highlight any differences, as well as open questions – and then develop a human factors testing plan to close any gaps that you’ve identified.

Serge  25:26 

Exactly. What I would like to add are considerations for actually conducting a study in China. First of all, it needs to be in Chinese. Not just the study moderation, but the documentation that will be sent to NMPA as well. So right there, it shifts a lot of the work to a local partner you’ll need in China. I also think for U.S. partners, it is important to have an effective and realistic workflow. We all know how difficult it is on both teams to work at night, or very early in the morning. I think we have the luxury at Worrell to have an office in London that shares half a day with China and half a day with the U.S. So I’d highly encourage U.S. based teams to use our London office to truly manage that project. There’s a lot of meetings and a lot of things happening, and you want to make sure that you have a smooth transition on these new studies.

Nick  26:49 

We always kick off the project with our UK team on the call, and our moderating team in China led by Otmar on the call too. The UK team will be responsible for managing your project, so you’ll be interacting with that team for weekly meetings and updates. We’re also responsible for being the overall reviewer and approver of the documentation created by our Chinese HF team. The UK team will also share updates on recruitment, logistics and scheduling. When possible, it’s more likely for our European clients to observe some of those studies virtually along with our UK team. We work very closely with Otmar and his team during the preparation of the study documents – the protocol, data collection sheet etc. all the way up to the study. Otmar and the moderating team in China will manage the local recruiter – obviously being in the same timezone, they gather the updates live. If they have any changes to the schedule, we can update our U.S. and European clients. We work collaboratively as a team following a study to synthesize the data and discuss key insights. Our Chinese team will create a high level presentation of the results as well as draft the final report in Chinese which needs to be submitted to the NMPA – as well as an English version again for clients to review. Even though time zones are difficult, our UK and China team will collaboratively present these deliverables with our client team. Otmar and his team are very flexible to make those key meetings to deliver the results firsthand.

Serge  28:58 

Otmar, what does that mean for all the logistics in China? Here we’re contacting recruiters, we rent equipment, we design the environment appropriate for usability testing. How’s that different in China?

Otmar  29:18 

I think in a lot of ways it’s quite similar but some of the differences are interesting because Shanghai is a pretty big city, you can get a lot done locally. Of course you need to spread out and get multiple regions and different kinds of hospitals too – but there’s a lot we can do locally in Shanghai. We can run evaluations at our office, which is always very convenient. In terms of renting necessary equipment that’s often quite easy too.

What’s interesting is that most of our clients have local marketing and sales teams, and what we’re seeing is they’re a little bit more used to setting up marketing events for the for the HCPs. Of course for a human factors study you don’t want it to be too much of a marketing event, you need it to be, you know, you need, you need to really get the real results, and you need to talk about risks and potential problems. So that’s a very interesting dynamic that we’re seeing here in China, where you know, us as the objective observer, and moderator—to be pretty strong in that role.

Nick  31:00 

Matt that’s a great question – it’s  good news for everyone as well. Manufacturers or consultancies like Worrell can conduct usability studies directly – they do not need to do this within an NMPA testing house. While NMPA have provided recommended usability facilities that we’re partnering with already, in general manufacturers or independent consultants can conduct the their own studies at their own testing facilities for submission in China.

Serge  31:38 

To add to that – I think that if you look at the FDA guidance, and 62366-2, the intent behind those documents is to help a manufacturer, that may not be doing human factors on a regular basis, on how to structure their work. So you’ll see a lot of places in those guidance documents that talk about creating an overarching plan, to help your organization understand what human factors is. This guidance is really targeted at internal groups within a manufacturer. So, I think that NMPA is smart in their approach directly approaching manufacturers in China telling them, “This is a new guidance you need to comply with, and here is the process you should follow. If you have issues or difficulty in applying it, we have labs where you can come in.” At the same time, if you’re working with Worrell or any qualified Human Factors group, they already know how to conduct and structure a study or a usability plan for your product properly aligned with this guidance and these standards. If you’re dealing with a competent and experienced HF group, you should be alright. But if you would prefer to go on your own as a manufacturer, there are resources, and labs available to guide you through the process, which I think is very nice. I don’t think it’s something that we have here in the U.S.

Otmar  33:36 

Yeah, I think it’s really exciting too. There are a lot of local manufacturers who only release products for the Chinese market and don’t have anything in the U.S. or EU market. They really don’t know anything about human factors engineering, and suddenly these companies need to quickly learn how to apply it properly. As Serge said, since it’s so new to them, it’s great the government, has these kinds of facilities available for those types of companies who may have little to no human factors experience.