MedTech Case Study
Pattern recognition algorithm testing on genomic data
4 years in MedTech among 2 different projects. Although different, they share common issues, challenges, testing tech stack.
What we did was introducing MLOps processes and ML features functional and non-functional testing
What we did was introducing MLOps processes and ML features functional and non-functional testing
Pattern recognition and patient/doctor applications
- Pattern-recognition ML-driven solution based on genomic data analysisCombination of flexible workflows and data visualisation tools have helped provide a diagnosis where other approaches have failed - all within a 20-day diagnostic window.
- Patient and Doctor mobile applicationsBrought digital experience to every phase of patien-clinic workflow including appointments, labs, vaccinations. Integration government Medical Information System licences to be used in clinics and modern Android and iOS UX
Challenges
Industry-specific challenges in MedTech projects related to QA, Data and MLOps
- Sensitive data and privacy
- Obfuscation
- Deliberate change of specific parts of the sensitive data for testing and development
- Manual process of data collection setup with legal, scientific team and data engineering team
- Big DataHuman genome and other medical information can hold tens of gigabytes of text in a single file which cannot be reduced through the whole development cycle
- Cloud infrastructure
- Compressing medicine-specific formats like FASTQ
- Reducing amount of data to train/test the tests to speed up development of the pipelines
- Chained interdependent pipelines including data preparation and processing stages
- Closed isolated environments
- Emulation on small scale
- Partially limited End-To-End validation on test environment
- Inability to control test data on legally unavailable services
- Testing using proxy and on gateway level
- Testing in production
- Environment limitations result into necessity to run part of end-to-end regression testing in production environment
- QA engineers proving their own personal data to test specific feature on production for them selves
- ML pipelines run 30+ hours
- Setup MLOps practices
- Test each chain segment of chained of the pipeline independently
- Realtime reporting and logging of the pipeline before run has finished
- Long release cycles (monthly, in some cases - yearly)
- Long design and analytical phases cause constant urgency in development
- Prepare test data and test pipelines on early stages
- Development and QA involved in analytics phase to reduce communication ping-pong in development
- Feature toggle on prod for partial releases
ML Testing
Automation of test regression
Data that is used by ML team for development is, to some extent, random, small scale and possibly can have crucial differences with production. Which makes us require to get our own data for E2E pipeline testing
Since its data is very privacy-sensitive, we have to work with compliance officer on client side and data engineer on our side to manually select production data, make it eligible for testing and upload to our AWS storage
After ML pipeline has finished running, we need to know if it succeeded. Output of such pipeline would be in our case a piece of genomic data in VCF and JSON formats. That is yet again work for ML engineer to tell us what expected VCF is.
Value of Automated regression
So far all the work has been done by ML engineers and compliance officers. Why do they need regular software engineers
On each new iteration of algorithm we dont know how truth data set should look like. What we can know is how old truth looked like.
Since we can't test new functionality manually - only ML engineers can. And they also can't btw. They only can train their model and hope it would perform well in production. So the value is: to protect us from regression bugs in our ML algorithms.
On each new iteration of algorithm we dont know how truth data set should look like. What we can know is how old truth looked like.
Since we can't test new functionality manually - only ML engineers can. And they also can't btw. They only can train their model and hope it would perform well in production. So the value is: to protect us from regression bugs in our ML algorithms.
Since ML pipelines take significant time, it's important to run all available test data options in parallel. Using Jenkins.groovy you can define data-driven pipelines that can trigger multiple agents to run in parallel on different pieces of data
We can provide our test data into that chain of pipelines early on, triggering not only validation and ML part on all our test data subsets, but also the whole e2e chain of data pipelines
Test data can be multi-dimensional. Jenkins.groovy provides ability to generate test cases based on Cartesian product of the dimensions
This is we again need to work very closely with data engineers to define error thresholds and report format.
In the end validation is still half-manual.
Even when pipeline passed and the error we have is less then defined threshold, we still want to see error that has been found. And no error at all is also bad sign.
In the end validation is still half-manual.
Even when pipeline passed and the error we have is less then defined threshold, we still want to see error that has been found. And no error at all is also bad sign.
AI effect
Modern AI agents can be stoped with bureaucracy, compliance and other regulation requirements. Especially in such sensitive areas as diagnostics. Other than that boost in pipeline test development in support was significant.
> 50%
> 50%
what have we learned
Output
We have made great progress verifying ML algorithms through 2 years on the project. Here are some bullet points and diagrams on that.
