Kaiko creates a strong foundation to scale

Creating a strong foundation to scale

Kaiko, a healthcare startup, needed a strong technical foundation to build its ML-based data framework for cancer research facilities. We accompanied Kaiko’s first engineering team and created its coding infrastructure, workflows, and CI/CD framework. We also helped Kaiko become more visible in open-source communities, helping it hire senior engineers and create a strong foundation for its product.

Our work involved

• Monorepo conception
• Workflow automation
• Engineering leadership
• Data engineering consulting
• Infrastructure DevOps

Founded by the Hartwig Foundation, Kaiko is a Dutch/Swiss tech startup that provides data framework and AI support for cancer research facilities and hospitals. It unites high skilled data scientists, software engineers, and medical experts to advance open-source data solutions, machine-learning tooling, and algorithm development – all aimed at improving patient outcomes. For example, Kaiko develops algorithms that use AI to find cancerous cells in medical images.

Challenge

Setting up an engineering team

It’s difficult for an early-stage startup to hire experienced engineers because it neither has brand awareness nor market presence. Yet, that’s the time when experience matters the most. Initial technical choices have far-reaching implications.

Being industry veterans, Kaiko’s founders Robert Berke and Thomas Hufener understood the importance of strong engineering leadership at the beginning of a startup’s journey. They hired Tweag, the open source program office of Modus Create, to create a successful foundation for Kaiko and help attract senior engineering talent. The team’s extensive experience with machine learning and advanced data engineering in healthcare made us an ideal partner for the project.

Kaiko’s leaders specifically asked us to introduce engineering best practices at their organization. This would not only create a strong technical foundation for their product but also sow the seeds for an agile, collaborative culture.

The team set two major objectives for the project:

• Boost Kaiko’s awareness in open source to attract leading engineering talent
• Advise on the initial technical choices and engineering processes to help Kaiko scale.

Attracting talent in open source communities

DICOM® (Digital Imaging and Communications in Medicine) is the international standard for medical images and related information. It defines the formats for medical images that can be exchanged with the data and quality necessary for clinical use.

Unfortunately, Apache Spark, one of the leading data processing engines, was missing DICOM support, making distributed data processing challenging for certain medical use cases. Therefore, we created the spark-dicom integration, which simplified data processing in healthcare and created awareness about Kaiko in open-source communities. Over the next few months, Kaiko received several applications from senior developers and data scientists.

Designing a monorepo

Kaiko’s CTO wanted to use a monorepo to share best practices and improve collaboration. We consolidated experiments that were disseminated in multiple repositories to build a monorepo from scratch.

As most Kaiko developers lacked experience with monorepos, we provided support and explained the ins and outs to ensure everyone understood the decision. We also supported with the following:

• Helping choose development tools such as a Python package manager that matched Kaiko’s constraints
• Providing guidance on structuring the technical deliverables like libraries and projects inside the monorepo
• Creating workflows, both for the development environment and for the continuous integration environment (CI)
• Placing engineering quality checks (formatting, linting, type checking, and testing) and creating automated alerts
• Setting a workflow system on Pants to ensure this increasingly large Python monorepo would scale well in the future.

Tools used

Improving developer experience with automation

You can fake agile, but you can’t fake automation. Automation is the key to a stellar developer experience. It helps your engineers focus on high-quality tasks instead of continuous context-switching and status update meetings.

We used GitHub Actions, a continuous integration and continuous delivery (CI/CD) platform, to automate Kaiko’s build, test, and deployment pipeline. This extensively automated the day-to-day tasks of developers to increase the team’s throughput, from using merge bots to automating releases.

Source: github.com

We also enabled automatic code synchronization from unrelated repositories, which made it possible for the monorepo to be the single source of truth and still contribute to other repositories. This was critical because Kaiko frequently collaborates with hospitals and research institutes. For this, we leveraged Google’s Copybara and extended it to support Azure DevOps repositories, in addition to GitHub ones.

Finally, we introduced the Kodiak mergebot to fasten the workflow of Kaiko’s developers, and to automate the copies done by Copybara.

A lot of startups would benefit from the experience Tweag brought to the table. It has set a very solid foundation on which we can grow now. - Thomas Hufener, Kaiko CEO

Impact

A strong foundation for growth

Additionally, workflow automation and distributed collaboration have future-proofed Kaiko’s engineering operations. For example, when contributions to the monorepo are merged, the changes are published automatically so that the hospitals can access them in real-time.

Today, Kaiko is collaborating with the Netherlands Cancer Institute (NKI-AVL) to build a secure and scalable data architecture, validate A.I. support, and enable machine learning research.