AI and ML in Technology Due Diligence

How does a company use AI/ML in their products?

Given the saturation coverage of Artificial Intelligence and Machine Learning everywhere, many companies now claim that it is a key part of their offering, and every company is “an AI company.” As an acquirer, you need to understand how AI functions within a company. Is it used to provide improved customer experience by creating suitable content that improves conversion? Is it to give accurate fraud detection capabilities, reduce risk and increase profitability? What about to provide a chat-based interface to an easy-to-use SaaS?

Each of these examples provides various levels of importance to organizations. A chat-based interface to a simple SaaS may be a recent add on, not core to a product. On the other hand, fraud detection and creating suitable text content provide real, tangible benefits and value to a company. It is important to look under the hood and make sure that AI/ML is incorporated in a suitable, substantial way, and not just a recent add on, with minimal value. Our approach to AI and ML in Technology Due Diligence helps you uncover the relevance of any claimed technology.

As an aside, with AI everywhere, you may be happy to know that I wrote this article personally, so I consider this to be at AI Influence Level 0. For more details about this have a look at this page for more details.

What AI/ML technology are they using?

There are several Artificial Intelligence and Machine Learning frameworks and platforms in the market. At the language level you have PyTorch, TensorFlow and many other frameworks. These are Python based and let you write your own Artificial Intelligence and Machine Learning systems. The company has full ownership and portability of data as well as code. In other words, these systems ensure appropriate ownership of intellectual property. However, this approach requires developers and data scientists.

At the other end of the scale, there are the AI/ML platform as a service (PaaS) offerings from cloud providers such as Amazon Web Services, Microsoft Azure and Google Cloud Platform.  These provide services in many different parts of Artificial Intelligence and Machine Learning. These can be generative AI offerings, or industry / task specific offerings. Areas include Text to Speech, Translation, optical character recognition or health imaging. Examples include AWS Bedrock, AWS SageMaker, Azure Machine Learning, Azure AI Studio, GCP Vertex AI Platform. These systems are less developer and data scientist oriented, and offer pre built solutions.

Assessing Value of AI and ML in Technology Due Diligence

If someone is using the PaaS or industry specific offerings, then you need to consider the value this provides for the company and your underlying investment thesis. Is AI/ML the core part of the value the company is providing? For example, what if a company is using AWS Textract OCR and the company’s core value proposition is document processing? The value of the AI/ML may be considered low, as the core processing is handled by a service provider, with minimal unique value add or competitive barriers to entry. On the other hand, if the company has large proprietary data sets and their own processing on top of existing platforms, more competitive value may be present.

When considering the value of AI/ML offerings, you need to also consider the COGS (Cost of Goods Sold) and how consuming AI/ML services will impact your cost base. PaaS offerings may offer accelerated development and deployment, but higher costs long term, or may not even be covered by the incremental revenue that AI/ML features provide.

The following sections of this article outline how to determine the maturity of Artificial Intelligence and Machine Learning in an organization from a holistic perspective. This is important when looking at AI and ML in technology due diligence because it helps us understand if the company has a handle on all aspects of the lifecycle of AI/ML, or if they are simply adding AI/ML as window dressing.

Model approaches with AI and ML in Technology Due Diligence

Models are the heart of current AI/ML technology, and the lifecycle of models is vital for understanding AI and ML in Technology Due Diligence.

How does a company train models?

The models are at the heart of Artificial Intelligence and Machine Learning. They determine how well a model will perform when used. Depending on how a company has created their AI/ML offering, the following situations may occur:

1. If using an external PaaS from a cloud provider, existing models created by the cloud provider may be consumed without modification.
2. Models may be augmented with data from other sources. This may often include fine tuning models found in the community. This is analogous to utilizing open source software and building upon it.
3. The company may have its own data sets to train from.

If a company uses pre trained models from other providers, then there are minimal training needs, which makes maintenance easier, but has limitations around flexibility of the model and being able to manage unique circumstances.

If a company makes use of their own data to either augment existing models, then the maturity level needed increases dramatically. The flexibility and ability to tailor behavior grows, but the company needs to do more. For example, consider the following questions:

1. How do you manage the data used for training? Is it stored in the cloud, in a data center, combinations of both? Is data stored in a vector database?
2. Is the data appropriately classified so that you can then train data with confidence?
3. How do you ensure an adequate split of training and test data? How do you avoid overtraining?
4. How frequently are models updated / revised?
5. How are training runs undertaken? What hardware is used for training? Are cloud based VMs with GPUs used? Are local machines with GPUs used?
6. What is the proximity of data sets to compute and GPU capacity?

When looking at AI and ML in Technology Due Diligence, these are key questions to understand to assess the robustness and value of a company as it impacts efficiency, cost and frequency of training.

How does a company use models?

Once a model is trained, there are extra steps needed to make it be usable in production. What they are and the complexity required depends on the circumstances:

1. Are you deploying in a cloud-based SaaS? This often means that models are accessed by clients via REST APIs by web or mobile clients. This is a straightforward operation that can leverage cloud hardware capabilities when needed.
2. Are you deploying to mobile / edge devices? Do these devices need to process AI/ML without network connectivity? In this situation, you need to consider the hardware capabilities of the devices. AI/ML workloads need to do significant amount of matrix manipulation, far more than a CPU alone can do efficiently. This is why GPU / NPU, and other AI extensions are in devices from iPhones to Android devices and CoPilot+ PCs. These have dedicated hardware for AI to speed up responsiveness. In the past, dedicated silicon for 3D graphics cards sped up gaming, and became the foundation of for AI hardware.  Now, the latest AI chips can dramatically improve the speed of client side computation.
3. How easily can you wrap the model up so that it is usable for your target scenario? Do the frameworks used to build the model enable easy transition to usage?
4. Are models combined with other data to undertake RAG (Retrieval Augmented Generation)?

These all need to be considered as it can easily make the difference between something that is usable and unusable, either for network or performance reasons. As always, the context matters, and when looking at AI and ML in Technology Due Diligence, we understand how AI/ML are meant to be used in the context of this company for this transaction.

Operational aspects of AI and ML in Technology Due Diligence

Once a company has trained their models, deployed their models, and have them in use in a product, this is something that needs to be done repeatedly and part of technical operations.

How does a company support AI/ML (MLOps)?

Using Artificial Intelligence and Machine Learning is not a “one and done” activity in a software offering. Key aspects need to be considered:

1. How do you manage model drift? AI/ML models are based on data, and data can change over time, making a model ineffective. If a fraud detection model is based on behavior of the past, new methods of fraud may not be detected unless new models are created.
2. How do you detect model drift? What steps do you do to measure effectiveness of AI/ML in the field, outside of training runs? You need to have the ability to detect when a model is going wrong, not just in a single situation, but in the aggregate.
3. How often do you retrain a model? Do you train on a regular basis, or only when performance drops?
4. Have you been collecting new data as the system runs? There needs to be a strategy in place to collect data when a system is running. Otherwise it can be difficult to train new models without new data, considering changes over time that cause model drift in the first place.
5. How do you adapt training and test data sets to new data? Care always needs to be taken that models are not over trained, so new data needs to be carefully curated so that test and training data are both available.

Once all these factors are considered when looking at AI and ML in Technology Due Diligence. Just as DevOps helps ensure a smooth, predictable, and systematic way of getting software into production, MLOps ensures that the entire lifecycle of AI/ML data and models is managed in a smooth data pipeline with predictable and systematic ways into production.

How is AI/ML performance checked?

As shown above, there are many distinct aspects to AI and ML in Technology Due Diligence. Assuming that all the models are created, able to be updated and deployed easily, how do you know that AI/ML is working correctly? This does of course depends on the unique circumstances of an individual company and its technology, but there are several common elements that need to be watched:

1. How long does it take to get a result from AI/ML related services? This shows if the provisioned hardware is sufficient and if there is a need to add extra provisioned hardware. In a cloud environment, this can often be resolved by using elasticity so that resources scale up and down as needed.
2. Model accuracy. How well are the models performing? Are they creating responses that are expected? Are the right predictions being made? Being able to measure and check accuracy is a fundamental part of detecting model drift. Without these detection mechanisms it is difficult to pre-emptively detect model problems and start to rectify them. Just as with traditional tech ops, if a customer is the one telling you there is a problem, your monitoring capabilities need to be upgraded.
3. Availability, data volume, resource usage. These are traditional operating metrics that help decide what resources are being used and how many are available. These end up impacting performance.
4. External resource usage. Depending on how AI/ML is implemented, other metered services are used. Having visibility into these helps track expected costs, which helps improve operational costs.

How does a company stay compliant?

Artificial Intelligence and Machine Learning make use of enormous amounts of data and need to continue to use it for training and testing purposes. This has many different regulatory implications, depending on the kind of industry a company works in:

1. What kind of data is used? Is it personally identifiable information? Is data from multiple different customer companies mingled together?
2. Do customers have a say in what happens with data? How do people remove data from training sets?
3. Is data deidentified before being used for training or testing?
4. If you use an underlying PaaS, who owns the data, models and know how in the system?

In addition to data protection perspectives, compliance also includes:

1. Data lineage and the rights to use data. Are the data sets owned by the company? Do they need to provide payment or royalties to a data owner?
2. Is the data widely used by other people?
3. Are there restrictions on how data can be used? Can it be used only for certain purposes in certain countries?

These are all issues that may be uncovered when conducting an anslysis of AI and ML in Technology Due Diligence.

How we help assess AI and ML in Technology due diligence

It is difficult to separate hype from reality when it comes to Artificial Intelligence and Machine Learning. This is magnified even more in a due diligence situation, where a target company wants to put its best foot forward and promotes aspects of the business that may be immature compared to how it is presented.

Through our Thought Source tried and true approach to technology due diligence, we can systematically evaluate AI/ML technology from many dimensions – is it useful to the business, is it tangential, a quick bolt on? How it is implemented, the maturity of the processes around it, what happens over time as a business runs and scales, along with making sure that data follows relevant regulatory requirements?

We are happy to help you with your diligence projects, so get in touch!