AI for Workforce Intelligence, using third-party APIs, dealing with cost implications of cloud providers & more
This week on Towards Scaling Inference, we chat with Nima Rad, CTO at Mosaic App
Nima is the CTO of Mosaic App, an AI-Powered Resource Management and Workforce Intelligence Software based out of New York-based that launched in 2018. It enables real-time collaboration for work planning and provides data-driven resource planning, reporting, and forecasting for project-based businesses. With powerful integrations, Mosaic optimizes who is working on what, and when, and delivers complete business visibility within one interface.
In this episode, he discusses how AI is used to personalize decisions based on forecasting and historical data. He also describes their process for training AI models, evaluating their performance, and addressing challenges during development and deployment. He advises organizations to stay on top of market trends and to consider buying AI solutions rather than building in-house.
Prefer watching the interview?
For the summary, let’s dive right in:
Q: Could you provide some specific use cases of how Mosaic's AI technology has helped your customers?
At Mosaic, we help our customers make better data-driven decisions. These decisions are now being boosted and enriched by AI tools. Specifically, in our case, AI is helping our customers make better decisions based on what they're forecasting or what they've seen so far. They want to adjust their existing plan to match the changes that have occurred, and these decisions are personalized based on what they care about at that time. They may be about schedules, financial points, or staffing.
Q: Is Mosaic's AI-powered software built on top of third-party APIs, or is it a custom solution developed in-house?
That's a great question. Nowadays, it's very difficult to start something from scratch in AI without utilizing existing services. AI is a fast-paced field, and being ahead of the market with your investments can make you look great while being late can make you look like you should just scrap all your work and go for an out-of-the-box solution from a service provider. The key is to adjust as quickly as possible. So, if something you're building is now offered by a third party, it's best to just jump into it. We constantly evaluate our in-house models and investments and make adjustments accordingly.
💡 For instance, initially, we built our solutions for recommendations and text inference on top of open-source libraries and models. However, we soon realized that it required heavy investments for only marginal benefits. Therefore, we decided to switch to out-of-the-box solutions and redirect our investments to areas where there are no alternative solutions.
Q: Could you walk us through the process of training an AI model for your use cases, from data collection to deployment?
To better understand how we deploy our models, let me provide further information about the problem we are solving. We help customers better understand how their project timelines and budgets are shifting by utilizing historical data for each individual customer. The data is segmented by different customers to ensure privacy. We use this historical data to train a model specific to each customer's needs, as the projects they work on differ from those of similar companies due to differences in staff, scheduling, and budgeting data.
💡 We begin by training our model on old data and then utilize data from the next year to see how accurate it is. As we move closer to the present, we expand the training data and incorporate staff and personality skills to improve the schedule and understanding of whether a project timeline is at risk. However, we faced the challenge of unclear connections between different data points in our dataset, resulting in poor performance when training our deep learning models. To solve this, we moved towards hyper-optimization and adjusting coefficients based on live usage data.
In our specific case of resource planning and workforce intelligence optimization, we believe it is better to ask specific questions rather than leave it to the deep learning model to come up with answers. For example, if we need to determine whether we can replace a senior Java developer with a junior, the answer depends on many data points that we may not have. The deep learning model would need extensive training to provide accurate answers, which can be a heavy investment. Instead, we use hyper-optimization to generate coefficients and guide our AI in solving the problem.
Q: What metrics do you use to evaluate the performance of your AI models, and do you use these metrics to optimize and refine the models?
One of the advantages of resource management and project management practices is that the associated data has been digitized over the past 10-15 years, and many companies already have a source of data. Therefore, you can use the same historical data to compare the decisions made by those in charge of starting a project, selecting project members, and determining the tasks for 2018, 2019, and 2020. You can then compare the results between different in-house models.
💡 Training each of the models was a major challenge for us, as we could not frequently train the models across the entire data set due to the cost implications on our cloud services. However, we approached this problem in a few different ways. First, we segmented the data into specific verticals, such as per industry. We wanted to see if the differences between the two-time series models were based on the industry, as providing more information might help one vertical but not the other. Second, we segmented the data based on time and across industries.
However, the results were not what we expected. We thought that feeding more classified information about project members and their skill sets would lead to better results, but it was actually the opposite. We discovered that we had too many zeros that AI deep learning models couldn't differentiate between a zero due to a lack of information and a zero that should be zero. These experiments, which took place about a year ago, led us to move toward hyper-optimization.
Q: What challenges did you face during the development and deployment process of the AI models, and how did you address them?
Regarding the retraining, we found that we didn't need to do it more than once a month. In fact, even once a month was probably too frequent based on the amount of data we had. The model was able to make decisions with the available data.
In terms of deployment to production, one challenge is ensuring adequate response time. In the developer environment, the model is trained on a subset of data, and responses are received in a way that developers can easily handle, often by adding additional query attributes. However, in production, the number of queries is much higher, sometimes by orders of magnitude. We sometimes experienced issues with warmed-up services not being warmed up enough, and we had to adjust the way we sent queries.
💡 One specific challenge was with Redshift, which was not ideal for storing our enriched data about features due to the way it puts locks on the table. When multiple customers are changing data and you need to write and adjust attributes and features on the table, it becomes unfriendly to many writers. Our challenges were mainly related to utilizing the already trained model and enriching queries based on the required features, rather than specifically training the model. And all of this had to be done within a short amount of time.
Need guidance? Have questions? 🤔
Q: What benefits have your customers seen since implementing the AI-powered solution, and how have they measured the ROI?
I need to answer a question using an analogy, so let's consider Gmail. When you want to send an email and start typing in the "To" field, you may see suggestions based on models that are giving you suggestions. To measure the effectiveness of these models, you might consider how many times you need to scroll down the list or add more characters before finding the right recipient. From a user's perspective, this may not seem like a significant improvement, but it does make the product more "sticky."
Our customers don't necessarily feel or understand the impact of AI in every operation, but they can point to the areas where it's used to make the application more personalized and sticky. We measure the performance of our models based on this stickiness factor, rather than a specific increase in productivity. As AI becomes more ubiquitous, customers will expect it to be present and simply want to get the job done.
For example, when setting up cloud operations, customers will assume that there are models and automation in place to detect anomalies and inform them of any problems. This assumption will be the sticking point for customers to stay on a DevOps platform.
Q: What advice do you have for organizations that are considering incorporating AI into their processes, and what are some common pitfalls they should avoid?
There are two types of organizations that use AI: those that use it to enhance an existing product or service, and those that work on hardcore AI solutions. While there are many ways to segment organizations, these two categories are the most common. For organizations that use AI to enhance their service, the pitfall is in trying to build too much in-house without looking for a solution. Even if there is no existing solution for the service you want to build, it may be better to wait and find someone who can do it for you. Building something in-house could be a distraction, and after months of investment, you may find that what you've built is nice but not helpful. In such cases, you will need to decide whether to continue investing in the project or absorb the losses.
For leaders and people building services that want to utilize AI, it is important to stay on top of market trends and find new services that can help them achieve their goals. Instead of building in-house, consider buying a new service that can help you achieve what you want to do. If you are asking whether you should buy or build, it is often better to buy.
For organizations that want to build hardcore AI services, the market is different. You need to find your niche and make investments that will move the needle for your customers. One specific niche that has caught many by surprise is the use of generative AI in DevOps. Building a hardcore model can have huge benefits, and many companies are looking for such services.
💡 When building hardcore AI services, it is important to invest more in datasets that can be used to train data. You will face challenges in terms of how useful your services are, so you need to have guardrails in place. The trend is towards niche market-specific use cases for hardcore AI services, which other companies will utilize. There are many cloud providers that will utilize such services if you build them.
Q: Finally, What are your top three Generative AI companies or companies who have successfully incorporated AI in their workflows and why do you like them?
It's challenging to pick the top three, but GitHub comes in first. It's amazing how they leveraged generative AI solutions early on and utilized their knowledge to build products around developers, putting them ahead of the market.
💡 Another promising use case, which is not yet out, are companies and products using AI to answer questions on their own data sets. We're exploring different services to build a chatbot that can help our customers find the answers they need from our help docs. Microsoft, for example, has early access to GPT4 models and uses Visual Agents to save significant time and headaches. Even if these models are not 99.9% accurate, they still save a lot of time, especially when dealing with Non-LLM generated VI's, which are helpful but not yet perfect. It will take some time for AI to generate more accurate results, particularly in the area of numeric forecasts. We hope someone will build a generic model that can help us instead of building it in-house.
Thanks for reading. 😄