The path to production for decision models may not always be a straight line. But that doesn’t mean it has to be a persistent maze of twisty passages. More often than not, a combination of bespoke tooling, unclear steps, ad hoc testing, and time-consuming handoffs are what stand in the way of realizing decision model value. Applying a few best practices goes a long way to increase your success of going from local development to remote testing and production environments.
In this blog post, I’ll recap six decision model best practices Carolyn Mooney and Tiffany Bogich highlighted in this techtalk. (Worth a listen on a walk or during a lunch break, if you prefer!) Let’s dive in.
1. Treat models like software applications
If there is only one thing you take away from this post, it is this: Treat your decision models like software applications. Why? Because software stacks are where decision models will ultimately live and provide value.
The software mindset among decision practitioners is still a growing space. It often requires going beyond what is traditionally taught in academia. As an Engineering and Data Science Manager at Grubhub said in a separate conversation:
“It comes down to this: You will be iterating. That is very different from when I first started dabbling in OR in school... Being prepared for that iteration is going to be important, especially when you sit in an engineering organization. That is how all DevOps thinking has been oriented. It’s not ‘Plan it in advance and build it’. It’s ‘Take some steps and then figure out what the next step might be.’ It's a total mindset shift from what you learn when you're studying OR.”
It means learning Git and using IDEs. It means thinking in APIs and not just MPS files. It means recording rich run history that includes unique IDs, time stamps, who ran it, and the version. It means thinking in structured workflows that are extensible to the entire decision technology ecosystem because decision teams are multi-solver, multi-framework, multi-tool, multi-user, and multi-use case.
Modelers who skip over this software mindset risk their project getting shelved. Run failures are hard to track and troubleshoot. Updates are challenging to test, roll out, and roll back. Collaborating on anything involves epic email threads, sifting through attachments, and many meetings.
What can you do about this? Follow me to the next section.
2. Keep model code and input/output separate
One integration pattern we see among teams is decision models are embedded directly into the service code. The good news is modelers are working directly with software teams in a real way. The bad news is there is added complexity for making changes (planned or reactive) to the model code. Modelers lose autonomy and agency to efficiently test, manage, and troubleshoot their decision assets. Model changes involve redeploying the entire service code instead of just a subset of changes. This often puts extra burden on software and modeling teams looking to be agile.
For example, when a system failure occurs, it’s hard to pinpoint the issue. Modelers can’t replay model runs outside of the service architecture. When they try, they have to pair with an engineer for what can take hours or days to find and piece together input files, stage runs for comparison, and share results. And after all that, the issue may actually be upstream or downstream of the model – not the actual model itself.
The action to take here echoes the best practice above: Your model should be structured as a microservice. Its job is to read input, make a decision, and write output. This means the upstream data collection and aggregation processes (e.g., querying databases, calling third-party APIs, etc.) and the actual decision code are separate and distinct layers. Lastly, you should have a mechanism for storing data files passed into and out of your model API. Doing so makes cloning a run to replay or troubleshoot something you do in seconds and not hours, days, or weeks. As Tiff said in the original presentation, “Splitting those layers makes the system highly maintainable and easier to iterate on in the long run.”
3. Make parameters configurable, not hard-coded
As it’s been said: The only thing that’s constant is change. People will want the knobs and levers to modify configuration. In the world of decision models this equates to shift splits, solver selection, prices, and coefficient values. A one-configuration-that-you-can't-change approach doesn’t cut it.
The practitioner community is often pre-disposed to this approach because many demo or sample models are set up this way. (That should change, by the way!) Those who do account for this and make parameters configurable are still faced with the challenge of exposing them in a systematic way. Changing configuration often requires a lengthy redeployment, impeding the planner-modeler-software loop and adding friction.
As Carolyn said in the talk on this topic: “[End users] can feel handcuffed by automation, but they need to leverage it to impact costs, revenue, or the bottom line. You want to make parameters available to them so they can develop good intuition about the decisions and what is possible.”
To avoid this, make your model read configuration or parameters from the environment or the command line. First, scan for hard-coded numbers (especially in objective function coefficients). Then, expose them as parameters so you can tweak them down the line. Consider setting defaults that work most of the time, but include a mechanism for safely overriding and validating ranges.
4. Make model versions accessible
Repeat after me: Managing model code versions is not the same as managing reuse of model versions. Git tooling (e.g., GitHub and Gitlab) isn’t a replacement for DecisionOps. Hold onto this thought.
How often do you have three or four different formulations or versions of a model live at any given time? Perhaps pay rate configuration is one value in Region A and another in Region B. Or you have a MIP running as your primary model, but you have a heuristic fallback should the first not return a feasible solution. These are all examples of managing model version reuse across different environments.
I once saw someone with 10 separate files representing the model code for the same problem type across different regions. I still wince at the thought of how that changelog gets tracked and managed. This creates bottlenecks and confusion. Changes get out of sync. Rollout and rollback are challenging. The modeler becomes a bottleneck. Software gets pulled in to support. (Sounds like a familiar refrain from Best Practice #2, right?)
While Git practices are not the end-all solution to this area, they are fundamental to it. The real solution you’re after is a model registry where versioned model artifacts are pushed, logged, and accessible. Plus, if you follow the model configuration best practice, you have infinitely more control over which version and configuration is deployed to a particular subset of your planning space, be that specific a geography or subset of your customer base.
Now, when you’re operating in a world where you have three or four different model formulations live at any given time, model version rollout and rollback are a lot easier and drive more value back to your business.
5. Standardize your model testing
The importance of model testing is generally embraced among practitioners. However, having a standardized testing framework or flow doesn't seem to be as prevalent. The challenges: Going from unit test to regression test isn’t a consistent progression. Online shadow or switchback tests are ad hoc and difficult (if possible at all) to share or reproduce. Test inputs are not organized or consistent. Conditions aren’t logged. The results are not where you thought they'd be.
The keys to making testing successful start with having a documented test plan – from unit tests to integration tests to regression tests on historical data to online tests happening on production data. Use the same test types and result presentation across all models. Create managed test sets and a process for updating them. Lastly, log all local and remote tests in the same way. This makes it easier to compare against baseline metrics in the future, leading to fewer surprises.
As Carolyn noted in the session, “...you can take runs from your production history to keep your test sets up to date. Having a way to programmatically take historical production runs and use them for testing helps address model drift. If you are testing against current production data, you can anticipate scaling issues. In rapidly growing systems, you need to test against larger inputs over time to know when you will start hitting limitations in solve time or memory.”
6. Consider the environment you’ll run in
The patterns we commonly see when it comes to model environment considerations include things like a misalignment between model run time and service run time, inflexible compute setups, or discrepancies in package versions or Python versions. These issues often result in excess costs and wasted time.
Having appropriate compute setup and flexibility is critical. Know what type of compute is needed and when. Running a model with a low memory footprint on premium compute results in excess cost. Understand where you hit diminishing returns for your solutions so you don’t run a model longer than necessary. Match the use of your model to your business process or SLAs. If dispatch needs an answer in under 3 seconds don’t run your model for 4 seconds.
These may seem obvious, but when teams are juggling so many requirements, it’s possible for these mismatches to slip through and cause challenges. And they’re not just limited to early project design, they are dynamic and changing aspects to successful decision modeling.
Go forth and best practice!
We’ve covered six best practices derived from our collective team experience and industry observations. In the spirit of starting small, I’ll leave you with Tiff’s and Carolyn’s responses to where they’d recommend practitioners start.
From Carolyn: “The best place to start is with configuration. Extracting configuration and parameters from your model code opens the door to experimentation and building a solid test suite. Thinking about where you should have configuration and levers, such as allowing people to turn constraints on and off or extracting thresholds, is a fantastic place to start. It introduces the core concept of keeping input and config separate from the model code itself."
From Tiff: “Treating models as services is the biggest gamechanger, along with pulling out input data. We have seen embedded inputs make things incredibly difficult for teams. Separating those layers into distinct services makes a massive difference.”
Another step to consider taking is working with the Nextmv local experience, which defaults you to best practices and structures that translate well into remote model runs. Here, a little structure goes a long way.
If you have best practices we should explore in future blogs and techtalks, let us know on LinkedIn!
May your solutions be ever improving.
