Which model gives you the best result based on your KPIs? How can you automate finding the best plan from a non-deterministic solver? With ensemble runs on Nextmv, the best plan (or run) is surfaced with all the supporting context you need to share results and make the best decision for your business. We’ll walk through what makes up an ensemble run, some real-life examples, and how to get started.
What are ensemble runs for decision models?
Ensemble runs evaluate multiple solutions with different models and technologies against the same dataset. How does it find the best run? Ensemble runs use your rules to rank the quality of each run and automatically reach a consensus (or choose the best run/plan) from those parallel runs.
When making a single run, it looks like this: you have an input, you use a specific instance of your model, run the model, and the model returns an output.
With ensemble runs, you can use that same input, create run groups (that outline the model instance, options, and repetitions), perform the runs, automatically choose the best run based on your predefined rules, and then return the output for that run.
Now that we understand the structure, let’s discuss why and when to use it.
When should you use ensemble runs?
Whether you’re working on a new model, exploring solvers, or deciding which model to promote to production, ensemble runs make finding the best plan as simple as selecting your criteria and kicking off the runs. Let’s look at a few use cases in more detail.
Converge on the best run (plan)
Find the best run from a model if you’re using the following:
A non-deterministic solver
If you’re using a heuristic, stochastic solver, or other type of non-deterministic method with your decision model, ensembling surfaces the best run from multiple repetitions using the same solver.
For example, Nextroute, Hexaly, Seeker, and VROOM are types of solvers that would fit into this category and would be good candidates for run ensembling. Create one run group that uses the same model (instance) and solver (option) and repeats it multiple times.
Multiple solvers
If you’re evaluating multiple solvers with your decision model, ensembling will help you select which solver to use by highlighting the best performing run across all the solver choices.
For example, if you’re using an OR-Tools MIP model, you may want to compare SCIP to GLPK by running them both and identifying which produces the best run. Create two run groups using the same model (instance) and different solvers (options).
Different model config variations
If you’re tuning your model with different configuration options, ensembling will automatically pick the model config that delivers the best run.
For example, if you’re solving a VRP, you may want to see if a run time of 5 seconds (vs a run time of 2 seconds) consistently delivers better plans. Create two run groups using the same model (instance) with different configurations (options).
A scenario test performs a similar function, offering a comparison where you can analyze various metrics. However, scenario testing isn’t opinionated, meaning it doesn’t highlight which run performed the best according to your definitions. A scenario test is better used earlier in the workflow for initially understanding how a model’s performance reacts to variable changes prior to creating ensemble runs.
Converge on the model that delivers the best plan
Which model returns the best plan? If you’re evaluating multiple models, set up your ensemble run definition to use different model instances. If you’re wondering whether to promote a new model to production, you can perform a series of ensemble runs as a rollout strategy – seeing how often a run from each model (baseline vs candidate) gets selected. Similar to a switchback test, using ensembling will allow you to compare the two models. Instead of the models making production decisions as they would in a switchback test, ensembling will run the models in parallel to find the best plan.
For example, let’s say your team is evaluating two different types of model approaches to route balancing: adding a constraint to the model or using an objective function that includes route balancing. You want to find the model that returns the plan with the shortest travel duration, the fewest unplanned stops, and the fewest maximum stops per vehicle. Your team runs both models in parallel using ensembling to automatically find the best run. You can repeat this to see if the same model is performing best each time.
Create two run groups using two different models (instances).
Validate a simple backup model
What do you do if your production model starts returning poor solutions? You can validate a simpler model using ensemble runs that use both the production model as well as a simpler (backup) model. While a shadow test on production data also allows you to see how a candidate model performs vs the baseline model, ensembling goes a step further, surfacing the best run based on your rules.
For example, if your inputs are getting bigger and your MIP model is suddenly not finding good enough solutions in the allocated solve time, you can perform ensemble runs with the MIP model and a simple heuristic model. If the heuristic model delivers the best runs, you can use it as a fallback while the team troubleshoots the more complex model.
Let’s dive into a detailed example!
Example: Choosing the best vehicle routing plan for delivering produce
At Nextmv, we like to put things in the context of our fictional farm share company that delivers produce from local farms to customers’ homes. We have a fleet of vehicles and a team of drivers who pick up vegetables, cheese, and other perishable goods from our partner farms and deliver them to customers who have placed an order the previous day. It’s Sunday evening and we need to plan routes for Monday. All customer orders have been placed and we know how many drivers and vehicles are available for tomorrow’s deliveries. Based on produce orders and driver availability, we want to find the best delivery routes that have the lowest solution value (minimizing time on the road) and the fewest unplanned stops.
Our team created a decision model that incorporates constraints such as compatibility attributes (e.g., ensuring dairy products are on vehicles with refrigeration) and uses Nextroute, a non-deterministic VRP solver to find route plans. We know that with a non-deterministic solver, we may get slightly different solutions when running the model each time. We’ve found that by repeating the run, slight changes in the returned solution can result in a more efficient delivery plan. To find the best plan in the time we have, the team creates an ensemble run definition that uses the latest version of our routing model, solves it with Nextroute, and repeats the run 10 times.
Next, the team defines the rule(s) that will be used to select the best run. In this case, we’re minimizing both solution value and the number of unplanned stops.
We perform the ensemble runs and below are our results. The UI surfaces the best run at the top, signaling that this run was performed the best according to your rules (smallest solution value and no unplanned stops). The UI also identifies which runs were within tolerance, and shows all the child runs underneath.
If we click on the selected run, we’ll see more details like the route on a map, plan details, and metadata. Below we see our plan’s solution value (which was the lowest of all the runs) and that it has no unplanned stops.
We can now share these results with operators and other stakeholders to discuss what the routes look like and why these routes were chosen for tomorrow’s deliveries (by analyzing solution value, travel duration, and number of unplanned stops for the selected plan).
How to get started
Choosing the best plan for your optimization problem involves more than finding a feasible solution. There are other KPIs to consider, models and solvers to choose from, and stakeholders to sync with before moving forward. With ensemble runs, you can easily evaluate multiple runs in parallel and find the best plan faster. Ready to dive in and try out ensemble runs? Sign up for a free Nextmv account and start a free trial.
.png)
