Final Paper

What-If Testing Framework

Manmadha Ambati

Senior Principal Quality Analyst

Manhattan Associates

Chaithra Rama

Technical Lead Quality Engineer

Manhattan Associates

AbstractOne of the most important business areas in any industrial facility is the Supply chain. It is a key component of productivity and it has evolved eventually. One of the most significant changes is the adoption of modern technology to enhance the efficiency and accountability in the entire supply chain. When it comes to Transportation supply chain world, having the ability to do what-if analysis is a critical requirement for making good business decisions.

What-If focuses on tactical and strategic level decision making for transportation and supply chain problems. It is a Decision-making tool which will help customers make tactical decisions based on the historical data. Testing the modeler tool which comprises of modelling, optimizing and simulating your supply chain network would be a challenging task as it involves huge amount of data and variations in usage from customer to customer.

This paper attempts to provide a generic framework what can be adopted for testing various decision making tools involving huge data set. This paper will throw light on the emerging technology tool in the supply chain world and various insights to the testing framework followed to make the tool efficient, bug free and evolve better with every change

OverviewShippers often struggle to develop robust transportation policies because it’s difficult to evaluate the effect of uncertainty and variability on their network. The problem is averages rarely reflect reality, yet most analysts still rely heavily on statistical averages.

With new suppliers and customers, store openings and closings, rate fluctuations, increasing service-level expectations, seasonality, business disruptions, and varying order quantities, every element of the supply chain is a moving target. To stay ahead, transportation professionals need a solution that can think strategically as well as execute optimally, ensuring the transportation network is always operating at the lowest cost and highest service levels.

What does ‘What If tool’ do

What if tool allows you to develop a policy that combines the best of both worlds—optimization modelling to make cost-based decisions along with the flexibility of a simulation framework to address supply chain variability and evaluate decisions over time. The technology also combines stochastic optimization and simulation, which enables analysts to incorporate known variability.

The tool empowers customers to challenge assumptions and test new ideas on transportation policies while simultaneously seeing the impact on operations. It gives an answer to complex Transportation business problems.

What-If Testing FrameworkHaving explained the tool, it is very crucial to plan a generic framework that can be adopted for testing various decision making tools involving huge data set. Here are various aspects of the framework outlined.

1. Historical DataData is very important for any analysis tool and forms the basis of testing. Testing with hypothetical data set cannot test the intricacies involved in the algorithm and make right decision.

What-If Testing

Framework

Historical Data

Data Analysis

Data Visualization

Algorithm Testing

Automation

Historical data from various customer segments are collected to form the data set for testing this tool. This would create artificial supply chain network specific to customer which will aide in testing the application as used in real time.

Once the historical data of the customers are loaded to the tool, it creates a subset of how exactly the customers network is currently operational. Optimizations can then be performed on their data and results can be compared. This gives the confidence to the tester that actual business flow has been validated on the real-time networks.

2. Data AnalysisData Analysis forms the basis of testing the Transportation business model application.It is a process of inspecting, cleaning, transforming and modeling data with the goal of discovering useful information, suggesting conclusions and supporting decision-making.

Business scenario created based on historical data is simulated with the existing setup which is marked as the baseline for benchmarking. A new scenario is simulated with the new setup via

either Optimization performed on scenario or manual manipulation. Both these simulations

are compared for assessments and comparisons. This kind of analysis on the data helps in validating the optimization flows that are embedded into the tool.

The snapshot below shows one of the ways where the metrics can be validated before and after the operations are performed.

3. Data VisualizationData Visualization is very effective tool which involves creation and study of visual representation of data. In this case maps were used to test change in the network post optimization. It gives a holistic view of the customer stores/Vendors assigned to DC, order schedules, static routes and other detailsThe accuracy of Optimization can be tested on Maps where they can be compared before and after optimization is performed. The change in the assignment of customer stores to DC, updated order schedules and new static routes can be verified. Let’s take an example of customer store assignments to DC depicted on map, mouse over the store locations would give the details of the store address, the DC from where it is sourcing from, Product details and the average demand from the store. Performing optimization on the current scenario would display a more effective way of customers stores sourcing from DC on maps.

Before Optimization After Optimization

As we can observe the way stores are assigned to DC is changed after the optimization based on the distance, average demand Nearby DC can be made operative to source out to stores around it to make the network more cost effective. By validating the maps tester can get the confidence on the validity of the operations performed.

4. Algorithm TestingAlgorithms are designed such that the problem is modeled by a set of constraints and best solution is found w.r.t objective function defined. Testing the algorithm becomes very crucial as various optimization decisions and solutions provided are validated.

Below image describes the methods used to validate the algorithms.

Sample data set created is imported to the What if tool and testing is performed. By using smaller set of sample data testers can accurately validate the flow of the algorithm which

otherwise becomes very tedious to validate the same with huge customer data.

Another effective way to validate the Optimization algorithms is by performing Pair Testing.Pair testing is typically performed between tester and developer or business analyst with both participants taking turns and testing the tool together. This method will help the tester to test the optimizations and simulations to the code level as the complete flow of the algorithms can be tested and hence nail down the root cause of important bugs surfaced. It is very effective as this would help both the members to learn more about the tool and help in narrow down the root cause of the problem/issues faced while continuous testing

5. AutomationUnit tests, sample data sets and Customer flows are automated using the Groovy framework.

Tests automated are integrated with the Continuous Integration(CI) repository which run two times a day with the latest code check-ins. TestNG - A reporting system tool identifies failures and issues immediately.

With the process of automation any regression issues can be identified hence reduces the manual testing efforts to some extent.

Below image describes the Automation cycle and the reporting system used to track the tests.

TestNG reporting approach followed to track the status of tests run .

Measurable Impact

Below are some of the key benefits of the What If Test framework: -

Functionality well testedWith the explained approach, we could release the product with good quality.

With the CI process, Regression breaks are identified Improved customer experience

Customers can achieve smooth upgrades and can exceed expected through puts right from the beginning.

Referenceshttps://www.investcalgaryregion.ca/blog/how-to-use-big-data-supply-chain-analytics

http://www.datasciencecentral.com/profiles/blogs/predictive-analytics-in-the-supply-chain

https://en.wikibooks.org/wiki/Fundamentals_of_Transportation/Modeling

Author Biography

Manmadha Ambati works as Senior Principal Analyst at Manhattan Associates for 12 years. He has 14 years of Industry experience in all phases of software Testing life cycle. Rich experience in Transportation Planning engines and understanding business challenges of transportation. Strong product development expertise using Agile methodologies. He holds interests at various industry interaction platforms UNICOM, Step-In etc.

Chaithra Rama works as Technical Lead Quality Analyst at Manhattan Associates for 6 years. She has over 7 years of industry experience as a Software Tester. Good experience in Transportation Planning engines. She has also submitted papers at Step-In summit. Good product development expertise using Agile methodologies.

THANK YOU!