Key Concepts About Experiments
About the Experimenter and Optimizer
After you've built and successfully validated a current-state model of your business system, you can move on to creating the ideal future-state model. (See Current vs. Future State Models for more information.) The Experimenter and Optimizer tools will help you design, test, and ultimately find the optimal business system.
The Experimenter is a tool that runs the same simulation model multiple times, changing one or more variables each time to see if the results are different. The Experimenter also includes the Optimizer, which can give you a high-level overview of the conditions that will produce the best possible solution for the problem you are trying to solve through simulation.
For example, you could design an experiment to discover the optimal or minimum number of employees needed at a particular workstation. You could run different experiments with the same simulation model with a varying number of employees at the work station. Then you could compare something such as the utilization rate for the employees and the impact on throughput or delay times. This information would help you to make hiring decisions for your business based on your desired outcomes and priorities.
In this same example, you could also run the Optimizer to find the optimal range of employees versus the optimal number of machines needed for a given process. It could identify several possible solutions to find the optimal balance between two or more trade-offs (such as throughput versus cost).
Key Terms
The following is an explanation of some common terms and important concepts related to the Experimenter:
Experiment
An experiment is a group of variables, replications, and scenarios that will be run one after another using the same simulation model.
Variable
A variable is a simple aspect of a simulation model that will be different in each scenario. You can test a number of different variables in your model, such as:
- Varying numbers of employees, workstations, or pieces of equipment in your simulation model
- Different flow item arrival schedules or patterns
- Different model layouts
The following image shows an example of a possible set-up for an experiment in which you are trying to find the optimal distance between two machines to maximize throughput:
Notice that each variable is a separate row. The first variable is the position of Processor2 and the second variable is the position of Processor3. There is an additional variable for the number of employees who will work at this workstation.
Scenario
A scenario is a set of one or more variables that will change when a simulation model is run. The scenario is where you define exactly how the variable will change each time the model runs.
The following image shows the same example used in the preceding section:
Each scenario is defined in a separate column. In this example, the Experimenter will test 5 different scenarios for each variable. Each scenario defines a different position for Processor2 and Processor3. Notice that the number of team members won't change in the different scenarios.
Replication
A replication is a single run of a simulation model for a specific scenario. If desired, you can decide to run only one replication of a given scenario, but it would probably be better to run multiple replications to get more accurate statistical data.
Your simulation model likely has a few different points at which it uses random numbers to create variability in the system. Each time the Experimenter runs a replication of a particular scenario, it will use a specific number stream to generate any random numbers that are required by the simulation model. For example, the Experimenter will use a specific random number stream if the simulation model requires flow items to arrive in random intervals or if a particular process takes a random amount of time to complete.
Each replication will have slightly different results because each one will use a different number stream to generate random numbers. The more replications you run, the more your results will be statistically reliable because predictable patterns will start to emerge.
One important thing to keep in mind is that the Experimenter will run an identical series of replications for each variable being tested. When a replication is repeated for additional scenarios and variables, it will use the same number stream to generate random numbers.
The following image shows an example of what an experiment looks while it's being run:
In this example, the Experimenter is running 30 replications of the five different scenarios. Each scenario is testing one a different model layout for the distances between two processors. The Experimenter is using 30 different combinations of random number streams to generate the random numbers used in the first scenario of the first variable (Scenario 1). It uses that same combination of 30 different random number streams to test the second scenario (All Left), and so on.
The Experiment uses the same patterns of random numbers for when testing each scenario so that you can accurately compare the results of the first replication of each scenario because they each used the same random number stream for their first run. The more replications you run, the more confidence you can have that the statistical results are accurate.
Performance Measures
Performance measures, also called key performance indicators (KPIs) or metrics are the statistical outputs of a given model. Performances measures provide hard data about the model's outcomes.
There's no point in running an experiment if you're not going to measure a specific outcome. You can't compare the results between two or more experiments unless you can compare an output of some kind. You need a metric of some sort to be able to determine which business system leads to the best outcomes.
Before running an experiment, make sure you have identified your key performance indicators. (See Determine Your Key Metrics for more guidance.) Then, make sure that you have designed a way to measure these metrics using some kind of statistic. You can use FlexSim's standard statistics or create your own custom statistics using the Statistics Collector. (See Key Concepts About Getting Data for more information.)
Best Practices
The following sections explain a few best practices to keep in mind when designing experiments:
Make Sure Your Model Was Built With Good Data Inputs
You need to ensure that you put good data into your model (inputs) in order to get good data out of your model (outputs). Make sure that your simulation model relies on accurate data and random number distributions that accurately represent your business system's requirements. (See Data Gathering Strategies for more information.)
Validate Your Model to Reduce Errors
Try running your simulation model at least one time from start to finish before you run an experiment. Once your simulation model can be run without any errors, you are ready to use the Experimenter.
Limit How Many Key Metrics You Measure
If you are studying too many key performance indicators, you will reduce your confidence level that all of the key metrics are being represented accurately in the simulation model. The more key performance indicators you have, the more you increase the statistical probability that one of them will be wrong.
Know Your Computer's Memory Limits
The more experiments or optimizations you run, the more memory it will require from your computer. The Experimenter and Optimizer will use all your cores unless you limit the number of cores on the Advanced tab. You might begin to experience problems if you are using lots of cores. Each core requires as much memory as a single model run. For example, using 8 cores means that it would require 8 times as much memory as a normal model run.
For that reason, make sure you are familiar with your computer's capabilities. Design your experiments that your computer can handle without causing an overload or taking longer than you wanted it to take. Running more experiments and optimizations can increase your confidence in the results you are getting, but you need to weigh the need for good data against your computer's capabilities.
How the Optimizer Finds Solutions
The following sections explain a few of the important key concepts and caveats to keep in mind when using the Optimizer.
The Optimizer Finds a Range of Possible Solutions
Sometimes when people use the Optimizer, they expect it to give them one single ideal solution for their business system. While the Optimizer can provide that kind of a solution, it's actually better to use the Optimizer to generate a range of possible ideal solutions.
The following image shows the result of an Optimizer run with the range of ideal solutions highlighted in purple:
Having a range of possible solutions rather than one solution gives you the ability to weigh the pros and cons of different solutions given the constraints of your business system. It helps you to choose between different trade-offs and determine which ones will provide the best value given the resources that you have available.
The Longer You Run the Optimizer, The More Solutions It Will Test
The Optimizer will explore more solutions by running simulations of different scenarios. The longer you let the Optimizer run, the more potential solutions it will explore. When deciding how long to run the Optimizer, you should take into account whether you'd prefer to have a high volume of potential solutions. The Optimizer can help you explore solutions that you might not have thought to consider, but after running for a while it might begin to test solutions that are increasingly unconventional.
You also need to consider your computer's processing capability when deciding to run the Optimizer for a long period of time. If you want your computer to devote a substantial amount of time to exploring solutions, you might not be able to use the computer while it's doing so. You can tell the Experimenter or Optimizer to leave one core free for other computer activities on the Advanced Tab if needed.
The Optimizer Will Use the Experimenter as a Guide
The Optimizer will use the Experimenter as its guide while determining which solutions to test. If needed, you can set the Optimizer to begin by testing the scenarios that you specified in your experiments and then explore similar solutions branching out from there.