The qualitative scoring does not allow researchers to compare the quality of the control system. It only states how well the control system handles the dynamic scenario (i.e., well: 2 stars, average: 1 star, or poorly: 0 star). This study can be completed to quantify to what degree the handling is effective. For example, if two control systems faced with the same dynamic scenario are rated with 2 stars, the resulting deteriorated makespan can be higher for one than the other. The degree of robustness is quantified in this step. This quantification is based on the results obtained with the dynamic scenarios weighted by the performance deterioration and evaluated with the reference scenario #0. This quantification also takes into account whether or not the objective function minimized or maximized.

If a minimized objective function is the goal, the QuanTitative Score (QTS) formula must be used:

If a maximized objective function is the goal, the following formula must be used:

where

• gi is the grade of scenario i (0, 1 or 2 stars)
• Opref is the value obtained in the optimization criteria in the reference scenario
• Opi is the value obtained in the optimization criteria in the dynamic scenario i
• Opbest is the overall best result obtained for the reference scenario #0.

With this formula, the best overall score is 30 but is unreachable. It corresponds to the situation in which the control system is awarded with 2 stars for each dynamic scenario, no performance deterioration occurs in all these scenarios, and the result for the reference scenario #0 is the best one.

Evaluating the robustness of a control system in the dynamic stage is not simple and is a hard research problem in itself. Our evaluation method can be discussed and improved, especially if the quantitative score requires the best solution (i.e., the optimal one). It has the advantage to be a first evaluation method for robustness, and researchers are encouraged to use it and improve it. This can open up new interesting research areas. Of course, this evaluation can be skipped by researchers using this benchmark or can be stopped at the second step.