© S.F. Beckert and W.S. Paim, published by EDP Sciences, 2017

1 Introduction

When monitoring parameters of manufacturing processes or inspecting products specifications, it is indispensable to carry out adequate measurements. Based on obtained measures that decisions on approval or disapproval of products are taken. Thus, it is necessary to evaluate the adequacy of the measurement systems used for a required task. According to Moheb-Alizadeh [1], almost all manufacturing organizations calibrate the control instruments used to extract the required data, whereas the measuring instrument is just one component of a measurement system. Thus, the suitability of the measuring instrument alone does not guarantee the correctness of a measurement system. ISO 14253-2 (2011) [2] lists different sources of uncertainty that may affect the quality of measurement results: environment, measurement setup, measuring instrument, appraiser, measuring object, measuring procedure, physical constants, definition of the characteristic, software and calculations.

The automotive industry is one of the most challenging and competitive industries. To achieve competitive advantage in the market, car and automotive components manufacturers must ensure the product realization in the shortest possible time and at minimum costs, while achieving high quality and low prices of products [3]. It is also in the automotive industry that the greatest demands on verification of measurement systems are made.

The technical specification for automotive sector quality management systems ISO/TS 16949 (2009) [4] has become one of the most widely used international standards in the automotive industry, harmonizing the different assessment and certification systems in the global automotive supply chain. IATF 16949 (2016) [5] published by the IATF (International Automotive Task Force) is replacing the technical specification ISO/TS 16949 (2009), defining the requirements of a quality management system for organizations in automotive industry.

IATF 16949 (2016) is an autonomous quality management system standard fully aligned with the structure and requirements of ISO 9001 (2015) [6]. So, it is not a stand-alone document, but is implemented as a complement, in conjunction with ISO 9001 (2015) that must be purchased separately. ISO 9001 (2015) generally describes that the organization must provide the necessary resources to ensure valid and reliable results when monitoring or measuring is necessary to verify the compliance of products with requirements. In addition, the organization should retain appropriate documented information as evidence that the monitoring and measurement capabilities are appropriate for its purposes. IATF 16949 (2016) details as an additional requirement that statistical studies should be conducted to analyze the variation present in each type of measurement system identified in the control plan. It specifies that the methods and acceptance criteria used should be in accordance with reference manuals of measurement systems analysis. It also describes that analytical methods and alternative acceptance criteria must be approved by the customer, and records of this acceptance must be maintained.

Although the IATF 16949 (2016) presents a note that MSA studies should be prioritized for critical and special characteristics of products and processes, no distinction on the acceptance criteria is established. It also does not describe about the possibilities of not performing MSA studies for part of measurement systems described in the control plan. This requires a proactive relationship between supplier and customer, so that measurement systems analyze are conducted with the rigor required for a characteristic to be controlled. Often, the supplier is forced to exclude a monitoring feature from the control plan because the requirement of measurement system analysis is the same as that adopted in a critical or special characteristic. This is costly, time-consuming and weakens the relevance of measurement systems analysis. This work aims to evaluate the acceptance criteria established in recognized reference manuals of measurement systems analysis, and propose additional recommendations to be followed in conducting these analyzes.

2 Reference manuals for measurement systems analysis

MSA [7] and VDA 5 [8] manuals are recognized internationally. The MSA manual, currently in its fourth review, was issued by the Automotive Industry Action Group (AIAG) in 2010. The VDA 5 manual was developed and revised in 2011 by the German Association of Automotive Industry. Due to the similarity of ISO 22514-7 (2012) [9] with the VDA 5 manual, only this one will be treated in this paper. Before evaluating the reference manuals, it becomes important to define the concepts of measurement system and measurement process.

For MSA manual, measurement system is the collection of instruments or gages, standards, operations, methods, fixtures, software, personnel, environment and assumptions used to quantify a unit of measure or fix assessment to the feature characteristic being measured; is the complete process used to obtain measurements. In the case of VDA 5 manual, this set of factors influencing the measurement result is called measurement process. In the context of VDA 5 manual, measuring system is the device used for making measurements, alone or in conjunction with one or more supplementary devices. In MSA manual, the measuring apparatus is called instrument or gage.

2.1 MSA manual

MSA manual presents guidelines for assessing the quality of a measurement system primarily used in the industrial world. Three fundamental issues must be addressed when evaluating a measurement system:

• the measurement system must demonstrate adequate resolution to detect changes in product or process variation. Typically, its applied that instrument discrimination should divide the tolerance (or process variation) into ten parts or more;

• the measurement system must be stable. Under repeatability conditions, the measurement system variation is due to common causes only and not special causes;

• the statistical properties (errors) are consistent over the expected range and adequate for the purpose of measurement (product control or process control).

The location characteristics are defined by Bias (systematic error component of the measurement system), linearity (the change in bias over measuring range) and stability (the change in bias over time). Figure 1 shows the graphical representation of these characteristics.

Repeatability and reproducibility are considered for variation characteristics (Fig. 2). Gage repeatability and reproducibility standard-deviation (GRR or Gage R&R) is the combined estimate of measurement system repeatability and reproducibility.

According to Wheeler [10], GRR study was developed back in the 1960s. And in 1990, AIAG edited the first edition of MSA manual. Throughout their revisions, statistical inference techniques were introduced for Bias and linearity characteristics, and the 99% confidence level was changed to 99.7% for GRR percentile calculation. When compared to the product tolerance (TOL), %GRR is calculated according to equation (1). (1)

Perhaps one of the items that had the greatest impact in the last review was the change in the acceptance criteria regarding the adequacy of the measurement system. Table 1 presents a summary of the acceptance criteria set out in the 4th edition of MSA manual. The analysis of these parameters is relevant only if the measurement system is stable.

For Bias and linearity characteristics, they are required to be significantly equal to zero for a 95% confidence level, and the errors obtained in experiments should be less than the maximum permissible errors established for the instruments. This requirement considers that systematic errors should be practically non-existent, presenting only random errors. However, it is common for equipment to remain repetitive, but with typical systematic errors. But, when compared with the product tolerance, it represents a small portion.

For the variation characteristic of a measurement system, the acceptance criteria is 10% of the tolerance or variation of the production process. The change in this last revision of MSA manual is that, depending on the application or the costs involved, values up to 30% for %GRR could be acceptable, but it must be approved by the client. The management of these approvals is complex, for different reasons: number of customers involved for the same type of product, number of measurement systems controlled by the organization and need for greater technical knowledge on methods of analysis of measurement systems by the customer. Lack of knowledge can generate unnecessary demands on the statistical properties of measurement systems.

Another parameter analyzed is the number distinct of categories (ndc). ndc is directly associated with instrument discrimination. This statistic indicates the number of categories into which the measurement process can be divided. Measurement system is not acceptable when it cannot detect process variation. Usually ndc is calculated from equation (2), where PV corresponds the standard-deviation of the parts variation used in the study. (2)

The application of the above formula represents a risk in decision-making process. Since the parts used in conducting a MSA study are a very small sample (usually 10), they may not represent the effective variation of manufacturing process, presenting a ndc smaller than the measurement system potential. If the purpose is to evaluate the measurement system quality in relation to product tolerance, equation (2) is not appropriate.

Fig. 1 Location characteristics of measurement system. Source: AIAG (2010).

Fig. 2 Variation characteristics of measurement system. Source: AIAG (2010).

2.2 VDA 5 manual

The aim of VDA 5 manual is to summarize the requirements and procedures of the existing standards and guidelines in order to gain a standardized and practice oriented model for estimation of the expanded measurement uncertainty. In principle, VDA 5 manual was developed for the evaluation of measurement processes of geometric quantities. This manual is based on use of the Guide to the expression of uncertainty in measurement (GUM) for the evaluation of expanded uncertainty for measuring system (instrument) and for measurement process.

The GUM sets out general rules for assessing and expressing measurement uncertainty that can be followed at various levels of accuracy and in many fields, from the shop floor to fundamental research (JCGM, 2008) [11]. The standard uncertainties for every influence factor are estimated using the mathematical model of the measurement process. An uncertain budget summarizes standard uncertainties, associated sensitivity coefficients and calculated combined and expanded measurement uncertainties. According VDA 5 manual, expanded uncertainty (U) is calculated for a coverage probability of approximately 95.45%.

Since the measuring system is an essential part of the measurement process, it is therefore evaluated separately. In expanded uncertainty calculation of the measuring system, the following sources of measurement uncertainty are considered: resolution of the measuring instrument, calibration uncertainty, repeatability on reference standard, uncertainty from bias, uncertainty from linearity and uncertainty from specific influence components. The estimation of variation for linearity can be obtained from calibration certificate, manufacturer specification or experimentally, using 3 or more reference standards. The standard uncertainties for repeatability and bias are obtained from an experiment called Type 1 Study, where a standard is measured at least 25 times.

In expanded uncertainty calculation of the measurement process, typical uncertainty components are determined experimentally: repeatability on test parts, reproducibility of operators, and uncertainty from interaction(s); but also taking into account the uncertainty arising from the measuring instrument. For standard uncertainty calculations obtained experimentally, when more levels or factors are being considered simultaneously, the use of analysis of variance (ANOVA) is strongly recommended, allowing to evaluate the occurrence of interactions.

Care should be taken so that there is no overlap of uncertainty sources considered in the measurement process analysis. For the repeatability, it is recommended to consider as the source of uncertainty the major contribution between instrument resolution, repeatability in the reference standard and repeatability in parts. Table 2 presents the acceptance criteria established in VDA 5 manual for approval of measuring instruments and measurement process.

As initial criterion for the measuring instrument selection, the recommendation is that the resolution does not exceed 5% of the tolerance.

The measuring capability system deals with the measuring instrument qualification to perform a certain measurement task. The calculation for capability ratio of measuring system Q_MS is given in equation (3), where U_MS is the uncertainty of the measuring instrument and TOL is the tolerance of the characteristic being controlled. (3)

The measurement process capability deals with the qualification for a specific measurement task. The calculation for capability ratio of measurement process Q_MP is given by equation (4), where U_MP is the uncertainty of the measurement process. (4)

The capability ratio Q_MP corresponds to the percentage by which the tolerance zone of the test characteristic is reduced or extended according to ISO 14253-1 (2013) [12].

It is recommended that the capability ratio quotient for measurement systems Q_MS reaches a maximum of 15%. And the capability ratio of the measurement processes Q_MP does not exceed 30%

2.3 Difficulties and care in measurement systems analysis

The measurement systems analysis takes time and resources from organizations. Therefore, they should be conducted in an appropriate way, so that the results in the experiments can effectively express the quality of the measurement performed. Studies performed only to meet regulatory or customer requirements can be conducted under ideal conditions, not representing their actual operating condition. On the other hand, when customers demand overestimated rates, they end up making the use of viable measurement unfeasible systems.

Regardless of the statistical method applied in the analysis of measurement systems, adequate communication between customers and suppliers is essential. And this communication requires technical knowledge from both parts involved. Not only the statistical tool, but also the application of the product features. Without this, any other action is incomplete and costly. If there is positive synergy between the people involved, some aspects should be observed and can contribute to a coherent measurement systems analysis.

A basic aspect is the measuring instrument resolution. In this sense, the orientation of the VDA 5 manual is assertive, when recommending that it be a maximum of 5% of the tolerance, to assist in the quality of readings. Without adequate resolution, it is not possible to detect the differences between measures from one product to another. According to Dietrich [13], if the resolution is inadequate, the variation will be estimated too low (it often approaches zero) and a reasonable evaluation is not possible. With proper resolution, the determination of the number of distinct categories (ndc) is also dispensable, provided care is taken that the parts used in the GRR study have different measures. Otherwise, the measurement system will not be able to detect the difference between the parts, since this difference is practically non-existent.

The MSA manual warns that the use of the GRR guidelines as threshold criteria by itself is not an acceptable practice for determining the acceptability of a measurement system. This is because the GRR only evaluates the random variation of the measurement system. Therefore, the recommendation of some customers is not satisfactory, by limiting the requirement only for GRR studies. A critical and technical assessment, by type of measurement system, should be performed to define which statistical properties should be studied. Typically, Bias and GRR would be the minimum studies to be conducted.

For Bias analysis, Type 1 study presented in VDA 5 manual is appropriate because it complements the calibration results of the measuring instrument and considers the normal conditions in which the measuring system will be used.

For GRR analysis, the most appropriate method of evaluation is through ANOVA, as it also allows to evaluate the interactions between the sources of variation. García and del Río [14] states that ANOVA gives more accurate estimates of the parameters involved in measurement system analysis. Moheb-Alizadeh [1] relates the positive points of ANOVA: it can handle any experimental setup, it can assess the variance more accurately and derive more information from experimental data. For Ha [15], in recent years, ANOVA is preferred because it provides abundant information for the assessment.

However, the biggest problem of the measurement systems analysis is the definition of the maximum permissible errors.

While MSA manual states that Bias should be significantly equal to zero, VDA 5 manual allows deviations by setting Q_MS ≤ 15% of tolerance.

For the variation of measurement process (measurement system), VDA 5 manual specifies Q_MP ≤ 30% of tolerance, while MSA manual assigns a limit up to 10% for %GRR. It is important to report that MSA manual performs calculations for a confidence level of 99.73%, reasonably increasing the measurement system variation range. The measurement uncertainty studies outlined in VDA5 manual are performed for a confidence level of approximately 95%. It is worth mentioning the saying by Wagner and Lang [16] apud Dutschke [17]: “The higher the confidence level, the greater the confidence interval and the more meaningless the statement”.

Thus, the rigor established in MSA manual limiting percentage of GRR up to 10% has made impossible the use of various measurement systems, in normal assessment conditions would be suitable for the required measurement task.

The current IATF 16949 (2016) requires the use of reference manuals for analysis of measurement systems. That is, it is indifferent to use of MSA manual or VDA 5 manual. Except, of course, if there are any specific customer requirements. In this context, there should be greater flexibility for GRR acceptance percentages.

3 Impact of the measurement system on the conformity assessment

For Pendrill [18], conformity assessment is broadly defined as any activity undertaken to determine, directly or indirectly, whether an entity (product, process, system, person or body) meets relevant standards or fulfil specified requirements. In context of this work, the conformity assessment consists of verifying compliance with product tolerance, established between lower specification limit (LSL) and higher specification limit (USL). Usually, decisions about compliance with the requirement depend on the measures obtained in the product inspection.

Since the decisions are made according measurement results, and these measurements are characterized by uncertainty, these decisions may be incorrect. Such incorrect decisions are of two types: an item accepted as conforming may actually be non-conforming, and an item rejected as non-conforming may actually be conforming.

According to ISO 14253-1 (2013), if no previous agreement has been made between the supplier and the customer, the principle behind the rules for proving conformity and nonconformity with specifications is the following: the measurement uncertainty always counts against the party who is providing the proof of conformity or nonconformity and therefore making the measurement. The supplier shall prove conformity with a specification. It is proved when the complete measurement result (including measurement uncertainty) falls within the tolerance zone of a workpiece characteristic (Fig. 3).

And the customer shall prove nonconformity with a specification to could reprove a product. Nonconformity with a specification is proved when the complete measurement result falls outside the tolerance zone of a workpiece characteristic (Fig. 4). Regions reduced or extended to the tolerance field are called guard bands.

JCGM 106 [19] points out that the use of these guard bands provides a way to limit the probability of making an incorrect conformance decision based on measurement information. However, this practice does not allow the rejection of measures obtained in the zone of uncertainty by the customers, but also cannot be approved by the suppliers. Depending on the manufacturing process capability, this can generate a volume of parts that are scrapped (or reworked) unnecessarily.

JCGM 106 also lists the rule known as shared risk. Under this rule, the customer and supplier agree to accept as conforming (and reject otherwise) an item whose property has a measured value in the tolerance interval. In this case, producer and user share the consequences of incorrect decisions. And a limit for the measurement uncertainty must be established.

In the MSA manual, the parameter related to the measurement uncertainty is Gage R&R, where 6 ∗ GRR represents the variation of the measurement system to a confidence level of 99.7%. In its most rigorous condition, %GRR cannot be greater than 10% of tolerance and Bias should be significantly equal to zero. If there is an agreement with the customer, %GRR could reach 30% of tolerance. VDA 5 manual points out that the Q_MP capability ratio is limited to 30%.

In order to make an additional analysis, it is necessary to estimate the probability of incorrect decisions that may be made in the product inspection. As highlighted by Pou and Leblond [20], the risks to the customer and supplier are due the combination of two random phenomena: the production of an item on one hand and the measurement error that appears during its control on the other hand. This vision, more in line with reality, has the merit of basing the reflection not only on measurement (the business of the metrologist) but also (and perhaps especially) on the manufacturing process. Thus, flaws in the inspection are related to the specification limits, to the measurement process and to productive process.

The acceptable limits of probability depend on the risks that each company intends to run and the costs related to the productive process. Considering that several production processes and several measurement methods are represented by normal distributions, it is possible to estimate the probability of occurrence for each type of event that generates risks to the consumer and the supplier.

Based on the formulations presented in JCGM 106, by Pou and Leblond [20] and Donoso [21], the Equation presented in (5) calculates the probability for the different events. For the interest of this work, it has shown in Table 3 the limits of integration for events that generate incorrect inspections. (5) being μ_rpp is the average of the real productive process, σ_rpp is the standard-deviation of the real productive process, μ_mpp is the average of the measured productive process, σ_mp is the standard-deviation of the measurement process.

The average of real productive process is obtained through equation (6), where B is the Bias of the measurement system. (6)

The standard-deviation of the real productive process is obtained by equation (7), where σ_mpp is the measured standard-deviation of the manufacturing process. (7)

Impacts of acceptance criteria established in the MSA manual and VDA 5 manual on the probability of incorrect inspections were evaluated.

Fig. 3 Conformity with specification. Source: ISO 14253-1 (2013).

Fig. 4 Nonconformity with specification. Source: ISO 14253-1 (2013).

5 Results and discussion

To assist in performing simulations and understanding the results, the following specifications were considered for application: nominal value = 0; LIE = −10 μm and LSE = +10 μm, which leads to a tolerance of 20 μm.

Among the acceptance criteria established for measurement systems, the most rigid is set out in the MSA manual, with the %GRR value limited to 10% of tolerance and Bias significantly equal to zero. Table 4 presents the results obtained for different productive process capabilities, considering the centralized process and displaced by a standard-deviation.

As observed in Table 4, regardless of the manufacturing process capability, the results presented low probabilities of incorrect inspection.

The marginal acceptance criterion presented in the MSA manual indicates acceptable values for %GRR up to 30% of tolerance. In Table 4, it can also be observed that the probabilities of incorrect inspection remained low.

In the application of the acceptance criterion established in the VDA 5 manual, two situations were dealt with: Bias equal to zero and Bias equal to 5% of tolerance. For this case, Bias is treated as a source of measurement uncertainty. And the standard-deviation of the measurement system was then estimated by equation (8). And Table 5 presents the probabilities of incorrect inspections. (8)

According to the acceptance criteria established in VDA 5 manual, only for Cpk = 0.67 percentages above 1% of incorrect inspections were observed. However, considering that an increase of 5% may be included in the specification limits (based on MSA's narrowest criterion), the probability of occurrence of error is reduced to 0.0004.

6 Conclusions

Of course, measurement systems must be adequate for the required measurement task, and statistical studies should be performed to assess the impact of measurement uncertainty on the manufacturing process. However, care must be taken not to disable appropriate measuring systems due to the use of overestimated acceptance criteria.

Based on the evaluations performed, it was found that the application of Bias up to 5% of tolerance and %GRR or Q_MP up to 30% indicates that the measurement systems are suitable to control manufacturing processes with a capability greater than 0.67. It was also observed that the average being offset by up to 1 standard-deviation of the manufacturing process does not affect the quality of inspection. Other specific cases can be analyzed based on the formulations presented above.

Based on the evaluations performed, the recommendation is that the acceptance criteria should be established for the normal conditions of the manufacturing process. This would avoid unnecessary treatments for conditional approval of the measuring system where it is already proven that it adequately meets the measurement conditions. In this way, measurement system suitable for inspection task would not be rejected for use.

On the other hand, in conditions that require a high degree of accuracy (or not), acceptance criteria would be agreed directly between customers and suppliers. That is, it would be up to the client to specify items that require more stringent acceptance criteria. And only in extraordinary situations, it would be up to the supplier to demonstrate that a measurement system with the highest measurement error still performs measurement task correctly, whether due to the high capability of the manufacturing process or due to the use of guard bands.

References

All Tables

All Figures

	Fig. 1 Location characteristics of measurement system. Source: AIAG (2010).
In the text

	Fig. 2 Variation characteristics of measurement system. Source: AIAG (2010).
In the text

	Fig. 3 Conformity with specification. Source: ISO 14253-1 (2013).
In the text

	Fig. 4 Nonconformity with specification. Source: ISO 14253-1 (2013).
In the text