A hypothesis tests can result in one of two errors:
Reject the null hypothesis when it is in fact true. This is a Type I error.
Accept the null hypothesis when the alternative hypothesis is in fact true. This is a Type II error.
The probability of making a Type I error is equal to the significance level of the test, by the very definition of the significcance level.
The power of the test is the probability of correctly accepting the alternative hypothesis. This probability is one minus the probability of making a Type II error.
In multiple testing, for example when many groups are compared in a pairwise manner, the probability of making a Type I error in at least one of these tests becomes magnified.
The family wide error rate is defined as the probability of incorrectly rejecting at least one of the null hyopothesis. In multiple testing, a value for the FWER is usually specified. This is then used to imply the signficance level for the individual tests.
An alternative to mutiple two sample -tests when we want to compare means across three or more groups is the ANOVA.
Given groups, ANOVA is used to test
vs.