How many times should an experiment be repeated?


Solution 1

The answer depends on the degree of accuracy needed, and how noisy the measurements are. The requirements are set by the task (and your resources, such as time and effort), the noisiness depends on the measurement method (and perhaps on the measured thing, if it behaves a bit randomly).

For normally distributed errors (commonly but not always true), if you do $N$ independent measurements $x_i$ where each measurement error is normally distributed around the true mean $\mu$ with a standard error $\sigma$: you get an estimated mean by averaging your measurements $\hat{\mu}=(1/N)\sum_i x_i$. The neat thing is that the error in the estimate declines as you make more measurements, as $$\sigma_{mean}=\frac{\sigma}{\sqrt{N}}.$$ So if you knew that the standard error $\sigma$ was (say) 1 and you wanted a measurement that had a standard error 0.1, you can see that having $N=100$ would bring you down to that level of precision. Or, if $\delta$ is the desired accuracy, you need to make $\approx (\sigma/\delta)^2$ tries.

But when starting you do not know $\sigma$. You can get an estimate of the standard error of your measurements $\hat{\sigma}=\sqrt{\frac{1}{N-1}\sum_i (x_i-\hat{\mu})^2}$. This is a noisy result, since it is all based on your noisy measurements - if everything has gone right it is somewhere in the vicinity of the true $\sigma$, and you can use further statistical formulas to bound how much in error you might be in the error of your estimate. There are lots of annoying/interesting/subtle issues here that fill statistics courses.

In practice, for a school project: define how you make your measurements beforehand, make 10 or more, calculate the mean and standard error, and look at the data you have (this last step is often missed even by professional data scientists!) If the data is roughly normally distributed most measurements should be bunched up with a few outliers that are larger and smaller, and about half should be below the mean and half above. If you want to be cautious, check that the median (the middlemost data point) is close to the mean.

If the data is pretty normal, estimate how many tries you need and do them.

If the data does not look normal - very remote outliers, clumps away from the mean, skew (more high or low data points) - then the above statistics is suspect. Calculating means and standard errors still make sense and can/should be reported, but the formula for the accuracy will not be accurate. In cases like this it is often best to make a lot of measurements and in the report show the distribution of results to get a sense of the accuracy.

Things to look out for that this will not fix: biased measurements (whether that is due to always rounding up, always measuring from one side with a ruler, a thermometer that shows values slightly too high), too crude measurements, calculation errors (embarassingly common even in published science), errors in the experimental setup (are you really measuring what you want to measure?) and model errors (are you thinking about the problem in the right way?) No amount of statistics will fix this, but some planning and experimentation may help reduce the risk. Biased measurements can be corrected by checking that you get the right results for known cases and/or callibrating the device. Having two or more ways of measuring or calculating is a great sanity check. Experimental setup and model errors can be corrected by listening to annoying critics (who you can then magnanimously thank in your acknowledgement section).

Solution 2

Pick a number, let's say ten. Record your measurements. Determine the mean. Determine the standard deviation. Determine the standard error. Mean +/- 2*standard error will give you a 95% certainty that your mean is accurate.

Doing a chi squared test will determine if your data distribution is acceptable.

If standard error is too high then do more trials to reduce the error. If chi squared is off then it indicates your data is skewed which likely means there's some error in your measurement process. Correct that and try again.


Related videos on Youtube

Алексей Наумов
Author by

Алексей Наумов

Updated on August 01, 2022


  • Алексей Наумов
    Алексей Наумов 3 months

    I am doing an experiment as part of a school project. In order to decrease the random error I repeat the measurements.

    How to define if I have made enough tries? Should it be 10? Or 20? Mathematically speaking the more tries I have done the better the precision is, however, this way I need to repeat the measurements an infinite number of times.

    • QuIcKmAtHs
      QuIcKmAtHs almost 5 years
      The ideal is 3 times or more
    • Admin
      Admin almost 5 years
    • lemon
      lemon almost 5 years
      The answer depends on what you're measuring. Giving some details about your experiment would help.
    • niels nielsen
      niels nielsen almost 5 years
      the answer also depends on a characteristic of the measuring device, called "gauge capability". this has to do with how accurately and repeatably your measuring device does its job. Knowing the capability of your gauge allows you to determine whether differences in your measurements are dominated by flaws in the gauge rather than real differences between your experimental measurements. My own rule-of-thumb is 5 measurements are suggestive, 10 are data, and 50 are information- but this assumes a "capable" gauge.