On one method of comparison experimental and theoretical data

The method for statistical comparison of data sets (experimental and theoretical) is discussed. The method now is in development. The key parts of the method are presented in the paper.


Introduction
The new approach for comparison of data, namely, method for statistical comparison of histograms was proposed in papers [1,2].This approach also was applied for statistical comparison of dependencies [3].Some of ideas of this approach are used for many practical applications.For example, the modification [4] of this method is used for detection of the changing of parameters in the context of wireless transmition.Corresponding formulae with reference to method for statistical comparison of histograms are used in studies [5,6].In the paper the key parts of the approach are presented.The statistical duality and confidence densities are discussed in Section 2. Section 3 is devoted to notion "significance of difference".The comparison of histograms and the comparison of dependencies are presented in Section 4. Section 5 contains the conclusion.

Statistical duality and confidence distribution
The reconstruction of a parameter by the measurement of a random variable depending on the parameter is one of the main tasks in statistics.In statistical inference, the concept of a confidence distribution and, correspondingly, confidence density has often been referred to as a distribution function on the parameter space that can represent confidence intervals of all levels for a parameter of interest.In this Section, the notions of statistically dual distributions [7] and confidence distributions [8] are discussed.

Statistical duality
Definition: If a function f (x, λ) can be expressed as a family of probability densities for variable x with given parameter λ, p(x|λ), and as a family of probability densities for variable λ with given parameter x, p(λ|x), so that f (x, λ) = p(x|λ) = p(λ|x), then distributions with these probabilities densities have a property of statistical duality and they can be named as statistically dual distributions.
This definition is a purely probabilistic (and, in this sense, a frequentist) definition.Nevertheless, statistically dual distributions considered also belong to conjugate families defined in the Bayesian framework (see, for example, [9]).
This property take place for several statistically dual and statistically self-dual distributions, for example: -Cauchy versus Cauchy (b=const) -Laplace versus Laplace (b=const) The notion is introduced in note [10].It is used in paper [11] for construction of a unified approach to measurement error and missing data.Also, this notion is used for analysis of household sizes in paper [12].Authors of paper changed their notation Poisson-Gamma frame in their previous papers to our statistical duality.We used [7] the statistical duality to prove the uniqueness of confidence density of parameter via construction of corresponding confidence intervals for parameter.

Confidence density
Let us construct the bidimensional function (here x! e −λ (Fig. 1 left side).Let us x is a random variable and λ is a parameter.Then put value x = 4 (for example, the number of observed events equals 4).The function f (4, λ) (i.e.Gamma (1,5)) is a confidence density of parameter λ.If chose the upper limit (λ 2 ) and lower limit (λ 1 ) along λ axis, we can construct any confidence interval for parameter λ if x = 4 (Fig. 1 right side) which contents the true value of parameter with given confidence (here 90%).
The identity does not leave a place for any prior except uniform in construction of confidence intervals for parameter of Poisson distribution [7,13].It means that Gamma(1, x + 1) is the confidence density of parameter λ if we observed x events in Poisson flow of events.
The uniqueness of confidence densities is true for other statistically dual distributions [8].This allows to construct and use confidence distribution of parameter λ under estimation of parameter via measurement of the random variable x.More details about confidence distributions can be found in reviews [14,15].
Note, statistical duality is duality between confidence and probability.In this sense the Eq. 5 can be considered as a law of conservation.

Significance of difference
The concept of "the significance" of a signal in presence of background in experiment [16] (or, more precisely, "the significance of the difference" between the number of signal events and zero) is widely used in data processing in high-energy physics.Let a sample (or samples) of realizations of some random variable be obtained from an infinite population within a given time.Each realization is called as an event.Number of realizations, which determine by some of conditions (for example, cuts), can be either a background events, or a signal events, which are indistinguishable.
Several methods exist to quantify the statistical "significance" of an signal (expected or estimated) in this sample.Following the conventions in high energy physics, the term "significance" usually is the "number of standard deviations" of an expected or observed signal above (or under) from the difference between an expected or observed value (signal plus background) and expected or estimated background.

Classification of significances
In the simplest case, the concept "significance" can be described with the help of two numbers: b -the number of background events and s -the number of signal events (signal and background events are indistinguishable) which appeared during the given time.
The distribution of the observed number of background events b and the observed number of signal events ŝ usually obey Poisson distributions with parameters b (expected number of background events) and s (expected number of signal events), respectively.Note, the realization of random value (number of events) allows to estimate parameter of Poisson distribution.It means that we must compare the estimated parameters of Poisson flows of events if we comparing two samples.For example, to assess the uncertainties that arise after (or before) measurements, significances and S 2 give incorrect results with small number of events [17].
Let us S characterizes the significance of signal.The choice of significance to be used depends on study.There are three types of studies and, correspondingly, three types of significances [19].
• Type A. Expected significance: if s and b are expected values then we take into account both statistical fluctuations of signal and of background.Before observation we can calculate only expected significance S which is a parameter of experiment.S characterizes the quality of experiment.[17] is an example of significance Type A.
• Type B. Observed significance: if s + b is observed value and b is expected value then we take into account only the fluctuations of background.In this case we can calculate an observed significance Ŝ which is an estimator of expected significance of experiment S .Ŝ characterizes the quality of experimental data.For example, Ẑ (or S cP ) [19].This significance corresponds a probability to observe number of events equal or greater than where σ 2 s+b and σ 2 b are corresponding variances of error distributions.If samples for estimation s + b and b have different volumes (different integrated luminisities of experiments) then formula for significance looks as where K is a ratio of integrated luminosities of experiments.

Asymptotical normality of significances
An important property of these significances is property that when comparing two independent samples taken from the same general population, the distribution of estimates of "the significance of the difference", obtained for these samples, is close to the standard normal distribution N(0, 1).This is shown for several significances (Poisson flows of events) in papers [18,19] (significances S c12 and Ẑ) and in paper [2] (significance S d ) by Monte Carlo experiments.M. Fisz [20] showed that significance S d in case of Poisson distributions is asymptotically normal N(0, 1).
It is understood implicitly that "significance" should follow (asymptotically) a Gaussian distributions with a standard deviation which equals one [2,19], i.e. the variances of such type significances are close to 1 and we have the property of statistical duality for these significances.

Comparison of data
The "statistical duality" property of significances allows to unificate the comparison of corresponding bins of histograms or corresponding points of dependences.
The famous slogan "God made men, but Samuel Colt made them equal" can be rephrased for these significances "Experimenter produces measured points, and only significance of difference make them equal".
Statistical duality allows to mix frequentist probabilities and confidence densities.This means that we can use the measurement of corresponding random variables as estimators of parameter and confidence density of this parameter.
For example, we plan to compare reference histogram or reference dependence and test histogram or test dependence.During comparison we can consider values in reference histogram or in reference dependence as observed random variables.Correspondingly, values in test histogram or values in test dependence we can consider as parameters.And vice versa.

Comparison of histograms
Suppose, there is given a set of nonoverlapping intervals.A histogram represents the frequency distribution of data which populates those intervals.This distribution is obtained during data processing of the sample (which is taken from the Poisson flow of events) with observed values of random variable.These intervals usually are called as bins of histogram.Consider as example of histogram comparison by the use formula where i is number of bin, nim is number of realization of events in bin i in histogram m, K is a ratio of histograms volumes, σ 2 ni1 , σ 2 ni2 are variances of number of events in bin #i.An example is presented in Fig. 2.Here K = 2, σ 2 ni1 = ni1 , σ 2 ni2 = ni2 , i is changed from 1 up to 1000.

Consistency or distinguishability of histograms
Often a goal of histograms comparison is a testing of their consistency.Consistency here is the statement that both histograms are produced during data processing of independent samples which are taken from the same flow of events (or from the same population of events).
In our paper [2] is proposed approach which allows to estimate the distinguishability of histograms and, correspondingly, the distinguishability of parent flows of events (or parent samples).We use the distribution of some test statistics (significances of difference) instead of single test statistics in other methods.This distribution has statistical moments (for example, the mean, rms, asymmetry, excess, ...), i.e. the distribution can be considered as multidimensional test statistics with, for example, statistical moments as coordinates.

"Distance" between histograms
We can calculate statistical moments for distribution of Ŝ i (Fig. 2(d)) and, in principle, we have information about distinguishability of samples under testing.Here we consider two moments: the mean value of significances distribution S and the rms (root mean square) of this distribution, i.e. bidimensional test statistics S RMS = ( S , rms) as a "distance" between histograms [1, 2]: a) if S RMS = (0, 0), then histograms are identical; b)if S RMS ≈ (0, 1), then samples are taken from the same flow of events; c) if previous conditions are not valid, the flows of events have difference.

Comparison of dependences
We can applied this approach for comparison of pair of dependences.The another formula where i is number of point in dependences for independent variable, nim is value of dependent variable in point i for dependence m, m = 1, 2, σ 2 ni1 , σ 2 ni2 are variances of dependent variable in point #i, is used in this case.
Due to statistical duality of significance we can use test statistics S RMS both as measured random variable and as a parameter which describes the pair of dependences under comparing.

Hypotheses testing
The using of approach described above for comparison of data has many problems, which can be avoid with the help of the method of repeated dependence.If the goal of the comparison of dependences is the check of their consistency, then task is reduced to hypotheses testing: main hypothesis H0 (dependences are produced during data processing of samples taken from the same flow of events) against alternative hypothesis H1 (dependences are produced during data processing of samples taken from different flows of events).The determination of critical area allows to estimate Type I error (α) and Type II error (β) in decision about choice between H0 and H1.The Type I error is a probability of mistake if done choice is H1, but H0 is true.The Type II error is a probability of mistake if done choice is H0, but H1 is true.
Distribution H0 is a confidence density of expected value of test statistics (which is used for hypotheses testing) if hypothesis H0 is true, distribution H1 is a confidence density of expected value of test statistics if hypothesis H1 is true.
The selection of a significance level (α) allows to estimate the power of the test (1 − β).Usually, the values of significance level are 10%, 5% or 1%.
If both hypotheses are equivalent, then other combinations of the α and β are used.For example, in task about distinguishability of the flows of events works a relative uncertainty α + β 2 − (α + β) for α + β ≤ 1 [21].Under the test of equal tails [21,22] the mean error α + β 2 can be used.

Method of repeated dependence
The hypotheses testing require the knowledge of the distribution of test statistics.The distribution of test statistics can be constructed by Monte Carlo.If errors of values in measured points of at least one of dependence (for example, reference) are known, than one can construct the set of similar dependences (clones) according with errors, which imitates the population of dependences which produced due to data processing of the samples taken from the same flow of events.This set of dependences is used for construction of the distribution of reference statistics for the case of H0 hypothesis (due to comparison of the reference dependence and the produced clones of the reference dependence).This procedure can be named as "method of repeated dependence" in analogy with "method of repeated sample" or "resampling" in bootstrap technique [23].
Further the set of dependences of such type is constructed for test dependence (second dependence under comparing).New set is used for construction of the distribution of test statistics for the case of H1 hypothesis.This is comparison of the reference dependence with the produced clones of test dependence (i.e. with clones of second dependence).
The comparison of the distribution of reference statistics for the case of H0 hypothesis (imitation population of S RMS , which produced by the comparing of reference dependence and its clones) and the distribution of test statistics for the case of H1 hypothesis (imitation population of S RMS , which produced by the comparing of reference dependence and clones of test dependence) allows to estimate the uncertainty in hypotheses testing.Note, there are another combinations for comparison depending on task (reference clones with test clones and so on).The procedure is used in paper [3].The application of this procedure in the case of histogram comparison can be found in paper [2].

Conclusion: advantages of this approach
• We have a measure of the "distance" between dependences.It is a relative uncertainty κ of the decision about consistency of dependences.

•
s + b in sample with Poisson distribution with mean b which converted to equivalent number of sigmas of a standard normal distribution, i.e. 1 − Φ( Ẑ) Type C. If s + b and b are an observed values of signal+backgroun and background with known errora of measuremenets then we can use the standard theory of errors to estimate the significance of signal S d .In case of normal distribution of errors the formula for S d looks as

Figure 2 .
Figure 2. Triangle distributions: (a) the observed values ni1 in the first histogram (left,upper), (b) the observed values ni2 in the second histogram (right, upper), (c) observed significances Ŝ i bin-by-bin (left, bottom), (d) the distribution of observed significances Ŝ i (right, bottom).