The Value of Graduate Education....and Experience

What are some of the additional benefits of graduate study? What if you just skip the time, money and energy spent in graduate school and went straight to writing code?

This made me think of a Talking Biotech podcast with Kevin Folta discussing the movie Food Evolution. Toward the end they discussed some critiques of the film, and a common critique about research in general is bias due to conflicts of interest. Kevin States:

"I've trained for 30 years to be able to understand statistics and experimental design and interpretation...I'll decide based on the quality of the data and the experimental design....that's what we do."

Besides taking on the criticisms of science, this emphasized two important points.

1) Graduate study teaches you to understand statistics and experimental design and interpretation and this requires a new way of thinking. At the undergraduate level I learned some basics that were quite useful in terms of empirical work. In graduate school I learned what is analogous to a new language. The additional properties of estimators, proofs, and theorems taught in graduate statistics courses suddenly made the things I learned before make better sense. This background helped me to translate and interpret other people's work and learn from it, and learn new methodologies or extend others. But it was the seminars and applied research that made it come to life. Learning to 'do science' through new ways of thinking about how to solve problems through statistics and experimental design. And interpretation as Kevin says.

2) Graduate study is an extendable framework. Learning and doing statistics is a career long process. This recognizes the gulf between textbook and applied econometrics.

Interpreting Confidence Intervals

From: Handbook of Biological Statistics http://www.biostathandbook.com/confidence.html

"There is a myth that when two means have confidence intervals that overlap, the means are not significantly different (at the P<0.05 level)… (Schenker and Gentleman 2001, Payton et al. 2003); it is easy for two sets of numbers to have overlapping confidence intervals, yet still be significantly different by a two-sample t–test; conversely… Don't try compare two means by visually comparing their confidence intervals, just use the correct statistical test."

A really cogent note related to this from the Cornell Statistical Consulting unit can be found here.

"Generally, when comparing two parameter estimates, it is always true that if the confidence intervals do not overlap, then the statistics will be statistically significantly different. However, the converse is not true. That is, it is erroneous to determine the statistical significance of the difference between two statistics based on overlapping confidence intervals."

More details using basic math here.

A 2005 article in Psychological Methods indicates a large number of researchers don't interpret them correctly. In an interesting American Psychologist article (2005) researchers determined that under a number of broadly applicable conditions 95% confidence intervals can overlap as much as 25% for groups that are actually significantly different at the 5% level and that with zero overlap statistical significance is actually at the 1% level (p~.01).

Dave Giles also brings up an Insect Science paper by Payton et al in discussion related to using CI to determine statistical significance that relates to this:

http://davegiles.blogspot.com/2017/01/hypothesis-testing-using-non.html?_sm_au_=iVVFv75rHPD1TSHs

"Here's a well-known result that bears on this use of the confidence intervals. Recall that we're effectively testing H0: μ1 = μ2, against HA: μ1 ≠ μ2. If we construct the two 95% confidence intervals, and they fail to overlap, then this does not imply rejection of H0 at the 5% significance level. In fact the correct significance is roughly one-tenth of that.  Yes, 0.5%!

If you want to learn why, there are plenty of references to help you. For instance, check out McGill et al. (1978), Andrews et al. (1980), Schenker and Gentleman (2001), Masson and Loftus (2003), and Payton et al. (2003) - to name a few. The last of these papers also demonstrates that a rough rule-of-thumb would be to use 84% confidence intervals if you want to achieve an effective 5% significance level when you "try" to test H0 by looking at the overlap/non-overlap of the intervals."

Actually according to the last paper mentioned (Payton,2003) the 84% CI is adjusted depending on the ratio of standard errors from the two populations you are comparing:

But all of the work above is predicated on a comparison of two populations. Considerations of multiple comparisons complicate things further. (see Rick Wicklin's post on doing this in SAS). Perhaps if a visual presentation is what we want we plot the CIs (as much as we may not like dynamite plots) but denote which groups are significantly different based on the properly specified tests (per the note from the handbook above).  Something like below:

http://freakonomics.com/2008/07/30/how-big-is-your-halo-a-guest-post/

References:

Belia, S, Fidler, F, Williams, J, Cumming, G (2005). Researchers misunderstand confidence intervals and standard error bars Psychological Methods, 10 (4), 389-396

Am Psychol. 2005 Feb-Mar;60(2):170-80. Inference by eye: confidence intervals and how to read pictures of data. Cumming G(1), Finch S

Payton, M. E., M. H. Greenstone, and N. Schenker, 2003. Overlapping confidence intervals or standard error intervals: What do they mean in terms of statistical significance? Journal of Insect Science, 3, 1–6.

Statistical Methods for QTL Analysis

See also QTL Analysis and Quantitative Genetics and  QTL Analysis in R.

In a previous post a gave a general overview of QTL mapping and analysis, and gave the motivation for the use of maximum likelihood to identify the approximate location of a QTL based on RFLP (or marker) variations. Below are more details related to the statistical methods that can be used in this process.

Analysis of Variance and Marker Regression

At each marker (RFLP) loci, compare backcross phenotype distributions for groups that differ according to their marker genotypes. As depicted in Broman (2001)

 

For markers a and c, we see that phenotype distributions differ for genotypes aa and aa’ but not for cc and cc’. Again this indicates that a QTL may be linked to the RFLP genotypes at locus “a” but not "c". 

The differences between genotypes and the associated phenotype distributions for two marker genotypes can be assessed using the t-statistic. For >2 genotypes, analysis of variance may be used. The ANOVA approach allows a flexible experimental design, allowing for the incorporation of covariates, treatment, and environmental effects. 

The model for marker regression, following the notation in Hu and Zu (2009) can be specified as follows:

y_i= X_iβ + Z_iγ +ϵ_i

such that  y_i  is the phenotype of the ith individual, β is the vector of control effects, X_i is design vector,γ is a vector for QTL effects, and Z is a genotype indicator vector.

Z = H₁ for A₁A₁ , H₂ for A₁A₂ , H₃ for A₂A₂  or more generally 

 

Tests on hypotheses related to QTL effects take the form H₀: γ = 0 . Hence we test the null hypothesis of no QTL associated with genotype Z. 

Maximum Likelihood

For maximum likelihood estimation, the following probability density of y_i can be stated as:

f(y_i) = Pr(y_i­|Z_i=H_k)  ‘ the probability of phenotype ‘y’ given genotype H_k­’

= (1/ √ 2π σ)  exp[ 1/2 σ ² (y_i-X_iβ +H_kγ )²

The log likelihood function can then be specified as L(θ) =Σ  ln(f(y_i))

The hypothesis H₀:  γ =0 can be tested using the likelihood ratio test:

λ = -2(L₀-L₁)

where L₀ represents the likelihood under a restricted model. This is equivalent to the general notion presented in Broman (1997) and my previous post:

Likelihood (effect occurs by QTL linkage) / Likelihood(effect occurs by chance)

References:

 Jones, N., H. Ougham, and H. Thomas. Markers and mapping:We are all geneticists now. New Phytol. 137:165–177.1997.

Broman KW. Lab Anim (NY). Review of statistical methods for QTL mapping in experimental crosses.

2001 Jul-Aug;30(7):44-52.

Zhiqiu Hu and Shizhong Xu (2009). PROC QTL - A SAS Procedure for Mapping Quantitative Trait Loci. International Journal of Plant Genomics 2009: 3 doi:10.1155/2009/141234.

QTL Analysis and Quantitative Genetics

See also Statistical Methods for QTL Analysis and QTL Analysis in R

Restriction Fragment Length Polymorphisms (RFLPs) and Restriction Enzymes

Restriction enzymes target specific DNA base sequences producing staggered cuts and variable length DNA fragments-‘RFLPs’.

RFLPs establish fixed landmarks in the genome. Subjecting DNA to restriction enzymes and gel-electrophoresis leads to gel patterns which can be described in terms of genotpype or allelic differences between RFLPs  at a given locus, which follow rules of Mendelian inheritance.  (recall a 'locus' is the location of a DNA sequence on a chromosome. A variant of a DNA sequence at a given locus is an 'allele.'

 

Backcrossing, Segregation, and Recombination

P1 AABB x P2 aabb --> F1 AaBb

F1 x P2 -->              AAab

                                Abab

                                aBab

                                abab     

With unlinked genes and independent segregation, recombinant genotypes Abab and aBab will result 50% of the time. With linked genes, crossing over will result in recombinants < 50% of the time.

 

The probability of crossing over is proportional to the distance between loci, therefore, crossover rates can be used to create genetic maps.

Loci                        % Crossover or Recombinants

a,b                          7

b,c                          19

 

Mapping Population

Use P1,P2,F1 to create a population with segregating QTL and RFLP profiles and phenotypic variation. RFLPs that are tightly linked to QTLs will segregate together, can serve as markers for that trait.

Central Dogma of QTL Marker Analysis

DNA: Δ mapped locus position --> Δ RFLP allele -->Δ Phenotype  -->likelihood that RFLP is linked to QTL 

Where a QTL is referred to as a bundle of genes associated with some quantitative trait of interest. As we move across a segment of DNA, we note changes in RFLP genotype alleles, and associated changes in the phenotype of interest. This data can then be assessed to determine the likelihood that a RFLP is linked to a QTL.

For example, suppose at locus "a" there are two observed allelic differences in the RFLP profile, denoted aa and aa’. If we notice distinct differences in the phenotypic distribution for plants with alleles aa vs. aa’ it may be inferred that the RFLP at locus “a” is linked to the QTL of interest. If at another location, we observe two allelic differences in the RFLP profile cc and cc’, and we find little difference in the associated phenotypes we may conclude that the RFLP at locus “c” is not linked to the QTL of interest. Intermediate differences between phenotypes would imply that the RFLP at that locus is closely linked to the QTL of interest. The following schematic from Jones(1997) et al illustrates these differences:

 

The map position of the QTL can be determined by the method of maximum likelihood from the distribution of likelihood values determined at each locus above. At each locus we compute the following:

Likelihood(effect occurs QTL linkage) / Likelihood(effect occurs by chance)

Depicted as  L₁/L₀ below:

 

The key to using markers to map QTL is to associate RFLP patterns with QTLs. Observing changes in RFLP profiles among plant DNA from a segregating population at various positions in the genome, and associating them with changes in phenotype allows us to find a statistical relationship between RFLPs and QTLs. Based on the observations above, the evidence strongly supports that a QTL may be found near the RFLP locus “a”.

Reference: Jones, N., H. Ougham, and H. Thomas. 1997. Markers and mapping:We are all geneticists now. New Phytol. 137:165–177

Topics Related to Linear Models

While attending a session at this year’s SAS Global Forum, I was reminded of several topics that I have not covered in previous posts, but should have.

Fixed and Random Effects Models and Panel Data (see also Mixed, Fixed and Random Effects)

Notes: Panel data, or repeated measures data, is characterized by multiple observations on individuals over time.  A consequence of this is that measurements on the same individual will likely be more correlated than measurements for (or between) different individuals.  Measurements taken closer together over time will also likely be more correlated than measurements taken at greater intervals. As a result, assumptions from OLS regression regarding independence and homogenous variance will likely be violated.  As noted during the session I attended, in SAS, PROC MIXED appropriately handles within subject and time dependent correlations and the covariance structure associated with repeated measures.

Let’s structure the model as follows: Yit = Xit + Ait +Uit

Where Ait is a unobserved individual effect. There are two assumptions that we can work under when estimating this model,

1)Random Effects (RE): Assumes that Ait is independent of X . X may be considered a  predetermined, or other fixed effect. Ait is a random effect.

Estimation: Feasible Generalized Least Squares-  B_fgls = (X’W^-1X)^-1X’W^-1y

i.e V(e) != σ²  I  which would be the case under OLS

2)Fixed Effects (FE): Assumes Ait is not independent of X.

Estimation: differencing or subtracting the respective means from X and Y  and running OLS on the adjusted data produces the estimate of B.

You could also subtract the lagged version of each variable X and Y from itself respectively to move the time-correlated component, or you could remove Ait through dummy variable regression.

Fixed and Random Effects in Analysis of Variance (in general)

In my previous posts related to AOV and Mixed Models,  I did not discuss these in the context of  AOV.

Typically fixed effects are repeatable factors that are set by the experimenter,  often the ‘treatments.’  Random effects are effects that are selected randomly from a population, often ‘blocks.’

Type I and III Sums of Squares

Type I Sums of Squares: often referred to as sequential sums of squares, this is the SS for each effect adjusted for all effects that appear earlier in the model.

Example:

                                   

Y1 = X1B1                       

Type I SS = SSR(Y1) = SS(x1)

Y2 = X1B1 + X2B2

Type I SS = SSR(Y2) – SSR(Y1) = SS(X2)

Type III sums of squares are adjusted for every X in the full model.

Example:

Y1 = B1X1 + B2X2

Y2 = B1X1

Type III SS = SSR(Y1) – SSR(Y2) = SS(X1)

Y3 = B2X2

Type III SS = SSR(Y1) – SSR(Y3) = SS(X2)

Means vs. LS Means

Means = overall mean for a treatment or factor level

LS Means = within group means adjusted for other effects in the model

Type I SS are used to test differences between means while type III SS are used to test differences between LS Means. 

Analysis of Variance

BIOMETRICS

COMPARING TWO MEANS: Xbar₁ – Xbar₂ / S_x1 – S_x2 ~ t 

can’t define numerator for >2 means

ANALYSIS OF VARIANCE: Comparing more than two means. When μ₁ not = μ₂, an estimate of variance using means will include a contribution attributable to the difference between population means. This ‘treatment’ effect will lead to a biased estimate of variance and large F.

F = σ ²+ trt / σ = S²_b  / S²_w ~ F(.)

If means are different, then  S²_b  > S²_w  , then F will be large. A large significant F value allows you to reject the null hypothesis that the means are equal. 

Linear Additive Model: Y_ij = U + T_i + E_ij

Or in matrix form: Y = Xb + e  where  Y = a matrix of individual observations. X = a ‘design matrix of 1’s and 0’s, & b =  a matrix of co-efficients of the population mean and treatment effects ‘t’.  or ( U + T_i) = Xb

SS Total = Y’Y – Y’CY’C(1/n)

SS Between/Model=b’X’Y - Y’CY’C(1/n)

SS Within/Error  =Y’Y’ – B’X’Y

Where ‘C’ is a matrix of 1’s

RCB:

                                    ( Blocks)

(treatments)               A            B            C             D             E

1            x             x            x            x            x

2            x            x            x            x            x

3            x            x            x            x            x

4            x            x            x            x            x

1.  If treatments or blocks differ, look at individual differences in means via LSD.
2.  Contrasts: Q = CiYi  ( orthogonal)   MS(Q) = Q² / r sum(Ci²), a subset of a linear function, a way of comparing different means.

LATIN SQUARE:

A	D	C	B
B	C	A	D
D	A	B	C
C	B	D	A

Each row is a comparable block with treatments A-D occurring only once in each block.

FACTORIAL EXPERIMENTS:

1. # treatments consists of several categories, levels
2. May be used to determine optimal levels
3. Interaction: difference in response levels-non additive,  measures homogeneity of  a response.

Polynomial Responses:  partitioning SS into linear and quadratic components- response surfaces

SPLIT PLOT DESIGNS

Whole plots divided into sub-plots where levels of factors are applied.

Block #1

A4B2 A4B1

A1B2 A1B1

A2B1 A2B2

A3B1 A3B2

A4B2 A4B1

Incomplete block re ‘A’ , complete block re ‘B’

SPLIT BLOCK EXPERIMENTS

Split plot in time vs. space ex:  time or cutting represents an abstract split plot ‘a’ cultivars, ‘b’ cuttings.

ANALYSIS OF COVARIANCE

1. Observed variation in Y is partly due to variation in X.
2. Use regression to eliminate effects that cannot be controlled by experimental design.

Ex: animals in a block with varying initial weights, use covariance to remove effect from experimental error.

Y_ij = μ + T_i + p_j + B(X_ij – Xbar) + e_ij

B = Cov(X,Y) / Var(X)

Carry out AOV on values adjusted for regression on an independent variable.

References:

Principles and Procedures of Statistics: A Biometrical Approach 3rd Edition. Robert George Douglas Steel, James Hiram Torrie, David A. Dickey

Mathematical Themes in Economics, Machine Learning and Bioinformatics

By Matt Bogard

Abstract

Graduate students in economics are often introduced to some very useful mathematical tools that many outside the discipline may not associate with training in economics. This essay looks at some of these tools and concepts, including constrained optimization, separating hyperplanes, supporting hyperplanes, and ‘duality.’ Applications of these tools are explored including topics from machine learning and bioinformatics.

Download Full Text