![]() ![]() Ultimately what we want to maximise our utility functions (e.g. For example, multi-armed bandits is another experimentation approach to solve the same problem. Recently, causal inference has also come to the foreground as a mean to get even greater insights by our observational data but that is a step following standard regression analysis techniques.įinally, it is notable that while permutation tests are indeed a type of hypothesis testing, A/B testing is a type of an experimentation approach to maximise business goals and using statistical hypothesis testing is only part of the methodology required (parametric or non-parametric, frequentist or Bayesian, etc). ![]() In order cases, I would strongly urge you to educate yourself in regression analysis and general experimental design principles too. Yes, permutation testing is enough in the majority of cases where everything has gone as planned, we observe no seasonality, no primacy or novelty effects, we have limited selection bias, and what not. We need regression analysis, we need to be able to be certain that there are not confounding variables that we accidentally ignored or unexpectedly included, and so forth. As such, think the question should not be if " permutation testing" is sufficient for A/B testing but if " hypothesis testing" is sufficient for A/B testing. ![]() Permutation tests therefore are not a statistical panacea. 17 Large-Scale Hypothesis Testing and FDRs in Computer Age Statistical Inference (2016) by Efron & Hastie for a more careful discussion on that. It might be even argued that permutation tests construct a null when there is not a well-defined one while parametric test never shy away from the explicit definition of a null - see Chapt. see The large-sample power of tests based on permutations of observations (1952) by Hoeffding as an early attempt to use permutation tests instead of standard parametric tests of hypothesis testing) and doesn't really have a definitive answer. Whether or not they should be used instead of standard parametric tests is an question that is really old (e.g. see " How do bootstrap and permutation tests work?" (2003) by Janssen & Pauls for a relevant comparison). Ultimately permutation tests are a type of statistical significance test we could have use bootstrapping if we want another frequentist non-parametric approach (e.g. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |