Research
Original research, with the data attached
Marketing is full of statistics with no source under them. Everything published here comes with the raw numbers, the scripts that produced them, and a list of the things it does not prove.
Published
How these are run
Six rules, and none of them are optional
No company is named, ever
Aggregate figures only. Publishing a scored league table of other people's businesses is a permanent public record built out of a single measurement, and a single measurement is not reliable enough to carry that. If your own site is in a sample and you want to know where it sat, ask me and I will tell you.
The sample is drawn at random, from a stated seed
Not the first hundred, not the ones I recognized, not the ones that made the point. A fixed seed means anyone can reproduce the identical sample from the identical source list and check that I did not pick favorably.
Every exclusion rule is published, with its effect measured
Deciding what does not count is where a study gets quietly steered. So each rule is stated, the number of rows it removed is stated, and the effect of removing them on the headline figure is measured rather than assumed. Where a rule happens to flatter the argument, that gets said out loud.
The raw data ships with the findings
Row level, with the identifying columns stripped, and with a column on every row saying whether it is in the analyzed sample and which rule removed it if not. Every figure published can be re-derived from that file with a filter.
The scripts are published too
Collection, measurement, analysis. A methodology paragraph is a description of code. The code is the methodology.
The limits are stated by me, not by whoever is arguing with me
Every study has a section saying what it does not show. It goes in before publication, not after somebody finds the hole.
Why bother
This is the same argument as the rest of the site
The Under the Hood page publishes this site's own numbers, including the bad runs, because an agency that will not show you its own measurements is asking you to take speed on faith.
Research is the same move at a larger scale. Instead of one site's numbers, a thousand of them, measured the same way on the same morning, with the file attached so you can argue with my arithmetic rather than with my adjectives.
If a study here is ever wrong, I would rather it was wrong in public with the data sitting next to it than quietly wrong in a sentence nobody can check.
Start here
Tell me what you are trying to fix
Send over the site you have now, or the one you wish you had. I will tell you honestly whether I am the right person for it.
info@palmprojects.com