Today in GWO you can track, for example, when a visitor purchases something and/or signs up for a newsletter. This is done by simply executing the goal tracking script for both of these events. However, some experimenters want to track these kinds of things independently: how do site variations influence purchases independent of sign ups and vice-versa, for the same visitors.
For the purposes of the description of this technique, I will assume a multi-variate experiment is being used. Later on, I will describe how to adapt the technique to an A/B experiment.
The experiments I created for my sample test.
Begin by creating two nearly identical GWO experiments (I recommend giving them the same name, followed by ‘1’ and ‘2’, respectively). The first experiment should be set up normally. Assuming an MVT experiment, put the control script at the top of the test page, the tracking script at the bottom, and the section scripts throughout. Also place the goal tracking script at the bottom of the your first goal page (or wherever you want the first goal to fire).
You can then validate the first experiment. and continue on to adding the variations for the experiment’s various sections. Once done with that, you can progress on to the launch page, but do not launch the experiment at this time.
Then create the second experiment for the same test page and the second goal page. Do not place the control script for this second experiment in any pages. Do place the test page tracking script from the second experiment just after the test page tracking script for the first experiment. You will not want to modify the section scripts in any way. Do place the second experiment’s goal tracking script at the bottom your second goal page (or wherever you want the second goal to fire).
When you try to validate the second experiment, because the control script is not present on the test page, it will fail. Simply click continue in the validation dialog box, then click the continue button on the install page (which should now be enabled). You will be presented with a confirmation dialog because validation did not complete. You can also upload temporary test and goal pages for the second experiment to get past validation (this will be required for A/B experiments because the A/B experiment wizard does not allow for skipping validation at this time).
For the second experiment you will need to create the same number of variations for each section as the sections in the first experiment. However, you will not need to fill in any alternative content, just leave the variation content alone for variations in the second experiment, that content will not be used. You should name the variations in the second experiment the same as the first. You should then launch the second experiment. It will not yet have any effect.
Then, place the following custom script immediately after the control script for the first experiment:
function CopyExperiment(src_key, dst_key, dst_id, dst_g, cookie) {<script>
(function() {
function set_cookie(name,value,timeout) {
if (_udn && _udn != "") value += ";domain=" + _udn;
value += ";path=" + _utcp;
if (timeout == 0 && _utimeout && _utimeout != "") timeout = _utimeout;
if (timeout > 0) value += ";expires=" + (new Date((new Date).getTime()+timeout*1000)).toGMTString();
document.cookie = name + "=" + value;
}
var cs = document.cookie.split(‘;’);
for (var i = 0; i < cs.length; i++) {
var c = cs[i].split(‘=’);
var s = c[0];
while (s.length > 1 && s[0] == ‘ ‘) s = s.substr(1);
while (s.length > 1 && s[s.length – 1] == ‘ ‘) s = s.substr(0, s.length – 1);
if (c.length == 2 && s == cookie) {
var es = c[1].split(‘.’);
var d = 0;
var dv = “”;
for (var j = 1; j < es.length; j++) {
var ek = es[j].substr(10, 10);
if (ek == dst_key) {
d = j;
} else if (ek == src_key) {
if (dst_g.length > 0) {
dv = dst_id + dst_key + ‘:’ + dst_g + es[j].substr(22);
} else {
dv = dst_id + dst_key + es[j].substr(20);
}
}
}
if (dv.length > 0) {
if (d == 0) {
es.push(dv);
} else {
es[d] = dv;
}
}
set_cookie(cookie, es.join(‘.’), 63072000);
return;
}
}
}// Custom variables, adapt them to your own experiments
var src_key = ‘3923492669‘;
var dst_key = ‘4234772301‘;
var dst_id = ‘0000370338‘;
var dst_goal = ‘2‘;
CopyExperiment(src_key, dst_key, dst_id, dst_goal, ‘__utmx’);
CopyExperiment(src_key, dst_key, dst_id, ”, ‘__utmxx’);
})();
</script>
You can obtain a copy of this script from the source of my example test page.
First note the four customizable variables near the end of the script. You will need to modify these to adapt the custom script to your new experiments. The effect of this script will be to copy experiment information from the GWO cookies for the first experiment to that for the second. The first three numbers need to be quoted, zero padded 10 digit numbers. The last is a single quoted digit.
For the value of src_key, substitute the experiment key for your first experiment. You can find this key near the beginning of the first experiment’s control script:
<script>
function utmx_section(){}function utmx(){}
(function(){var k='3923492669',d=document,l=d.location,c=d.cookie;function f(n){
if(c){var i=c.indexOf(n+'=');if(i>-1){var j=c.indexOf(';',i);return c.substring(i+n.
length+1,j<0?c.length:j)}}}var x=f('__utmx'),xx=f('__utmxx'),h=l.hash;
d.write('<sc'+'ript src="'+
'http'+(l.protocol=='https:'?'s://ssl':'://www')+'.google-analytics.com'
+'/siteopt.js?v=1&utmxkey='+k+'∓utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='
+new Date().valueOf()+(h?'&utmxhash='+escape(h.substr(1)):'')+
'" type="text/javascript" charset="utf-8"></sc'+'ript>')})();
</script>
Remember that in the custom script above, this must be enclosed in quotes and be a 0 padded, 10 digit number.
The dst_key is similar to the src_key; it can be found at the same place in the second experiment’s control script.
The dst_id and dst_goal are a bit more difficult to find. To obtain them, load the following URL into Firefox, or use the curl command (other browsers, like IE, will not show you the source of the returned script). Be sure to substitute the 10 digit utmxkey with the experiment key of your second experiment (the value of dst_key).
https://www.google-analytics.com/siteopt.js?utmxkey=4234772301&utmx=&utmxx=
This URL will return a chunk of JavaScript which, probably around the third line (which will be rather long), will contain something that looks like:
set_cookie('__utmx','00003703384234772301:2:0-1',63072000)
The single digit located between colons is the value you will need for dst_goal. In this example it is 2. Find this value and substitute it for the one in the custom script. Make sure this is the set_cookie call for __utmx, not __utmxx.
Then, locate the 20 digit, zero-padded number, just before the number for the dst_goal. The first 10 digits of this number is what you’ll use for the dst_id. Substitute this number for the one in the custom script.
You can now preview the first experiment to make sure it looks good, and then launch the first experiment. You should now have two new experiments, each of which are running. If you need to pause the multiple-goal experiment, you need only pause the first experiment.
Example of GWO’s __utmx cookie created when visiting sample test page.
When a visitor loads a test page, after the control script has run and has updated the GWO cookies to contain information about the first experiment (which combination the visitor is to see, most specifically), the custom script will execute and update the GWO cookie to mimic, for the second experiment, the combination chosen for the first experiment. This way, both experiments, on a per visitor basis, will be in sync with each other. However, because the two experiments have different goals, visitors will tracked differently with respect to these goals.
The report for the first experiment will show you how the first goal converts for all the users of the experiment, while the report for the second experiment will show you how the second goal converts for the exact same set of visitors.
Note that you will need to perform the preceeding customization for any page which is tested. That is, anywhere you would have placed the control script, you need to also include the custom script above, and anywhere you would have placed a test page tracking script, you will need to have both test page tracking scripts for both experiments.
With respect to goal page tracking scripts, these are treated exactly as any normal experiment. For example, you can trigger the test page tracking script in response to a click event. There is no need for customization of the goal tracking scripts.
Example Files
I have constructed an example using this technique. You should be able to visit the test and goal pages and see your cookies reflect the presence of the two experiments (which are actually running live):
Customizing A/B Experiments
If you want to use this technique for an A/B experiment, you will perform a very similar setup. The only modification you will need to make is to edit the control script of the first experiment in your test (A) page. At the end of the control script for an A/B experiment, you will find the following script:
<script>utmx("url",'A/B');</script>
You will need to remove this from the control script and place it to be after the custom script. This script is executed to perform the redirection to alternative pages, and needs to take place after the editing of the GWO cookies for multiple goals.
Reporting
I have noticed that the reports for the two experiments can sometimes show a different number of visitors for the same combinations. But I have found that over time the two reports synchronize.
Because the two experiments have different goals, they will probably not recognize the same winners, or have the same amount of confidence. You will have to choose which combination, on whatever experiment, satisfies you.
Also, because you are measuring two goals simultaneously, you should wait for a combination for either goal to become a particularly clear winner before acting upon it. The idea behind this is that you are waiting for two events to show significance without controlling for it. One of two events has a better chance of looking significant than one alone, so wait a bit longer for a winner to take a particularly significant lead.
Future Compatibility
This technique replies on the fact that, currently, the control script creates and edits GWO cookies (actually, siteopt.js does this, but the control script fetches siteopt.js). I have been considering changing this so that instead of the control script manipulating cookies, the test page tracking script will do this. This probably won’t happen for a while, but when it does, you will have to modify this technique slightly.
How will you know when this may be the case? I will probably modify the control script to have a new version number. Here is the line from the control script which would be modified:
+'/siteopt.js?v=1&utmxkey='+k+'&utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='
The new version number will be 2 (or greater), indicating that the control script is different and may no longer be setting cookies. You can verify this by making the above request for siteopt.js for one of your new, running, experiments. I recommend the second experiment which you launch early. You will want to check to see if it still performs a set_cookie call. It is also possible that the control script’s version number may not change, and siteopt.js will simply not set cookies for new experiments. Thus, you’ll just need test to see what the situation is.
When the control script no longer edits cookies, there may be three ways to continue to use this technique (assuming GWO has not already built in multiple goals):
- Place the custom script to come after the first experiment’s test page tracking script, which will have created the GWO cookies. Then, when the second experiment’s tracking script is called, the custom script will have edited them properly.
- Revert the version of the control script back to the last version which still has it setting cookies.
- Instead of the control script editing GWO cookies it will probably edit a the GA global variable to contain the new values to be assigned to the GWO cookies which the tracking script will set. Edit the values in the GA global variable.
In any case, until this change takes place, I’ll not know what the correct workaround will be, so make sure you test this technique before using it. When I do effect the change, I’ll try to remember to update this article. Remind me if I’ve forgotten 🙂
More than Two Goals
This technique should also generalize to more than two goals. Simply create a new experiment for each additional goal and treat it in a similar fashion as the second experiment above.
Stopping, Pruning, Creating Copies and Follow Up Experiments
To stop this kind of experiment, simply stop both experiments at the same time. Also, when making copies or creating follow up experiments, simply perform the same operation on both experiments.
When pruning, one need only prune combos from the first experiment. If you want, you can prune the same combos on the second. In general, performing any operation should be applied to both experiments.
Report Discrepancies
In general, the number of impressions that one sees for variations in both experiments should be exactly the same. However, because the experiments are, essentially, being tracked independently, differences can arise. There are a variety of reasons for this which are, to a great extent, out of your control.
For example, if a visitor to your site closes the window immediately after loading the site, one of the tracking requests might get through, while the other is canceled.
However, the discrepancies should either reconcile after a couple of days, or be small. For example, my example experiments currently read:
Showing a slight discrepancy. I would not be too concerned unless it starts to differ by more than, say, 5%.
Interpreting Reports
So, what does one do if a variation is a winner in one report and and a loser in another? Well, that’s up to you and how much you value each conversion. I would recommend running a multiple-goal follow up experiment. The more metrics you measure, the greater the chance that one of them may be misleading.
Comments:
Man you rock Eric, I attempted this a while back, but gave up, thanks a ton.
Erik Vold : March 25, 2009 11:38 PMThanks for this, Eric! We’ll try this method out on the next multi-goal test.
Chris Goward : March 29, 2009 7:14 AMWow, so glad I stumbled upon your blog.
I have been trying to monitor two goals (email sign-up within cart and over site conversion) in the following manner.
1) Integrate GWO and GA using ROI code
2) Set up exp with conversion script as an onclick event (newsletter signup)
3) track overall site conversion by checking out index value for each combination.
This should allow for cleaner results.
While I’m here, I’ve got a question you may be able to answer. Are you using a Tukey’s multiple comparison for the reports? If not, you guys may be playing a little fast and loose when comparing different levels of a condition.
If I’m misunderstanding the stats used, feel free to enlighten me.
Keep up the great work!!
Beck : April 12, 2009 8:32 PMWe do correct for multiple comparisons using the Bonferroni adjustment. We’ve looked into others, but they don’t offer that much more improvement over this conservative approach.
Eric Vasilik : April 14, 2009 9:44 AMI tried this technique and it works great.
Few things which you might come across to see working version of this test are:
1) check if number visitors are same in both experiments, if not there is something messed up in java script. This tests if you have followed the process correctly.
2) while determining the dsd_id and dst_goal, if you don’t find the value instantly then clear the cookies and again try with the said url(https://www.google-analytics.com/siteopt.js?utmxkey=4234772301&utmx=&utmxx=) in browser. it will give out the a big javascript. it is easier to do ctrl+f and find for value of your dst_key to determine the following dst_id.Also close any open experiment pages. clearing cookies and restarting browser helps to determine the dsd_id value easily.
Follow the steps as Erik has mentioned. start the experiment2 before experiment1 or even before looking for dsd_id.
3) When you are copying the experiment, the value of dsd_id changes, so once you copy the experiment again go through the described process to determine the new dsd_id and dsd_goal value. Need to do it again.
This was my personal experience. feel free to correct me.
Erik thanks again for the post.
osris : April 14, 2009 11:34 PMHi Erik,
Somehow my things are good in my prototype for test but when i rolled same procedure to an ecommerce site, the number of page visitors are different by 40, which ideally should be same in for both the tests. Is there any other way to check if tests are in synch. I am wondering is it because of lag I tend to see different numbers for page visitors.
Something that i did differently which i think shouldn’t affect this process were: My names of two experiments were quite different not like goal1 and goal2. and I kept the conversion script for goal2 in header.
if a test is not doing good on a variation in goal1 does good on goal2, what would you recommend– keep it or leave it or create follow up tests.
Osris Qusba : April 16, 2009 4:57 PMOsris, I’ll answer your questions in the post so others will more easily see them.
Eric Vasilik : April 17, 2009 12:58 PMThanks Erik, that will be really very helpful.
Osris Qusba : April 17, 2009 1:23 PMErik, you got one of the most amazing blogs I’ve read recently… Thanks a lot to share your knowledge and intelligence 🙂
I will try this technique in my next experiment.
Keep up your great work!
Edu Barredo : April 28, 2009 1:22 AMInteresting. Often we have weights for each success event, is there any way to use some measure of total value as the dependent variable such that we are maximizing total returns?
Related to that, can Google’s website optimizer be set up to be adaptive? I think what is often desired is to select the best combination of attributes in order to max returns/min regret based on some specific reward function. While hypothesis testing is great for assessing experiments, it is not normally the best approach for optimizing action selection since it follows a uniform exploration policy in the action space.
Are there any plans to include some sort of adaptive learning behavior?
Thanks
Matt Gershoff
: May 10, 2009 2:46 PMThe idea of comparing different arms of an experiment by using a function over multiple goals is one we’ve considered for GWO. The first step is to get multiples goals into the product 🙂
With respect to your second question, we’ve also considered the explore versus exploit aspects of experimentation. The idea of giving more traffic to arms which seem to perform better in the hopes that the average conversion rate (or whatever metric which one wants to optimize) during the experiment becomes higher is one we are considering as well.
Eric Vasilik : May 11, 2009 9:26 AMCool. Thanks!
Matt Gershoff
: May 11, 2009 1:35 PMBecause we want to keep the layout consistent for one session.Thanks!
ming.li : June 18, 2009 9:49 AMThanks for this post. Exactly what we needed!
We are having a problem in the implementation though. Our test page does not show any conversion starting from the second experiment (we are setting up about 5 experiments). The first experiment is ok.
I know it is hard for you to comment on without seeing the code, but can you think of any obvious issues that might cause this?Thanks.
Burak
Burak Tansug : August 12, 2009 9:47 AMHard to say what the problem is. I need to write an article about debugging techniques.
Eric Vasilik : August 20, 2009 6:15 PMNot sure if I understand this correctly, so this may be a silly question. I would like to test two sales funnels – each with two goals – to see how the conversions are for goal 1 and goal 2.
I would like the user to initially be shown content that is either of two landing pages (L1 or L2). If the user gets to L1, then the goals would be S1 and P1. If the user gets to L2, then the goals would be S2 and P2.
Is this even possible? Are the preceding steps how to do this?
: September 9, 2009 12:21 PMAndrew D. – The situation you describe is possible. The key is that the two goals for the two funnels should be instrumented with the the same conversion tracking scripts. That is S1 and S2 conversions use the same script for the first experiment and P1 and P2 use the same script from the second experiment.
Eric Vasilik : September 10, 2009 1:05 PMwe are running a multi-goal A/B test for 2 landing pages. Everything is working fine except 2 things:
1. For some individual experiments, the control page is showing more visitors than test page. I understand that the visitor count between pages will not be completely 50-50 but i would expect that to be a small deviation from that, however the difference we see is about 60%.Why would this be?
2. I asked this before and you did reply saying it is difficult without seeing the code, which makes perfect sense. Our first experiment shows much higher visitor counts than other epxeriments. I think it is picking up the visitor counts from experiments that come after that (in terms of individual experiment tracking scripts placed after the first one)
just wondering if I can e-mail the code for you to look at. This is still not resolved and I am under the gun to provide an explanation :)Thanks.
Burak
Burak Tansug : September 10, 2009 1:12 PMThank you so much for your quick answer. You have created a wonderful resource here!
: September 10, 2009 7:42 PMAs always – great info!
Personally, I prefer to integrate the variation number into GA and just use the GA goals (yes, I need to calculate the results myself).
Keep up the great work!
Ophir Prusak : September 23, 2009 9:10 AM