{"id":479,"date":"2017-04-20T17:51:40","date_gmt":"2017-04-20T14:51:40","guid":{"rendered":"https:\/\/rees46.com\/blog-en\/?p=479"},"modified":"2017-04-20T17:52:30","modified_gmt":"2017-04-20T14:52:30","slug":"real-price-of-an-ab-test-between-service-providers","status":"publish","type":"post","link":"https:\/\/rees46.com\/blog-en\/2017\/04\/20\/real-price-of-an-ab-test-between-service-providers\/","title":{"rendered":"Real Price of an A\/B Test Between Service Providers"},"content":{"rendered":"

\"\"Michael Kechinov
\n<\/b>CEO at REES46 Technologies Inc.<\/span><\/p>\n

Logic tells us that every online store should choose the marketing automation and personalization services that drive its revenue. And never use the tools that hinder that or, what\u2019s even worse, incur losses. However, in real life, people often make weird choices.<\/span><\/p>\n

Digital marketers like to limit their logic to \u201cThis company provides its services to some big and famous retailers. And those are no slouch when it comes to making money, so I\u2019d better make the same choice.\u201d. Guess what \u2013 a lot of those big retailers based their choice on the same logic.<\/span><\/p>\n

That when it is so easy to run an A\/B test and understand the real value of the services to you? Before putting any big money on the table.
\n<\/span><\/p>\n

I\u2019ve recently learned to my surprise that occasions of such useful A\/B testing are rare to none. The reason, according to the market, being that running such tests is a very costly practice. Expressed in numbers, an A\/B test of two recommendation engines would cost over $10000. Having run the calculations three times, I asked the only question on my mind: \u201cWhoa..?! Where did you get these numbers from?\u201d.<\/span><\/p>\n

To that I was listed the following, quoting:<\/span><\/p>\n

    \n
  1. The price of the correctly set server traffic distribution and sending custom dimensions to Google Analytics via Measurement Protocol.<\/span><\/li>\n
  2. Considering the count\/division of the experimental sessions must start from the first page, even for the new visitors, the testing company will have to use its own clientID instead of Google CID (that is generated on the client\u2019s side).<\/span><\/li>\n
  3. Order status tracking in Google Analytics via Measurement Protocol (full sync with CRM\/ERP).<\/span><\/li>\n
  4. Full dataset about every order and session loaded from Google Analytics via their API after the test has been finished.<\/span><\/li>\n
  5. Matching this data with the marginal income of every order to calculate EBITDA for each.<\/span><\/li>\n
  6. Loading the result numbers into R\/Python\/Excel and assessing the sample distribution.<\/span><\/li>\n
  7. Finding and isolating all the anomalies (95\/99-percentile) and calculating the standard deviation.<\/span><\/li>\n
  8. Also, calculating the confidence intervals and statistical validity of the sample.<\/span><\/li>\n
  9. Continue testing if the confidence intervals values are too high and the required statistical significance has not been reached.<\/span><\/li>\n<\/ol>\n

    All that would cost you over $10000. And that is if you are able to find an expert, who can manage it, and a store that will agree to test.<\/span><\/p>\n

    How can we be talking about a profitability analysis when 90% of online stores do not know how to run server-to-server transaction management?<\/span><\/p>\n

    End of the quote.<\/span><\/p>\n

    My reaction was literally:<\/span><\/p>\n

    \"\"<\/p>\n

    What would I understand from this text if I were a marketer with very limited experience in coding? I would look at the phrases \u201cserver traffic distribution\u201d, \u201cmeasurement protocol\u201d, \u201cpercentile\u201d, \u201cconfidence intervals\u201d and think: \u201cWow! What is all that?! That\u2019s way too complicated and would take the next half a year to understand.\u201d.<\/span><\/p>\n

    Unfortunately (for my opponents), I\u2019ve spent half of my life coding, and half of this time \u2013 managing different developer teams. I know well what devs tell their managers when they want to get rid of the project: \u201cThis will be unbelievably complicated and will take a year.\u201d. They also use techy slang like \u201clegacy\u201d, \u201cneeds a lot of refactoring\u201d, \u201cthe architecture won\u2019t support it\u201d and so on, to validate their response. By the way, if you hear something like this from your developers, just tell them to stop whining.<\/span><\/p>\n

    It\u2019s the same situation in our case with A\/B testing.<\/span><\/h4>\n

    I\u2019ll try to translate the requirements into a more \u201chuman\u201d language. I\u2019ll also give you the code and the real price for this job. But first, let\u2019s speak about the basics! In my area, the average price for a qualified dev outsourcing a project is about $80 per hour. Referring to the cost of the job stated above ($10000), \u00a0we can presume that an A\/B test would require 125 hours of work. In other words, ~4 weeks of coding (we also presume that devs really work 5-6 hours a day spending the rest drinking coffee, smoking and visiting bathroom).<\/span><\/p>\n

    So, a month-long test. A qualified dev consistently working on the project for 1 month. Will this person alone maintain the server to ensure its stability throughout the project? Or will this person assign orders to the chosen segments in the manual mode, each time a new order comes?<\/span><\/p>\n

    Let us return to the \u201cunbelievably complicated\u201d task list I\u2019ve been given by my market colleagues.<\/span><\/p>\n

    The first point of attention is Google Analytics. The author says you cannot trust it, so we need to assign the visitors to the segments on the server ourselves, then we should upload this data to Google Analytics, and finally download it back from GA and match to our dataset. This scheme is rather complicated, and I cannot understand why you would still use Google Analytics if you cannot trust it. If so, it\u2019s better to skip it altogether.<\/span><\/p>\n

    Although, I agree: if the goal of the test is to measure the net profit, then Google Analytics or Google Optimize is not the best choice. Why? Because we need to work only with the orders have been paid for, not the orders in total, and with the whole set of resources, not partial resources like in Google Analytics.<\/span><\/p>\n

    Let\u2019s assign the visitors to the segments on the server and add the info about the segment to each order ourselves.<\/span><\/p>\n

    A code (PHP) that successfully nails items 1 to 4:<\/span><\/p>\n

    if( !$_SESSION[`abc_test_segments`] ) {<\/span><\/i><\/p>\n

    \u00a0<\/span><\/i> $segments = [`A`, `B`, `C`];<\/span><\/i><\/p>\n

    \u00a0<\/span><\/i> $segment = $segments[random_int(0, count($segments))];<\/span><\/i><\/p>\n

    \u00a0<\/span><\/i> $_SESSION[`abc_test_segments`] = $segment;<\/span><\/i><\/p>\n

    \u00a0<\/span><\/i> mysql_query(`INSERT INTO segments (segment) VALUES (\u00ab` . $segment . `\u00bb)`);<\/span><\/i><\/p>\n

    }<\/span><\/i><\/p>\n

    If we translate it into human, it means: if the customer has not yet been assigned to a segment, assign, and create a record in the database for this visit.<\/span><\/p>\n

    If this customer made an order, \u00a0update the segment with this data:<\/span><\/p>\n

    mysql_query(`UPDATE orders SET segment = \u00ab` . $segment . `\u00bb WHERE id = ` . $order_id);<\/span><\/i><\/p>\n

    A code that successfully nails items 5 to 7:<\/span><\/p>\n

    SELECT (order_items.amount * items.price * items.price_margin) AS net_profit_segment_A FROM order_items LEFT JOIN items ON items.id = order_items.item_id WHERE order_items.order_id IN (SELECT id FROM orders WHERE date BETWEEN (\u2026) AND segment = `A` AND status = `complete` AND order_sum <= (SELECT MAX(order_sum) * 0.95 FROM orders WHERE date BETWEEN(\u2026) AND status = `complete` ) AND order_sum >= (SELECT MAX(order_sum) * 0.05 FROM orders WHERE date BETWEEN(\u2026) AND status = `complete` ) );<\/span><\/i><\/p>\n

    It may look complicated to you, but it would take only a couple of minutes for an experienced dev to work it out. Translated into human, it means: get the net profit for all completed orders from the A segment, excluding the most (5%) and least (another 5%) expensive as anomalies. To get the net profit for the B and C segments, you only need to change the letter A in the request to B or C respectively.<\/span><\/p>\n

    Item 8 \u2013 the confidence intervals. What is a confidence interval? In our case, it is a difference in proceeds (range of likely values) in varied segments that, when exceeding a certain level, can have a target value for the parameter. For instance, if the A segment value is $1746 and the B segment \u2013 $1754, the difference ($8) is too small to say that the B segment will stably outperform A segment by 0.5%.<\/span><\/p>\n

    However, the difference of $349 is significant enough to have a target value. What about $78? Confidence interval estimate will help you in this case \u2013 a concept (and the formula) known to every student of math or economic faculty of any university. <\/span><\/p>\n

    The code to determine the confidence interval (close your eyes, if you\u2019re not a dev): <\/span><\/p>\n

    SELECT ( 1.96 * SQRT( ( (SELECT COUNT() FROM orders WHERE segment = `A`)\/(SELECT COUNT() FROM segments WHERE segment = `A`) * (1 \u2014 (SELECT COUNT() FROM orders WHERE segment = `A`) \/ (SELECT COUNT() FROM segments WHERE segment = `A`) ) ) \/ SELECT COUNT(*) FROM segments WHERE segment = `A`) ) AS confidence_interval_for_A <\/span><\/i><\/p>\n

    This type of request is run for every segment to determine the confidence interval for each of them.<\/span><\/p>\n

    If you paid attention, you noticed that the solution is only 1147 symbols long, while the problem took 1429 symbols to describe. If you subsequently divide $10000 (stated cost of the test) to 1147, the cost of one symbol will be $8.72. A lawyer in my area makes less by the symbol!<\/span><\/p>\n

    With all due respect, it took me 15 minutes to create this code and around 30 to google \u201cconfidence interval mysql\u201d. It is less than an hour in total. Even if you give this project to a junior and pay this person as a qualified senior (say, $120), it would take only around 20 hours to complete it. So, the final price would be $2400 \u2013 circa 4 times less than the initially stated price.<\/span><\/p>\n

    All in All<\/b><\/h4>\n

    It\u2019s a good idea to run an A\/B test of a service you think to use. Don\u2019t be tricked by \u201cWe are the leading company in this area\u201d or \u201cWe can !guarantee! you progress\u201d, or \u00abA\/B testing is a very costly procedure\u201d, or \u00a0\u00abMillions of lemmings can\u2019t be wrong\u201d. <\/span><\/p>\n

    \u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

    Michael Kechinov CEO at REES46 Technologies Inc. Logic tells us that every online store should choose the marketing automation and personalization services that drive its revenue. And never use the tools that hinder that or, what\u2019s even worse, incur losses. However, in real life, people often make weird choices. Digital marketers like to limit their … Continue reading “Real Price of an A\/B Test Between Service Providers”<\/span><\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"yoast_head":"\nReal Price of an A\/B Test Between Service Providers<\/title>\n<meta name=\"description\" content=\"Read article about Real Price of an A\/B Test Between Service Providers by REES46.\" \/>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/rees46.com\/blog-en\/2017\/04\/20\/real-price-of-an-ab-test-between-service-providers\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Real Price of an A\/B Test Between Service Providers\" \/>\n<meta property=\"og:description\" content=\"Read article about Real Price of an A\/B Test Between Service Providers by REES46.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/rees46.com\/blog-en\/2017\/04\/20\/real-price-of-an-ab-test-between-service-providers\/\" \/>\n<meta property=\"og:site_name\" content=\"eCommerce Marketing Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/rees46com\" \/>\n<meta property=\"article:published_time\" content=\"2017-04-20T14:51:40+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-04-20T14:52:30+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/rees46.com\/blog-en\/wp-content\/uploads\/2017\/04\/image00-1-150x150.png\" \/>\n<meta name=\"twitter:card\" content=\"summary\" \/>\n<meta name=\"twitter:creator\" content=\"@rees46com\" \/>\n<meta name=\"twitter:site\" content=\"@rees46com\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/rees46.com\/blog-en\/#website\",\"url\":\"https:\/\/rees46.com\/blog-en\/\",\"name\":\"eCommerce Marketing Blog\",\"description\":\"by REES46 eCommerce marketing automation and personalization suite\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/rees46.com\/blog-en\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/rees46.com\/blog-en\/2017\/04\/20\/real-price-of-an-ab-test-between-service-providers\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/rees46.com\/blog-en\/wp-content\/uploads\/2017\/04\/image00-1.png\",\"width\":683,\"height\":683},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/rees46.com\/blog-en\/2017\/04\/20\/real-price-of-an-ab-test-between-service-providers\/#webpage\",\"url\":\"https:\/\/rees46.com\/blog-en\/2017\/04\/20\/real-price-of-an-ab-test-between-service-providers\/\",\"name\":\"Real Price of an A\/B Test Between Service Providers\",\"isPartOf\":{\"@id\":\"https:\/\/rees46.com\/blog-en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/rees46.com\/blog-en\/2017\/04\/20\/real-price-of-an-ab-test-between-service-providers\/#primaryimage\"},\"datePublished\":\"2017-04-20T14:51:40+00:00\",\"dateModified\":\"2017-04-20T14:52:30+00:00\",\"author\":{\"@id\":\"https:\/\/rees46.com\/blog-en\/#\/schema\/person\/9855ad28ad6aa4f0ae2e7ef1fdcbbbce\"},\"description\":\"Read article about Real Price of an A\/B Test Between Service Providers by REES46.\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/rees46.com\/blog-en\/2017\/04\/20\/real-price-of-an-ab-test-between-service-providers\/\"]}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/rees46.com\/blog-en\/#\/schema\/person\/9855ad28ad6aa4f0ae2e7ef1fdcbbbce\",\"name\":\"Roman Chigirev\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/rees46.com\/blog-en\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ca38cae63803b3d0cbd7fc79cba72467?s=96&r=g\",\"caption\":\"Roman Chigirev\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/rees46.com\/blog-en\/wp-json\/wp\/v2\/posts\/479"}],"collection":[{"href":"https:\/\/rees46.com\/blog-en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rees46.com\/blog-en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rees46.com\/blog-en\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/rees46.com\/blog-en\/wp-json\/wp\/v2\/comments?post=479"}],"version-history":[{"count":2,"href":"https:\/\/rees46.com\/blog-en\/wp-json\/wp\/v2\/posts\/479\/revisions"}],"predecessor-version":[{"id":483,"href":"https:\/\/rees46.com\/blog-en\/wp-json\/wp\/v2\/posts\/479\/revisions\/483"}],"wp:attachment":[{"href":"https:\/\/rees46.com\/blog-en\/wp-json\/wp\/v2\/media?parent=479"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rees46.com\/blog-en\/wp-json\/wp\/v2\/categories?post=479"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rees46.com\/blog-en\/wp-json\/wp\/v2\/tags?post=479"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}