University of Ljubljana
Faculty of Computer and Information Science
Borislav Stojkovski
Web application for monitoring the effectiveness of promotions
BACHELOR THESIS
PROFESSIONAL STUDY PROGRAMME FIRST CYCLE
COMPUTER AND INFORMATION SCIENCE
Mentor : Assist. Prof. Dr. Martin Moˇ zina
Ljubljana, 2021
Univerza v Ljubljani
Fakulteta za raˇ cunalniˇ stvo in informatiko
Borislav Stojkovski
Spletna aplikacija za spremljanje uˇ cinkovitosti promocije
DIPLOMSKO DELO
VISOKOˇSOLSKI STROKOVNI ˇSTUDIJSKI PROGRAM PRVE STOPNJE
RA ˇCUNALNIˇSTVO IN INFORMATIKA
Mentor : doc. dr. Martin Moˇ zina
Ljubljana, 2021
Copyright. Rezultati diplomske naloge so intelektualna lastnina avtorja in matiˇcne fakultete Univerze v Ljubljani. Za objavo in koriˇsˇcenje rezul- tatov diplomske naloge je potrebno pisno privoljenje avtorja, fakultete ter mentorja.
Faculty of Computer and Information Science issues the following task:
Theme:
The goal of this thesis is to develop a dashboard application for sales promo- tions analysis in retail industry. The application should contain 4 modules:
a) a module with promotion overview showing basic statistics and vizualiza- tions, b) a module for customer analysis based on several customer segments, c) analysis of items sold in promotion, and d) an estimation of promotion up- lift. The candidate should identify and analyze different methods for promo- tion evaluation in different experimental settings and compare them qualita- tively and quantitatively. Prior to developing the application, the candidate should select the most suitable environment/tool for this task.
Fakulteta za raˇcunalniˇstvo in informatiko izdaja naslednjo nalogo:
Tematika naloge:
Naloga kandidata je razviti spletno aplikacijo za spremljanje uˇcinkovitosti raznovrstnih promocij v trgovini na drobno. Aplikacija naj omogoˇca a) pred- stavitev osnovnih statistik promocije, kot je npr. ˇstevilo kupcev, ki so koristili promocijo, b) analizo kupcev glede razliˇcne lastnosti - segmente, c) analizo artiklov, ki se v pojavljajo v promociji in d) oceno dviga celotne prodaje zaradi promocije. Pri oceni dviga naj preuˇci in izbere primerno metodo za tip promocij, pri katerih sta bili izbrani testna in kontrolna skupina ter za promocije, ko kontrolne skupine ni na voljo. Pred implementacijo naj kandi- dat preuˇci in izbere najprimernejˇse okolje za razvoj aplikacije.
Contents
Abstract Povzetek
Razˇsirjeni povzetek
1 Introduction 1
2 Business intelligence tool or custom built solution? 4
2.1 Tableau Desktop . . . 4
2.2 Python . . . 5
2.3 Comparison . . . 5
2.4 Our choice . . . 7
3 Domain description 8 4 Methodology 11 4.1 A/B testing . . . 11
4.2 Difference in differences (DID) . . . 12
5 Data description 15 5.1 Customers table . . . 15
5.2 Items table . . . 16
5.3 Groups table . . . 17
5.4 Events table . . . 18
5.5 Sales table . . . 18
6 Application 20 6.1 General tab . . . 21
6.2 Customers tab . . . 24
6.3 Items tab . . . 28
6.4 DID tab . . . 29
6.5 A/B tab . . . 32
6.6 Discussion on why DID and A/B results differ . . . 34
7 Application infrastructure 36 7.1 Tools . . . 36
7.2 Application development . . . 37
7.3 Application structure . . . 38
8 Conclusions 41
Literature 43
Abstract
Title: Web application for monitoring the effectiveness of promotions Author: Borislav Stojkovski
The aim of this thesis is to develop an interactive web application that allows analysis and evaluation of a promotion. Promotion is a type of marketing activity where customers are informed that they can get certain products at a discounted price. The application should provide insights about the outcome of the promotion, which can be used to optimise the next promotion.
It should be simple and intuitive to use. We decided to develop a custom solution instead of using a business intelligence tool because the necessary analyses could not be done using a business intelligence tool. The application is developed using the Python ecosystem. We have data on retail sales before and during a promotion. Two different methods are used to evaluate the promotion, which lead to different results. We compare the methods and discuss why they lead to different results and which results we think are more accurate in our case.
Keywords: dashboard, difference in differences (DID), A/B testing, promo- tion evaluation.
Povzetek
Naslov: Spletna aplikacija za spremljanje uˇcinkovitosti promocije Avtor: Borislav Stojkovski
Namen naloge je razviti interaktivno spletno aplikacijo, ki omogoˇca analizo in oceno promocije. Promocija je vrsta trˇzne dejavnosti, v okviru katere so stranke obveˇsˇcene, da lahko doloˇcene izdelke dobijo po zniˇzani ceni. Ap- likacija mora zagotoviti vpogled v potek promocije, ki ga je mogoˇce upora- biti za optimizacijo naslednje promocije. Njena uporaba mora biti preprosta in intuitivna. Namesto uporabe orodja poslovne inteligence smo se odloˇcili razviti reˇsitev po meri, saj orodje poslovne inteligence ni omogoˇcalo izdelave potrebnih analiz. Za razvoj aplikacije je uporabljen ekosistem Python. Na voljo so nam podatki o maloprodaji pred in med promocijo. Za oceno pro- mocije se uporabljata dve razliˇcni metodi, ki vodita do razliˇcnih rezultatov.
Metodi primerjamo in razpravljamo o tem, zakaj vodita do razliˇcnih rezul- tatov in za katere rezultate menimo, da so v naˇsem primeru natanˇcnejˇsi.
Kljuˇcne besede: nadzorna ploˇsˇca, razlika v razlikah (DID), A/B testiranje, evalvacija promocije.
Razˇ sirjeni povzetek
Vsi smo priˇca maloprodajnim promocijam. V sodobnem svetu se jim je nemogoˇce izogniti. Promocija je vrsta trˇzne dejavnosti, v okviru katere so stranke obveˇsˇcene, da lahko doloˇcene izdelke dobijo po zniˇzani ceni [16]. ˇCe stranke vedo, da lahko doloˇcene izdelke kupijo po niˇzji ceni od redne, se njihov nakupni interes poveˇca [14]. Toda ali imajo od promocije korist le kupci ali tudi prodajalec na drobno? Namen promocije je lahko razliˇcen, na primer:
pritegniti nove kupce, razprodati preseˇzne zaloge, predstaviti nov izdelek, spodbuditi obstojeˇce kupce k veˇcjim nakupom. V tej nalogi analiziramo promocijo, ki obstojeˇce kupce spodbuja k veˇcjim nakupom. Promocija je namenjena kupcem, ki postajajo vse manj zvesti. Ti kupci prejmejo kupon, ko znesek na njihovem raˇcunu preseˇze doloˇcen prag. S kuponom jim pripada popust za doloˇcene artikle. Cilj je kupce spodbuditi k veˇcji porabi. Promocija traja dva meseca, v prvem mesecu se kuponi izdajajo, v drugem mesecu pa se izdani kuponi unovˇcujejo. Podatke za nalogo smo pridobili pri veliki multinacionalni maloprodajni korporaciji.
Oddelek za trˇzenje, ki vodi promocijo, rezultata promocije ne pozna vnaprej. ˇCe promocija ni uspeˇsna, je treba pri naslednji promociji nared- iti nekaj sprememb. ˇCe je uspeˇsna, znamo oceniti, kaj je privedlo do us- peha? ˇCe znamo prepoznati vzorec, ki je privedel do uspeha, bi to znanje lahko uporabili pri naslednji promociji, da bi bila ˇse uspeˇsnejˇsa. Oddelek za trˇzenje pa za to potrebuje orodje, ki prodajne podatke pretvori v uporabno obliko. Eden od naˇcinov za reˇsitev te teˇzave je spletna aplikacija, ki analizira prodajne podatke. V nalogi se osredotoˇcamo na izdelavo interaktivne spletne
aplikacije, do katere dostopamo prek spletnega brskalnika. Cilj je razviti ap- likacijo, ki je intuitivna in preprosta za uporabo. Z le malo usposabljanja jo lahko uporabljajo tudi uporabniki brez znanj o analitiki podatkov. Aplikacija mora vsebovati jasne podatke o izidu promocije. Poleg izida promocije mora aplikacija zagotavljati tudi uporabne podatke. Ti podatki se lahko uporabijo za prilagoditev naslednje promocije. S tem se poveˇca moˇznost za uspeˇsno promocijo.
Pri razvoju aplikacije sta moˇzna dva pristopa: izdelava od samega zaˇcetka ali uporaba obstojeˇcega orodja poslovne inteligence. Orodja poslovne in- teligence nas omejujejo na vgrajene funkcionalnosti. ˇCe doloˇcena funkcional- nost ni implementirana, jo moramo dopolniti z drugim orodjem. S tem pristopom se stvari zapletejo in jih je teˇzko obvladati. Ce aplikacijo ses-ˇ tavimo od zaˇcetka, jo lahko prilagodimo svojim posebnim potrebam. Ust- variti reˇsitev po meri pa pomeni tudi porabo veˇc virov. Ko smo razmiˇsljali, kateri pristop uporabiti za izdelavo aplikacije, smo se morali odloˇciti med dvema tehnologijama. Naˇs kandidat za orodje poslovne inteligence je bilo orodje Tableau. Kandidat za izdelavo reˇsitve po meri pa je bil Python, ker je programski jezik. Obe tehnologiji sta dobro razˇsirjeni in se pogosto uporabljata. Naˇsa odloˇcitev je bila predvsem pogojena s tem, da mora konˇcna aplikacija izpolnjevati vsa naˇsa merila. Za namen te naloge smo se odloˇcili ustvariti reˇsitev po meri z uporabo programskega jezika Python.
To odloˇcitev smo sprejeli, ker je bila to edina moˇznost za izdelavo popolne aplikacije. Izvesti smo morali bolj zapletene analize, ki z orodjem Tableau ni bilo mogoˇce narediti. Na teˇzavo smo naleteli, ko smo priˇsli do razprave o implementacijo metode DID in A/B testiranja v aplikacijo. Ugotovili smo, da teh dveh metod ni mogoˇce izvesti z uporabo orodja Tableau Desktop.
Za boljˇso uporabniˇsko izkuˇsnjo in produktivnost je aplikacija razdeljena na pet zavihkov: Sploˇsno, Stranke, Artikli, DID, A/B. Vsak zavihek je na- menjen drugemu vidiku promocije. Analize obsegajo osnovne sploˇsne analize in vse do bolj specifiˇcnih analiz. Uporabniki si s preklapljanjem med zavihki lahko ogledajo informacije, ki jih potrebujejo. Aplikacija je interaktivna, kar
pomeni, da lahko uporabnik izbere parametre in pridobi analizo, ki temelji na teh parametrih. Ti parametri se izbirajo s spustnimi seznami in izbirnimi gumbi. Spustni seznami in izbirni gumbi so na vseh zavihkih. Interaktivnost uporabniku omogoˇca analizo promocije z razliˇcnih vidikov. Uporabnik mora biti sposoben prepoznati vzorce, ki pokaˇzejo, zakaj je bila promocija uspeˇsna oziroma neuspeˇsna.
Doloˇciti je treba, kaj pomeni, da je promocija uspeˇsna ali neuspeˇsna.
Poenostavljeno bi lahko rekli, da je bila promocija uspeˇsna, ˇce so kupci s promocijo potroˇsili veˇc, kot bi brez nje. Imamo dve skupini kupcev, obrav- navano in kontrolno. Kupci v obravnavani skupini so tisti s kuponi, v kon- trolni skupini pa so kupci brez kuponov. Primerjamo porabo obeh skupin z namenom prepoznave razlik med njima. ˇCe so med njima velike razlike v porabi, lahko sklepamo, da je promocija na kupce imela vpliv. Za primer- javo uporabljamo dve metodi, A/B testiranje in metodo razlika v razlikah (metodo DID). Obe metodi primerjata prodajo pri obeh skupinah. ˇCeprav sta metodi podobni, se razlikujeta v osnovah. Rezultati metod so razliˇcni. Pri rezultatu promocije je metoda DID bolj optimistiˇcna. Kateri rezultati pa so pravilni? Na to vpraˇsanje ˇzal ni mogoˇce dati preprostega odgovora. Metodi se v osnovi razlikujeta. Glavna razlika med obema metodama je razvrstitev kupcev v obravnavano oziroma kontrolno skupino. Pri A/B testiranju so kupci z enako verjetnostjo nakljuˇcno razporejeni med obe skupini. Pri metodi DID nakljuˇcja ni. Kupce, ki so uporabili kupon, razvrstimo v obravnavano skupino, kupce, ki kupona niso uporabili, pa v kontrolno skupino. Medtem ko so lahko v obravnavani skupini pri A/B testiranju ukljuˇceni kupci, ki niso unovˇcili kupona, so pri metodi DID v obravnavani skupini vsi kupci unovˇcili kupon. Pri metodi DID lahko pride do pristranskosti pri vzorˇcenju.
Analiza z vkljuˇceno pristranskostjo pri vzorˇcenju bi lahko vodila do napaˇcnih zakljuˇckov. Ker sta na voljo tudi kontrolna in obravnavana skupina, ki sta doloˇceni nakljuˇcno, bi lahko A/B testiranje ˇsteli za natanˇcnejˇse, ker pri njem ni prisotne pristranskosti pri vzorˇcenju.
Chapter 1 Introduction
We all witness retail sales promotions, it is impossible to avoid them in the modern world. Promotion is a type of marketing activity where customers are informed they can get certain products at discounted prices [16]. When customers know they can buy certain items at lower prices than usual, their interest in shopping increases [14]. But is it only the customers who benefit from the promotion or the retailer as well? A promotion can have different objectives, some of them are: to attract new customers, to sell surplus stock, to introduce a new product and to encourage existing customers to buy more.
In this thesis, we analyze a promotion that encourages existing customers to buy more. The promotion targets customers who are becoming less loyal.
These customers are given coupons that allow them to get a discount on certain items. The goal is to encourage them to spend more. The promotion ran for two months. The coupons were issued in the first month and were redeemable in the second month.
The marketing department running the promotion does not know the outcome in advance. If the promotion was a failure, changes must be made for the next promotion. If it was successful, can we tell what led to success?
How should we determine whether the promotion was successful or not? We could simply say that the promotion was successful if the customers spent more with the promotion than would spend without it. If we can identify
1
2 Borislav Stojkovski the pattern that led to success, we might be able to use it to make the next promotion even more successful. To do this, the marketing department needs a tool that transforms sales data into insight. One way to solve this problem is to utilize a web application to analyze the sales data.
The thesis focuses on building an interactive web application accessible through a web browser. The goal is to develop an application that is intuitive and easy to use, so even users with backgrounds other than data analytics can use it with little training. The application should provide explicit informa- tion about the promotion’s outcome. Apart from promotion’s outcome, the application should also provide insight. We should be able find the patterns that led to the promotion’s outcome. This insight can be used to adjust the next promotion. For example, if we find that a particular customer segment responded better to the promotion than the other customer segments, we can target them in the next promotion, thus increasing the possibility for successful promotion.
There are two approaches for developing such an application: to build it from scratch or to use an existing business intelligence tool. The goal is to have a complete application that meets all of our criteria and provides all of the functionality we need. With business intelligence tools, we are limited to built-in functionalities, if a certain functionality is not implemented, we would need to augment it with another tool. This approach is complicated and difficult to manage. If we build the application from scratch, we can cus- tomise it for our specific needs and meet all of our criteria, but this does not come without tradeoffs. Building a custom solution requires more resources.
Before we decide on which approach to take, we need to fully understand the problem.
Because we are using two different methods for evaluating the promotion, they may produce different results. Which results are the correct ones in this scenario? Unfortunately, we cannot give a simple answer to this question.
The two methods differ in their fundamentals and both have advantages and disadvantages, and one may be better than the other in different scenarios.
Bachelor thesis 3 In Chapter 6.6, we discuss why they produce different results and which method we think is most accurate in our case.
The thesis consists of eight chapters. In Chapter 2, we describe and compare the tools we considered for building the application, we also present arguments for our decision. In Chapter 3, we describe our domain, explaining why we need such an application. In Chapter 4, we describe the two methods used to evaluate the promotion. In Chapter 5, we describe the data provided by the retailer. In Chapter 6, we explain how to use the application, and since the application is divided into five parts (tabs), we describe each tab separately. In this chapter, we also discuss why the different methods produce different promotion outcomes. In Chapter 7, we present information on the tools used to build the application, describe the project root directory with its parts and how they interact with each other. In Chapter 8, the final chapter, we present our results and suggestions for future work. In this chapter, we also provide the user feedback.
Chapter 2
Business intelligence tool or custom built solution?
When considering which approach to take when building the application, we had to decide between two technologies: Tableau was our candidate for a business intelligence tool, Python was our candidate for building a custom solution because it is a programming language. Both technologies are highly scalable and widely used. The key condition for our decision was that the application meets all of our criteria and provides all needed functionality.
2.1 Tableau Desktop
Tableau Desktop is one Tableau Software’s many products. Tableau Software is an interactive data visualization software company focusing on business intelligence and its software is commercially available. In 2008, Tableau Software was awarded ”Best Business Intelligence Solution” by the Software and Information Industry Association (SIIA) [9]. Tableau Desktop has a graphical user interface (GUI) that enables interactive data visualizations to be created easily and in a timely fashion, mainly by means of its drag-and- drop feature. Once the visualizations are created, they can be integrated into a dashboard. The visualizations can interact with each other if needed
4
Bachelor thesis 5 [8]. The visualizations created using Tableau Desktop are high quality and professional in appearance.
2.2 Python
Python is a high-level, general purpose, and open-source programming lan- guage. Python has frameworks and libraries that are suitable for various problems, resulting in Python being widely used. It is supported on all the major operating systems. One Python’s advantages over other program- ming languages is the readability of the code. Python has consistently been among the ten most popular programming languages in the TIOBE Program- ming Community Index since 2003, and as of 2021, the third most popular.
Many large organizations use Python, including Wikipedia, Google, Insta- gram, Nasa and Amazon [7]. Python is not a business intelligence tool, so we need to utilize frameworks and libraries suitable for this kind of problem to best develop an interactive web application.
2.3 Comparison
Both technologies have their advantages and disadvantages, so their utiliza- tion is case specific, meaning one technology may be better than the other for particular scenarios and vice versa, so we should need to understand which technology is better in which scenario. Accordingly, we present their advantages and disadvantages below.
Advantages of business intelligence tools:
Time efficient web application development because there is no need to develop an application’s infrastructure. Our job is to implement the business solution.
Less room for error because there is no need to worry about the appli- cation’s infrastructure.
6 Borislav Stojkovski
More stable because they are developed by professionals and the soft- ware is regularly updated.
Very little or no need for coding. The analyses can generally be per- formed with their drag and drop features.
Disadvantages of business intelligence tools:
Cannot perform all types of analysis because they are built-in function- ality limited.
Expensive because subscription charges for using the software are levied.
Advantages of custom built solutions:
Access to the application’s backend because we are developing it our- selves.
Can perform all types of analyses because we are writing code for the necessary analyses in the application’s backend.
Open-source technology, so there are no subscription fees.
Flexibility because we can adjust the application to meet our needs.
Disadvantages of custom built solutions:
Less time efficient web application development because we will need to develop the application’s infrastructure ourselves.
More room for error because we may unintentionally create a bug in the code while developing the application’s infrastructure.
It requires coding skills to develop the application’s infrastructure and to perform the required analyses.
Bachelor thesis 7
2.4 Our choice
We decided to develop our own solution using Python, doing so because it is the only way to create a complete application. We needed to perform analyses that were not possible with Tableau Desktop. Our key problem arose when we discussed implementing DID and A/B testing into the application, concluding that Tableau Desktop could not do this. Tableau Desktop is great for analysing existing datasets, but is poor when needed to perform calculations in order to create new datasets. It is complicated or impossible to perform more complex analyses. Its main drawback is its lack of flexibility when performing analysis that is not built in. Because our application is built from scratch using Python, flexibility is no longer an issue. It will be possible to perform the required analyses. The data can be manipulated in any way we choose with Python’s backend, and this is why we chose Python. Another of Python’s advantages is that it enables future application upgrades. If we decide we need additional functionality, we can implement it ourselves.
Chapter 3
Domain description
Promotion is a strategy that uses short-term campaigns to increase demand for products and services. It is a marketing activity by which customers are informed they can get certain products at discounted prices. Customers who are aware they can buy certain products at lower prices than usual are more interested in shopping. For this thesis, we have retail sales data from before and during a coupon promotion, which provides customers with discount coupons. These coupons have to be redeemed within a specific time period.
In our case, customers received coupons in November 2020 to be redeemed in December 2020. Customers get discounts on certain items when they redeem said coupons. The promotion targeted customers who were becoming less loyal and these customers received a coupon when their shopping receipts exceeded a given threshold, in this case ¿30. Each customer could receive up to three coupons. The goal was to encourage them to spend more.
When such promotions end, we need to be able to answer some questions to clarify its outcome. We can define the outcome of the promotion in terms of the additional sales it generates. Understanding of how one promotion ends allows us to better plan the next one. To best understand a promotion, we need to view it from different perspectives.
First, we need some general information about the promotion, includ- ing how it progressed over time, especially in relation to coupon issue and
8
Bachelor thesis 9 redemption. To best do this, we need to answer the following questions:
How many coupons were issued and redeemed?
Was coupon issue and redemption more active at the beginning or end of the promotion?
Which of the three coupons generated the most sales?
Which of the three coupons had the highest response rate?
Does the value of shopping carts differ in relation to which coupon the customers used to shop?
How many customers received one coupon, two coupons, and so on?
In addition to general information about the promotion, information on how customers responded to the promotion is of great benefit. Because we have different customer attributes, we need to perform analyses based on these attributes. Different customer segments may respond differently to the promotion. If we know how each customer segment responds to the promotion, we can better plan the next promotion. If we have information that a particular customer segment responded better than the others, the next promotion can target more of those customers. This information can be extracted from the following questions:
Which customer segments received the most coupons?
Which customer segments were most engaged?
Which customer segments were most active?
Because the coupon could be used to purchase multiple items, we are in- terested in knowing what the most frequently purchased items and categories are, and the following questions address this:
Which items were most frequently purchased by means of the coupons and which categories did they belong to?
10 Borislav Stojkovski
Which items and categories generated the most sales?
If the application does not enable promotion evaluation, it is of no use. It must provide explicit information about the promotion’s outcome expressed in terms of sales because this metric is the most important when evaluating the promotion’s success, and the following questions address this:
Was the promotion successful?
What is the average difference in spending between customers who received coupons and those who did not?
How much additional sales did the promotion generate?
On which customer segments did the promotion have the greatest im- pact?
Chapter 4
Methodology
It is necessary to define success criteria for the promotion to determine whether it was successful or not. We could simply say that if customers spend more with the promotion than they would have without it, the pro- motion was successful. We had two groups of customers, those who received coupons and those who did not. We compared their spending patterns to find a difference. If there is significant difference in spending, we can conclude that the promotion impacted on the customers.
We used A/B testing and Difference in Differences (DID) to evaluate cus- tomer behaviour in relation to the spending of the two groups of customers.
Although A/B testing and DID appear similar, they differ fundamentally and produce different results, DID being more optimistic about the promotion’s outcome.
4.1 A/B testing
A/B testing is a research methodology, consisting of a randomized exper- iment with two versions. A/B testing compares two versions of a single variable by testing the response to both versions, its goal being to determine which version performs better [13]. A/B tests are considered the simplest form of controlled experiment. A/B testing can be executed when there is
11
12 Borislav Stojkovski interest in introducing change but not sure if it will be more effective than the current version. We need a control group and a treatment group for this type of analysis, where samples are randomly allocated to both groups with equal probability. A/B testing requires that both versions are compared simultaneously, not one after the other [15].
To run a successful A/B test, we need to determine a few things. First, we need to decide which metric we want to improve. Second, we need to decide on the change we think will lead to said improvement. This could be expressed as, ”If we make this change, will there be improvement?”. The control group uses the standard version, the treatment group the experimen- tal version. We then compare the two groups’ outcome variables and if they differ, we can conclude that the treatment is effective.
Because we have a control group and treatment group, A/B tests are eas- ily executed. Customers in the treatment group receive coupons, and we can then compare the average spending of customers in both groups and deter- mine if customers in the treatment group evidence an increase in spending.
The average change in spending for the treatment group can be calculated using the following formula:
A/B =T −C (4.1)
Where T represents the average spending of customers in the treatment group during the promotion and C the control’s group.
We can now express the outcome of the promotion in terms of sales by using the following formula:
R=A/B∗T n (4.2)
Where Tn is the number of customers in the treatment group.
4.2 Difference in differences (DID)
DID is a statistical technique that mimics experimental research design when the control and treatment groups are not selected randomly. DID calculates
Bachelor thesis 13 the effect of a treatment. To perform DID, measurement data for two groups in two or more different time periods is required. In the first time period, neither of the groups is exposed to a treatment, in the second period, one of the groups is exposed to a treatment [11].
DID is best explained by using a hypothetical example with two groups and two time periods. Suppose we have two groups, denoted P and S, and two time periods, denoted 1 and 2. The example is visually displayed in Figure 4.1.
Figure 4.1: DID example with two groups in two time periods (Wikipedia).
Points P1 and S1 represent the outcome variable for both groups in Period 1. After measuring the outcome variable in Period 1 for both groups, Group P, our treatment group, receives the treatment. After the treatment, the outcome variable is once again measured for both groups in Period 2. The outcome variable in Period 2 is represented by P2 and S2 in Figure 4.1. Not all of the difference between P2 and S2 is an effect of the treatment because the groups did not start at the same point in Period 1. We marked Q as the outcome variable for Group P when the treatment was not applied. The difference between Q and P2 represents the treatment effect.
DID assumes a parallel trend, meaning the difference between the out- come variables for both groups should be equal in both periods when no
14 Borislav Stojkovski treatment is applied [17]. Both groups should follow the same trend. Or, to return to the example above in Figure 4.1, the P and S lines should have the same slope in the absence of treatment.
Since we have sales data for before and during the promotion, we can apply DID. Coupons were issued in the first month, in the second month, customers could make purchases with the coupons. The treatment group is the customers who used coupons. The control group is the customers who did not use coupons. Customers in both groups received coupons. After the promotion ended, we compared the spending patterns of both groups. If there is a difference in the spending of the customers in the treatment group, we can conclude that the promotion was effective. DID is calculated using the following formula:
DID= (T2−C2)−(T1−C1) (4.3) T1 and C1 represent average customer spending in the first month for the treatment and control groups respectively. T2 and C2 refer to the sec- ond month. The above formula evidences the average increase in customer spending in the treatment group.
We can now express the promotion’s outcome in terms of sales using the following formula:
R =DID∗T n (4.4)
Where Tn is the number of customers in the treatment group.
Chapter 5
Data description
The data for this thesis was provided by a large multinational retail corpo- ration and is composed of five tables: Sales, Events, Items, Customers and Groups. We have sales transaction data for two months. For this thesis, we used a subset of the real data, so it is impossible to draw conclusions about real sales. We will briefly describe each table and its attributes. In addition to the attributes used for this thesis, the tables contain other attributes that are not relevant for this thesis. Some of the attributes are shared by multiple tables. These attributes are usually unique identifiers for specific objects.
For example, the Customers table and the Sales table share the cust index attribute, which represents a unique identifier for individual customers. The shared attributes are used to join the tables. If we join the Sales and Cus- tomers tables, we can analyze sales based on specific customer attributes.
The shared attributes are described once. For privacy reasons, some of the attributes are not included in the figures following.
5.1 Customers table
The Customers table contains data for 10,000 customers. The different at- tributes determine customer segmentation. Customer segmentation is a pro- cess of dividing the customers into more homogeneous segments based on
15
16 Borislav Stojkovski certain common attributes. This allows us to analyze how the different cus- tomer segments responded to the promotion.
Figure 5.1: Part of the Customers table.
Column descriptions:
seg age: customer age. There are four age categories. This attribute can also be empty if customers did not provide their age. We created a new category for these customers.
seg freq: customer shopping frequency. There are three categories.
seg mon: customer weekly spending. There are four categories, from low to high spenders.
generacija: customer generation.
5.2 Items table
The Items table contains data about the items. There are different types of attributes for the 718,163 items we received data on. We use these attributes in the Items tab. For example, we analyze the top 10 item categories which were purchased with a coupon.
Column descriptions:
artikel: unique item identifier.
tksn 1: item category.
Bachelor thesis 17
Figure 5.2: Part of the Items table.
tksn 4: item sub-category.
naziv: full name of item.
5.3 Groups table
The Groups table contains data crucial for A/B testing. We have information on the 10,000 customers concerned as to which group they belong. There are two groups: the control group and the treatment group.
Figure 5.3: Part of the Groups table.
Column descriptions:
skupina: customer group.
18 Borislav Stojkovski
5.4 Events table
The Events table contains data on coupons. We have data for November 2020 and December 2020. For the first month, we have data on the 38,003 issued coupons. For the second month, we have data on the 18,512 coupons redeemed, each row representing purchase of one item.
Figure 5.4: Part of the Events table.
Column descriptions:
datum: date of coupon issue or redemption.
akcija id: coupon id.
dokument: shopping bill id, this column is empty in the records for coupon issue.
pozicija: item position on the bill, this column is empty in the records for coupon issue.
dogodek: category attribute indicating whether the row concerns coupon issue or redemption.
5.5 Sales table
The Sales table contains sales transactions for November 2020 and December 2020. We have data on the 1,458,505 item purchases in 150,648 shopping carts. Although the promotion occurred in December, November data is used for DID.
Bachelor thesis 19
Figure 5.5: Part of the Sales table.
Column descriptions:
kolicina: quantity of items purchased.
rvc1: margin value.
npv: net sales value.
Chapter 6 Application
The application is divided into the following five tabs for better user experi- ence and productivity: General, Customers, Items, DID and A/B. Each tab is dedicated to a different aspect of the promotion and can be used for general and specific analyses. Users can switch between tabs to see the information they need. For example, if users want to analyze customer engagement in the promotion, they open the Customers tab.
The application is interactive, meaning the user can select parameters and receive analysis based on these parameters. These parameters can be selected using dropdown menus and radio buttons. There are dropdown menus and radio buttons on all of the tabs. The interactivity allows the user to analyze the promotion from different perspectives. Users should be able to find patterns to determine why the promotion was successful or not.
In addition to the visible information on the charts, most of the charts also provide tooltip information. This information is accessed by hover action, meaning users get the additional information displayed on labels when they place the mouse cursor in close proximity, and said information is usually the data from which the chart was calculated. If a chart is calculated by dividing some numbers, it may be misleading when datasets are small. The charts are small and we cannot display all the information on them, moreover, if we put all of the information on the charts, they lose meaning. So, the hover
20
Bachelor thesis 21 action ensures that the user has access to the numbers and sample size of the dataset from which the chart was calculated. This is necessary to avoid drawing false conclusions.
6.1 General tab
The application’s first tab displays general information about the promotion, coupon analysis and a comprehensive overview of the promotion flow. This tab is primarily concerned with coupon issue and redemption.
Figure 6.1: Top of the General tab. Three metrics: number of coupons issued, number of coupons redeemed, and average coupon response rate. Two line charts: coupon issue over the month and coupon redemption over the month.
Three metrics are displayed at the top of the page: number of coupons issued, number of coupons redeemed, and percentage of redeemed coupons, which is the ratio of the previous two numbers.
Two dual-axis line charts are displayed below the aforementioned metrics, illustrating promotion progression over time, and this is where time patterns can be discerned.
The chart on the left of the page evidences coupon issue over the month, with the grey line evidencing number of coupons issued per day, the red line the cumulative sum of issued coupons. The end of the red line matches the
22 Borislav Stojkovski metric above, number of issued coupons. The chart has a tooltip, where two labels appear when the cursor hovers over: when the cursor hovers over the grey line, a label appears and evidences the exact number of coupons issued on that day, as this is difficult to read from the chart itself; when the cursor hovers over the red line, a label appears and evidences the cumulative number of coupons issued up to and including that day, and percentage of the total number of issued coupons up to and including that day. It is noticeable that the cumulative line follows a logarithmic trend, meaning most of the coupons were issued at the beginning of the promotion, which can be verified using the tooltip, with 18,326 out of 38,003, or approximately 48%, being issued by 5 October, 2020.
The chart on the right, following the same logic as the chart to the left, evidences coupon redemption over the month. The grey line evidences the number of coupons redeemed per day, the red line the cumulative amount.
This chart’s tooltip evidences: when the cursor hovers over the grey line, the exact number of coupons redeemed on the day; when the cursor hovers over the red line, the number of coupons redeemed up to that day, and percentage of the total number of redeemed coupons up to and including that day. Coupon redemption, unlike issue, is relatively evenly distributed over the month and follows the linear trend.
Figure 6.2: Centre of the General tab. Three bar charts: number of each of the three coupons issued, average value of cart when shopping with each coupon, and response rate for each coupon.
Bachelor thesis 23 The next section of the page is devoted to coupon analysis. Because the target group can receive and redeem up to three coupons, it is necessary to understand the performance of each coupon. The coupons were issued in the following order, 558687, 558675 and 558674, and the following bar charts allow visualisation of coupon performance, facilitating better understanding.
The bar chart on the left evidences the number of each of the three coupons issued, with the first coupon issue expected to account for the ma- jority of coupons issued. The number of coupons issued is sequential, with the first coupon being issued the most, and so on, and this is a metric of great utility.
The second bar chart evidences average basket value measured in net sales value. Even though the first coupon was issued the most, it was not the most profitable. The customers shopping with the third coupon evidenced the highest average basket value. If datasets for said charts are small, the charts may be misleading, and that is why we have included a tooltip with a label evidencing total shopping basket quantity and total sales, the bar chart being derived from these two numbers.. If the dataset is small, the chart is considered irrelevant.
The third bar chart evidences response rate per coupon, with the final coupon recording the highest response rate. The customers receiving the third coupon had already redeemed the first two, so they are more consistent in their shopping habits and the promotion had a positive effect on them.
The following formula evidences the number of additional sales each coupon generated:
N umberOf Coupons∗AverageBasketV alue∗CouponResponseRate (6.1) In Table 6.1 we have the sales generated by each coupon.
Although the first coupon was issued 6.1 times more than the third, it only generated 2.08 times more sales, and this is mainly because the third coupon has a response rate of 10% while the first has a response rate of 3.8%.
24 Borislav Stojkovski Coupon ID Quantity issued Generated sales
558687 26,896 ¿38,245
558675 6,721 ¿22,112
558674 4,386 ¿18,342
Table 6.1: Number of coupons issued for each coupon ID and generated sales.
Figure 6.3: Bottom of the General tab. Funnel chart evidencing quantity of coupons customers received.
The last chart on this page is the funnel chart, for which we created six customer groups, determined by the number of coupons issued to each customer, and placed each customer in one of them. The groups accordingly, range from one to six or more coupons. The funnel chart narrows because 95% of the customers received up to three coupons and only 5% received more than three. Customers were able to receive more than three coupons because they had several loyalty cards.
6.2 Customers tab
The application’s second tab is for customer analysis and this is the most important tab in terms of future promotion optimisation, as customer en- gagement is analysed in terms of customer characteristics. We can see how
Bachelor thesis 25 many coupons each customer segment received and their response rates. The dropdown menus enable to select the x-axis and y-axis of the charts. The four customer attributes described in Chapter 5 are the available options that can be chosen from, enabling customer analysis from different perspectives.
Figure 6.4: Top of Customers tab. Three metrics: percentage of customers who received a coupon, average customer response rate, and average number of items purchased with a coupon. Two heatmap charts: distribution of the issued coupons, percentages of customers who received coupons.
The three following metrics are found at the top of the page: percent- age of customers who received coupons, customer response rate and average number of items purchased with a coupon. Customer response rate is the number of customers who redeemed coupons divided by the number of cus- tomers who received coupons. These metrics are of utility when evaluating overall promotion outcome, for example, if a target response rate was set, we can see if it was achieved. The response rate metric tells about promotion engagement. The goal of any promotion is to increase response rate, the charts below provide useful insight.
Four heatmap charts are found below the aforementioned metrics to pro- vide detailed analysis of the customers based on their attributes, we can combine all of the customer attributes using the dropdown menus.
The first chart evidences issued coupons distribution, has a tooltip to
26 Borislav Stojkovski access actual numbers instead of percentages and evidences the number of coupons issued to each customer segment and from this we see that this promotion targeted customers from older generations with higher weekly net sales value. If we select seg age as our x-axis and y-axis, the chart will evidence the number of coupons issued to each age segment, as seen below in Table 6.2:
seg age Percentage
60-80 37.8%
40-60 36.5%
20-40 8.3%
nedoloceno 17.4%
Table 6.2: Distribution of issued coupons. How many coupons each age- determined customer segment received.
The second chart evidences the percentages of customers who received a coupon and is an extension of the first chart because it provides similar information. There is one outlier in this chart related to the customers belonging to the Generation Z segment, with a net sales value higher than
¿80, showing that all customers received coupons. However, using the tooltip action, we can see that there are only two customers in this segment, and both have received a coupon. Since the dataset from which the calculation was made is so small, this outlier can be considered irrelevant. Aside from using the tooltip action, this can also be verified by the first chart, which shows that 0.0% of the coupons ended up in this segment. In truth, this percentage is not 0, but has been rounded.
The third chart, evidencing the average number of items purchased by coupon, that is, customer segment activity, is calculated by dividing the number of purchased items by the number of coupons issued. We can view these numbers using the tooltip action.
The fourth chart enables customer response rate analysis, which is of utility for future promotion optimization. Customers from high response
Bachelor thesis 27
Figure 6.5: Bottom of Customers tab. Two heatmap charts: average number of items purchased by coupon, customer response rate.
rate segments are considered excellent candidates for future promotional tar- geting. For this particular promotion, the customers with higher net sales values evidence higher response rates. The tooltip in this chart accesses the raw data used to calculate it. If we select seg mon as the x-axis and y-axis, we get response rates in terms of weekly net sales value per customer. The results are shown in Table 6.3:
seg mon Response rate Sample size
TPV >= 80 45.4% 485
40 <= TPV <80 34.2% 1259 20 <= TPV <40 19.3% 1339
TPV< 20 7.8% 893
Table 6.3: Customer segment response rates, which are determined by their weekly net sales values (TPV). Number of customers belonging to each seg- ment.
We can conclude from this chart that customers with higher net sales val- ues evidence higher response rates because the datasets used are of sufficient size.
28 Borislav Stojkovski
6.3 Items tab
The application’s third tab enables item sales analysis, as it evidences in- formation about the top categories and items purchased with coupons. Top selling items and categories can be identified in terms of quantity sold and sales generated. The two approaches produce different results and there is a radio button to select the appropriate one. If we select the quantity sold, the most frequently purchased categories and items are found at the top, such as everyday household items, including milk, bananas and bread. If we select sales generated, we find more expensive items at the top because they gener- ate more sales even though they are less frequently purchased. However, we cannot conclude from this tab whether the promotion was successful, or not because it does not contain data on customers who didn’t receive coupons, so cannot be used for future promotion optimization.
Figure 6.6: Top of Items tab. Two bar charts: top 10 categories purchased with a coupon, top 10 items purchased with a coupon.
Two bar charts are found at the top of the page. The bar chart on the left evidences the 10 categories most frequently purchased with coupons, determined by the tksn 4 attribute, a subcategory of the tksn 1 attribute, because tksn 1 is too general and we need specific information. The bar chart on the right evidences the top 10 purchased items with a coupon. Both charts have tooltips that show net sales value and quantity sold regardless of which approach we select to calculate the top categories and items.
Bachelor thesis 29
Figure 6.7: Bottom of Items tab. Treemap chart evidencing the top 10 categories purchased with a coupon and each category’s top 10 items.
The final chart on this tab is a treemap chart, which is an expanded combination of the first two charts, in which the top 10 categories and their top 10 items are displayed. The chart is tksn 1 coloured to make it easier to identify the top categories.
Figure 6.8: Treemap chart evidencing the top 10 items purchased with a coupon within a given category.
This chart is interactive, meaning that when users select a category, the items within it are displayed, including data on quantity sold, net sales value generated and total category sales share.
6.4 DID tab
The application’s fourth tab enables DID sales data analysis, which is of utility when evaluating promotion’s outcome because it determines success
30 Borislav Stojkovski in terms of additional sales generated. DID is described in Chapter 4.
Figure 6.9: Top of DID tab. Two metrics: average increase in sales per customers in the treatment group, additional sales generated by the promo- tion. Two line charts: average daily spending per customer group before the promotion, average daily spending per customer group during the promotion.
Two metrics are found at the top of the page: average net sales value increase per customer in the treatment group and the cumulative outcome of the promotion expressed in terms of sales. The first metric is calculated using the formula (4.3), the second using the formula (4.4). According to DID, the promotion generated an additional ¿31,786.
Two line charts are found below the metrics: the one on the left evidences the average daily spending per customer group before the promotion and the one on the right evidences the average daily spending per customer group during the promotion in terms of net sales value or margin value, which are selected by using the radio button at the top. We divided the customers into three groups: customers who received a coupon and redeemed it (treatment), customers who received a coupon but did not redeem it (control), customers who did not receive a coupon. There is an outlier in the chart on the right: a customer group evidenced unusually high average daily spending on October 25, 2020. However, when we use the hover action, we see that only three
Bachelor thesis 31 customers purchased items on that day, so pursuant to the dataset being very small, we cannot draw meaningful conclusions. Both charts have the same hover action, enabling access to the data used to calculate them.
We cannot draw any conclusions about the promotion’s success because it is difficult to deduce anything meaningful from the daily data, so we have added tables below the line charts, containing groups’ monthly averages to facilitate this.
Figure 6.10: Bottom of DID tab. Two heatmap charts: DID results, promo- tion outcome in terms of sales.
Below the line charts are two heatmap charts. The left chart shows the re- sults from the DID method. This chart is calculated using the formula (4.3), with the difference that the results are displayed by customer segments in- stead of as individual results. By displaying the results in segments, we get a more detailed analysis. In this chart, we can see for which customer seg- ments the promotion had the greatest impact. The heatmap has a hover action that displays all the elements of the formula (4.3). The most impor- tant information we get from the tooltip actions is the sample size of each customer segment. This is useful when we want to assess whether the results are relevant. For example, if a small customer segment happens to be high spenders, we cannot conclude that all customers belonging to that segment have the same spending behavior.
Two heatmap charts are found below the line charts: the chart on the left evidences DID results, which is calculated using the formula (4.3), with the
32 Borislav Stojkovski difference in results displayed by customer segment, not as overall results.
We get a more detailed analysis when results are displayed according to customer segments because we can easily see on which customer segments the promotion had the greatest impact. The heatmap has a hover action that displays all of the elements in the formula (4.3). The most important information we get via the tooltip is the sample size of each customer segment, which is of utility for assessing result relevance because we cannot draw meaningful conclusions from small datasets. For example, if customers of a small dataset happen to be high spenders, we cannot conclude that all customers belonging to that segment have the same spending behavior.
The chart on the right evidences promotion’s outcome expressed as sales, which is calculated using the formula (4.4). The sum of this chart matches the metric above, promotion’s outcome expressed as sales.
6.5 A/B tab
The application’s final tab enables A/B testing sales data analysis. A/B testing is described in Chapter 4. This tab has an identical layout to the DID tab and along with the DID tab enables promotion evaluation because it evidences additional sales generated by the promotion.
The same two metrics as in the DID tab are found at the top of the page, but they are calculated by A/B testing. The first metric is calculated using the formula (4.1), the second metric is calculated using the formula (4.2).
A/B testing metrics are less optimistic that those derived using DID. Ac- cording to the A/B tab, the promotion generated an additional ¿27,526 in sales, which is 13.4% less than that derived using the DID. We discuss in Chapter 6.6 why does this difference occurs.
Two line charts are found below the metrics, evidencing average daily cus- tomer group spending. This time, customers have randomly been allocated to either the control or treatment group. These charts provide the same information and have the same hover action as the charts in see Figure ??.
Bachelor thesis 33
Figure 6.11: Top of A/B tab. Two metrics: average increase in sales per customer in the treatment group, additional sales generated by the promo- tion. Two line charts: average daily spending per customer group before the promotion, average daily spending per customer group during the promotion.
Figure 6.12: Bottom of A/B tab. Two heatmap charts: A/B testing results, promotion outcome in terms of sales.
Two heatmap charts are found below the line charts which follow the same logic as the heatmaps in the DID tab.
The chart on the left shows A/B testing results, that is, the difference in spending between the control and treatment groups. On this chart, we can see where the promotion had the greatest impact according to A/B testing.
At first glance, the promotion had the greatest impact on Generation Z
34 Borislav Stojkovski customers, with a weekly net sales value of between ¿20 and ¿40. For these customers, the chart shows an average increase of ¿45.5 in sales. Compared to the other customer segments, this appears to be an outlier. With the hover action, which evidences dataset size, we can see there are 11 customers in the treatment group and 10 in the control group. The datasets are too small to draw meaningful conclusions.
The second heatmap shows additional sales per customer segment and the sum derived from this heatmap represents promotion’s outcome expressed in terms of sales.
6.6 Discussion on why DID and A/B results differ
The main difference between the two methods lies in the way they allo- cate customers to the treatment and control groups. A/B testing randomly assigns customers into both groups with equal probability. DID lacks ran- domness, assigning customers who used a coupon to the treatment group and customers who did not use a coupon to the control group. While the A/B testing treatment group may include customers who did not redeem a coupon, the DID treatment group consists only of customers who redeemed a coupon, and this does not accurately represent the population. For this reason, the DID treatment group may evidence higher average spending than the A/B testing treatment group, leading to incorrect conclusions. DID is vulnerable to sampling bias, meaning some members of the intended pop- ulation have a lower or higher sampling probability than others [12]. This is one of the reasons why DID testing produces more optimistic promotion outcomes than A/B testing.
We can ascertain the difference in customer group spending before and during the promotion because we have tables evidencing monthly averages for customer group spending. We can check if the groups had equal or nearly equal spending before the promotion and if this is not so, there is sampling
Bachelor thesis 35 bias because the groups do not represent the same population. In Figure 6.11, the groups’ average net sales values pursuant to A/B testing only differs by 0.1% before the promotion, meaning we can assume that both groups rep- resent the same population. The difference during the promotion of 11.8%
can be attributed to the promotion effect. In Figure 6.9, the groups’ average net sales values pursuant to DID differ by 13.7% before the promotion and by 22.4% during the promotion. The high difference between the two groups before the promotion can be attributed to sampling bias because DID’s treat- ment group is composed only of customers who redeemed the coupons they received, so we can assume they are more loyal and generally spend more.
Consequently, not all of the 22.4% difference during the promotion can be attributed to the promotion effect.
Because A/B testing randomly assigns customers into the control and treatment groups, it is considered more accurate because there is no sampling bias. Moreover, we are convinced that the assumption of comparing both groups simultaneously holds. In contrast, DID is sampling biased, and we cannot be convinced that the assumption of parallel trend holds. However, if we did not have randomly assigned control and treatment groups, the DID could be a good alternative to A/B testing.
Chapter 7
Application infrastructure
7.1 Tools
The application was built in Python (3.8.5) augmented with the following open-source frameworks and libraries:
Dash [2]: The backend part of the application was done using the Dash framework. Dash is a framework built on top of Flask, Plotly.js, and React.js. Flask is a Python framework used for building web ap- plications [3]. Dash is an excellent choice for building web analytic applications because it abstracts the many technologies and protocols required to build an interactive web-based application.
Pandas [5]: The Pandas library was used for data analysis and data manipulation. Pandas offers data structures for storing data efficiently.
Pandas has a lot of built-in functions and operations for manipulating numerical tables and time-series data. Pandas makes common data operations easy to manage.
Plotly[6]: Data visualizations were generated using the Plotly library.
Plotly is an interactive plotting library which supports numerous chart types. Its charts are interactive and of high quality. Plotly’s plots are easily rendered by Dash.
36
Bachelor thesis 37
dash-bootstrap-components[1]: The application’s layout was done using dash-bootstrap-components, which is a Python library with Boot- strap components, such as dropdown menus, radio buttons, rows and columns.
Jupyter Notebook [4]: A web application used for interactive com- puting. The notebook consists of input and output cells. The backend can run in many programming languages, including Python. We used it during the development stage for testing and exploring new ideas.
Visual Studio Code [10]: A code editor that can be used with a variety of programming languages, including Python. It has extensions for code highlighting, debugging, autocomplete, and much more. These extensions make coding easier and more efficient.
7.2 Application development
Application development was divided into two parts. The first part was done using the Jupyter Notebook. We created a notebook for each of the application’s tabs and each tab was developed independently. We explored different types of analyses and visualizations. We started with basic analyses and visualizations, then added more advanced ones. In this part, our goal was to create many visualizations from which to select the ones which provide the most relevant information. These visualizations were further improved by means of additional functionalities.
Once we finished the notebooks, we started integrating them into an application using Visual Studio Code. We needed to make the code more compact. We started by identifying code repetitions in the notebooks, then we organized this code into functions and placed them in modules, which are imported and used across multiple tabs. Each module contains related functions. For example, there is a module containing the visualization func- tions, which accept arguments and return charts. Three of the tabs generate
38 Borislav Stojkovski heatmap charts by means of the same function. With this approach, we implemented the software principle of ”do not repeat yourself”, which is beneficial when debugging code or when it needs updating, as code change is done only once instead of several times.
7.3 Application structure
The application’s structure is shown in Figure 7.1. It is structured in a way that related files are placed into packages and directories. The application is built in a way so it can read data from a database or a local directory. The ap- plication’s source code is available at https://github.com/borislav1/diploma.
Here is a short description of each file in the application’s directory:
Figure 7.1: Application’s root directory.
Bachelor thesis 39
app.py: a script that defines Dash instance and its parameters.
index.py: a script that glues all five tabs onto a navigation bar and defines its URLs. This script has to be executed to run the application.
datasets: a directory that contains the five dataset tables necessary for the application: customers.csv, sales.csv, events.csv, items.csv and groups.csv.
apps: a package that is further organized into three sub-packages.
vizs: a sub-package responsible for data visualizations. It contains two Python modules:
– graphs.py: the functions that render charts are defined here.
– templates.py:the tooltip templates displayed by hover action are defined here.
data conntector: a sub-package that deals with data reading and data preprocessing, it contains two Python modules and two JSON configuration files:
– db config.json: the configuration data for connecting to the database has to be entered here.
– sql queries.json: SQL queries for each data table (customers, items, events, sales groups) in the database have to be written in this file.
– reader.py: this is composed of two Python functions, one for reading data from a database, the other for reading data from a local directory, and this is where users define from where the data will be read.
– data prep.py: this module preprocess and prepares the data for further analysis.
tabs: this sub-package is the heart of the application and contains the following five modules: general.py, customers.py, items.py, did.py and ab testing.py, that is, one for each of the application’s tabs. Each of them performs the data manipulation and analyses needed for indi-
40 Borislav Stojkovski vidual tabs. Each module contains listeners for user actions, such as changing the value of dropdown.
Chapter 8 Conclusions
In this thesis, we have presented an interactive web application for analyzing and evaluating a promotion. The application contains different tabs to ana- lyze the promotion from different perspectives. We have seen the application in action with real data and were able to gain insight. With this insight, we are better able to plan the next promotion. We can say without a doubt that the application helped us better understand the promotion. We were able to identify the most active customer segments. This information is of great importance when preparing subsequent promotions.
We found a possible reason for the difference in results between promotion evaluation methods. Our conclusion is that whenever there are randomly assigned control and treatment groups, A/B testing should be preferred.
The thesis evidences that building a custom solution is the better option in our case. Building a custom solution is not as time efficient as using a business intelligence tool, but it is personalized and consequently of greater utility. This ensures that our application satisfies user needs. Although the application is not being used in production, we believe that its development was successful because it met our expectations.
The application was presented to the retailer’s employees from different departments, including the marketing department, the business intelligence department, and executives. Some found it difficult to understand all of the
41
42 Borislav Stojkovski application’s parts. Apart from those in the business intelligence department, the others experienced DID for the first time. They were confused why we have two different results for the promotion outcome. Even so, the general reaction to the application was positive. The marketing department found it different from their existing method of evaluating promotions. They cur- rently use standardized reports to evaluate promotion and receive the same type of report after each promotion. Although reactions to the application were positive, it did not replace their existing method to evaluate promo- tions. The marketing department considers it advanced and said they would need time to get used to it. Although the application is not one of their daily work tools, some parts of it are included in their standard reports.
Like most applications, this one also has room for improvement. The application is not appropriate for use in production because of possible per- formance issues. One performance issue being that the application is slow when processing large amounts of data. Calculations are performed in real time because the application is interactive, meaning that when users per- form an action, such as changing the value of the drop-down menu or radio button, calculations are performed simultaneously. The calculations are not time efficient when processing large datasets, meaning users have to wait for results in our case. One way to solve this problem is to cache the results in an in-memory database like Redis. The results are temporarily stored in the database once a combination of input parameters is calculated. If the same parameters are used subsequently, the results are easily retrieved from the database. This approach of caching the results instead of recalculating them each time greatly improves the performance of the application.
Bibliography
[1] dash-bootstrap-components library. https://dash-bootstrap- components.opensource.faculty.ai/. [Accessed: 28. 7. 2021].
[2] Dash framework. https://dash.plotly.com/. [Accessed: 28. 7. 2021].
[3] Flask framework. https://dash-bootstrap-components.
opensource.faculty.ai/. [Accessed: 28. 7. 2021].
[4] Jupyter notebook. https://jupyter.org/. [Accessed: 8. 8. 2021].
[5] Pandas library. https://pandas.pydata.org/. [Accessed: 28. 7. 2021].
[6] Plotly library. https://dash.plotly.com/. [Accessed: 28. 7. 2021].
[7] Python. https://en.wikipedia.org/wiki/Python_(programming_
language). [Accessed: 8. 8. 2021].
[8] Tableau desktop. https://www.tableau.com/products/desktop, . [Accessed: 8. 8. 2021].
[9] Tableau desktop wikipedia. https://en.wikipedia.org/wiki/
Tableau_Software, . [Accessed: 8. 8. 2021].
[10] Visual studio code. https://code.visualstudio.com/. [Accessed: 8.
8. 2021].
[11] Joshua D. Angrist and Alan B. Krueger. Chapter 23 - empirical strate- gies in labor economics. Handbook of Labor Economics, pages 1277–
1366. Elsevier, 1999.
43
44 Borislav Stojkovski [12] Corinna Cortes and Mehryar Mohri. Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science, 519:103–126, 2014.
[13] Emilie Kaufmann, Olivier Capp´e, and Aur´elien Garivier. On the com- plexity of a/b testing. InConference on Learning Theory, pages 461–481.
PMLR, 2014.
[14] Azam Kaveh, Mohsen Nazari, Jean-Pierre van der Rest, and Seyed Abol- ghasem Mira. Customer engagement in sales promotion. Marketing Intelligence & Planning, 2020.
[15] Ron Kohavi and Roger Longbotham. Online controlled experiments and a/b testing. Encyclopedia of machine learning and data mining, 7(8):
922–929, 2017.
[16] Jennifer Rowley. Promotion and marketing communications in the in- formation marketplace. Library review, 1998.
[17] Guido Schwerdt and Ludger Woessmann. Chapter 1 - empirical methods in the economics of education. In The Economics of Education (Second Edition), pages 3–20. Academic Press, second edition edition, 2020.
