28_voltage_effect.md

The Voltage Effect

Reasons for a drop in voltage

1. False positives, e.g., DARE

2. Selection Bias: Initial sample != population, e.g., Opower.

3. Non-negotiables

   • constraints not understood initially
     • e.g., success dependent on presence of certain people
     • quality control at scale is harder
   • being tampered because of various reasons
     • fidelity of treatment is lost (mission drift) because people less motivated, etc.
       • solution: adding original scientist or founder on the implementation team, teaching implementors why each aspect is important

4. Behavioral Adaptation

   • e.g., risk compensation may cause seat belts to be less effective (esp. over time)

5. General Equilibrium

   • Rise in base fares for uber drivers -> driver earnings back to the usual by week 6 because new drivers enter the market. Apparently same when they instituted tipping.
   • $5 coupons on Friday afternoon rides (people who got the coupon vs. not in the same market)
   • Large cash infusion in villages in Siaya -> untreated people also better off
   • FB/Linkedin
   • Reducing class size -> more teachers needed -> lower quality, etc.

6. Basic financial planning: "Once you’re ready to launch, calculate what your ongoing operating costs will be, and then assume they will be at least 10 percent higher than expected. Then look at your estimated revenues versus those operating costs and try to figure out how much you need to scale before your idea will be profitable (or, conversely, how much time you have before your coffers run dry)."

7. Org. design may be important for scaling -- anecdata. Uber/Kalanick issues.

Other points

1. "My thoughts were that a subscription model could scale only if it gave access to goods or services that weren’t readily available and accessible without a subscription."
   Arthur Lewis’s 1941 paper on “two-part tariffs”—economese for a business requiring customers to pay an entry fee to gain access to products they then pay for. In a 1971 paper titled “The Disneyland Dilemma,” economist Walter Oi provided the theoretical conditions ... can you induce people to buy more or does it lead to self-selection of prosumers. For instance, amz. prime.

2. Resentful demoralization

   • people who learned their peers earned more, worked less hard at canvassing, etc.
   • After analyzing the results, they found that employees who were told their manager made 10 percent more than they originally thought actually worked slightly harder; the average number of hours they worked increased by 1.5 percent over the next ninety days. Conversely, when employees learned that peers made 10 percent more than they did, they worked 9.4 percent less over the next ninety days.

3. Tipping

   • When we looked at the data on how often passengers tipped—or didn’t tip—we found that only 1 percent of Uber passengers tipped on every trip they took. That’s right, only one in every hundred people was sure to tip their driver! Meanwhile, 60 percent of people never tipped, and 39 percent sometimes did.

4. Loss Aversion

   • clawback: we gave it to one group of employees before they had actually earned it. The catch was, we told them, it was provisionally allocated, to be disbursed not immediately but at the end of the week, provided they met their production ... "the clawback approach outperformed the classic bonus approach, with total team productivity increasing by over 1 percent in the loss treatment group—an effect that didn’t diminish over the course of the six-month experiment."
   • "in the suburbs of Kampala, Uganda, we had 1,200 workers who sort beans participate, and we saw an increase in productivity that blew Wanlida away—a staggering 20 percent!"
   • "at the end of the school year based on the improvement (from September to June) in their students’ overall percentile scores on a standardized test—$80 per percentile, with a maximum possible reward of $8,000." "What did we find? The teachers in the loss group saw tremendous gains in their students’ test scores. Their desire not to lose the bonus they had already been given had indeed incentivized teachers to work harder—even harder than the teachers who were promised a bonus of equal value later. On top of all of this, the teachers in the clawback group seemed to have developed some good habits: in looking at teacher quality for the next five years after our experiment ended, we found evidence that their improved performance in the classroom lasted, even in the absence of the incentives—which meant that students who had one of these teachers"

5. Social pressure

   • The first group received reports on their fuel efficiency from the previous month; the second received the same type of reports, with the addition of a message encouraging them to reach the personal targets of fuel conservation that we created for them; and the third group received the same reports and encouragement as the second group, and were also informed that a small donation to a charity of their choice would be made in their name for each of three targets met (this is called a “prosocial incentive”). The fourth group, the control group, received a letter merely notifying them that their fuel usage would be measured.

   • So the company implemented a program called the Home Energy Report (HER), a social nudge that involves sending out periodic ... average consumers reduced their electricity consumption by 2.4 percent when they received information comparing their usage to that of the other households in the area.

6. Marginal thinking

   • Spend $1M on FB ads for say 10k customers for an average of $100 CAC. But last $10k of spend probably much lower marginal returns. Useful to verify as you saturate a market.

Didn't pass the smell test for me

1. "we had a treatment group of potential applicants, who saw a posting with an EEO statement, and a control group, who saw a posting with no such statement. Nearly 2,500 people applied. We also offered potential job-seekers a $10 ... It turned out that the EEO statement discouraged racial minorities from applying, and by as much as a shocking 30 percent. "

2. $10 canvassers in van where people in mixed van ($10 and $15) were less productive and stole more.