Simon McManus

GitHub - facebook/Ax: Adaptive Experimentation Platform

Ax is an accessible, general-purpose platform for understanding, managing, deploying, and automating adaptive experiments.Adaptive experimentation is the machine-learning guided process of iteratively exploring a (possibly infinite) parameter space in order to identify optimal configurations in a resource-efficient manner. Ax currently supports Bayesian optimization and bandit optimization as exploration strategies. Bayesian optimization in Ax is powered by BoTorch, a modern library for Bayesian optimization research built on PyTorch.

tagged:

facebook

experimentation

08-01-2023

Under the Hood of Uber's Experimentation Platform | Uber Blog

Experimentation is at the core of how Uber improves the customer experience. Uber applies several experimental methodologies to use cases as diverse as testing out a new feature to enhancing our app design.

tagged:

uber

experimentation

08-01-2023

Experiment Reporting Framework. By Will Moss | by AirbnbEng | The Airbnb Tech Blog | Medium

At Airbnb we are always trying to learn more about our users and improve their experience on the site. Much of that learning and improvement comes through the deployment of controlled experiments. If you haven’t already read our other post about experimentation I highly recommend you do it, but I will summarize the two main points: (1) running controlled experiments is the best way to learn about your users, and (2) there are a lot of pitfalls when running experiments. To that end, we built a tool to make running experiments easier by hiding all the pitfalls and automating the analytical heavy lifting.

tagged:

airbnb

experimentation

reporting

08-01-2023

Scaling Airbnb’s Experimentation Platform | by Jonathan Parks | The Airbnb Tech Blog | Medium

At Airbnb, we are constantly iterating on the user experience and product features. This can include changes to the look and feel of the website or native apps, optimizations for our smart pricing and search ranking algorithms, or even targeting the right content and timing for our email campaigns. For the majority of this work, we leverage our internal A/B Testing platform, the Experimentation Reporting Framework (ERF), to validate our hypotheses and quantify the impact of our work. Read about the basics of ERF and our philosophy on interpreting experiments.

tagged:

airbnb

experimentation

08-01-2023

4 Principles for Making Experimentation Count | by Lindsay M Pettingill | The Airbnb Tech Blog | Medium

For over two years I’ve been a Data Scientist on the Growth team at Airbnb. When I first started at the company we were running fewer than 100 experiments in a given week; we’re now running about 700.

tagged:

airbnb

experimentation

08-01-2023

Building Pinterest’s A/B testing platform | by Pinterest Engineering | Pinterest Engineering Blog | Medium

As a data-driven company, we rely heavily on experiments to guide products and features. At any given time, we have around 1,000 experiments running, and we’re adding more every day. Because we’re constantly increasing the number of experiments and logging corresponding data, we need a reliable, simple to use platform engineers can use without error. To eliminate common errors made by experimenters, we introduced a lightweight config UI, QA workflow and simplified APIs supporting A/B testing across multiple platforms. (For more information about our dashboards and data pipeline, check out our previous experiments post.)

tagged:

experimentation

08-01-2023