Adding Automatic Anomaly Detection Functionality to Mixpanel
Mixpanel machine learning algorithms surface insights and detect anomalies in the project data. These automatic insights can be found in the Notifications menu of a project and will provide links to more details about the surfaced information. Anomaly Detection, Explains, and Automatic Segmentation are all Mixpanel Machine Learning models that aid in uncovering data that requires attention but may have otherwise been missed.
Mixpanel users often miss important events in analytics reports.
AI-powered notifications system supporting all reports in Mixpanel that detects important events in reports and notifies the user about them.
We started the project with some whiteboarding sessions where I, a PM, and one of the developers discussed different options where we can show notifications in Mixpanel and how we approach the design of Notification Center. The developer helped us identify the reports where detecting anomalies would be most challenging from the technical perspective. This helped us to select which reports will initially support this feature.
We conducted a series of interviews with Mixpanel's power users that indicated a strong need for such a feature. Talking to these users helped us to understand what they expect from it. We realized that we need to add smart anomaly detection to all reports and to push the most important notifications to the Notification Center.
After that, we sketched out a few options for the Notification Center. We analyzed several companies that use different approaches and stopped by the Invision approach and Facebook approach. If we would do it the Invision way it would be a sliding panel with emphasized filtering by anomaly type. The Facebook way would be more minimalistic and easy to build. We made 2 prototypes and tested them with power users. The test group preferred Facebook approach as it was easier to understand. We already knew it would be more scalable as we could reuse it in the reports. Everyone on the team agreed that it was the way to go.
We tested several options of how we can indicate anomalies in different scenarios: on a graph, in a table, on a bar, etc. It was pretty clear that the bell is the right metaphor for the Notification Center but to indicate anomalies we chose to show a bright circle with an icon inside. Different types of anomalies would have colors and icons assigned to them. Although it did not work for highlighting cells in tables. It was too large and we came up with a smaller dot that expands by mouseover.
While working on the AI algorithms for the project, our engineers discovered that user's feedback on insights will drastically improve their quality. We explored several ways of asking for the feedback and on the later stage of the project decided to add the "Other" option to let the users type in their answer.
When we were working on the project, another team was preparing a redesign for the whole Mixpanel product. We had to make sure everything we build would work equally well in old and new Mixpanel.
Highliting anomalies in different reports
In old Mixpanel
And in the new Mixpanel
We successfully launched the anomaly detection feature for the 3 most used report types and then extended to all Mixpanel reports. The feature changed the way people interact with the product and become many customers' favorite feature: