Last year, I made a career move from a smaller startup to Fora. One of the things I appreciate about Fora is that we are a small team that operates a lot like a startup, but with a vast portfolio of online communities and tens of millions of monthly visitors.

Joining Fora was an exciting leap for me. It presented an opportunity to work on a product with a large user base – one that would enable our team to test new designs using quantitative user insights and to make data-informed decisions.

Coming from a startup that relied primarily on qualitative research with potential target users, the idea of incorporating quantitative data in the design process was appealing to me. However, as powerful as quantitative data can be, numbers alone don’t give us the full picture of how well a design works, as I was reminded of during a recent design challenge.

Earlier this year, I worked on a design to give users the ability to browse thread content more easily. We collapsed long posts that previously required extensive scrolling before a visitor could view any replies. We hypothesized that collapsing these posts would allow users to see more posts in fewer scrolls and follow discussions more easily. Other community sites offer similar functionality, so our team felt fairly confident this would work.

I put together designs, reviewed them with my fellow EPD (Engineering, PM, and Design) leads, iterated based on our community managers’ feedback, and conducted an A/B experiment on several production sites. The test data showed positive engagement, with a drop in bounce rate and increase in session duration. After reviewing the quantitative data with leadership, we rolled out the feature globally across all of our sites. Happy ending in hand, we moved on to the next design project.

A few weeks later, we conducted a user survey for an unrelated experiment. To my surprise, a number of users noted their unhappiness with the collapsed post changes. They were frustrated that they had to expand almost every post to read it. Users typically don't go out of their way to provide feedback unless they really like something and want to see more of it, or they really dislike something and want it fixed. The fact that there was a noticeable pattern in the feedback made it clear that the pain point had to be resolved. Together with my fellow EPD leads, we identified and launched a number of design improvements. Since then, we’ve seen a significant drop in the number of user complaints related to collapsed posts.

You might have heard the saying that unhappy users are the greatest source of learning. Negative user feedback is inevitable. What matters most in determining the success of the product is how we respond and learn. This starts by ensuring that there is an easy way for users to share their opinions.

Here are the top three lessons I learned through this experience:

1. An increase in engagement – in this case, as evinced by more clicks and longer session durations – doesn't always mean users are getting more value. On the surface, they might be engaging more, but on the inside, they might be stewing with frustration. It’s important to talk with users, and complement the numbers with qualitative insights to increase confidence about how users are really feeling about the design changes.

2. Using real user content for design QA is essential to catching UX issues early on. In this case, design QA was initially performed on a staging environment with mocked content. Real user content is almost always more complex and nuanced. For example, users like to add quotes, customize signatures, and structure posts in unique ways. This resulted in use cases that our design originally didn’t account for. Had we performed the design QA on real user content, I think it’s likely that we would have identified some of these UX issues earlier.

3. EPD collaboration is key to quickly and efficiently resolving user pain points. It’s important to have a strong, close-knit team with confidence and trust in each other so that we can all contribute to creating great user experiences.