Sticker Mule built a social network from scratch

https://www.stimulus.com/stickermule

Good feedback, agree completely :)

We use Split: https://github.com/splitrb/split

Most managers decide things without tests. Doing this well is easier said than done, but it's the reality of why some companies outperform others.

Exceptional growth doesn't happen because you're great at a/b testing. It happens when your team is good at decision making period.

Thanks for commenting on the post. :)

To take another contrasting opinion. Who cares if you know exactly why you're going. We know in a general sense why we're growing: we constantly improve the experience of our service and make customers happier.

It's quite hard to maintain aggressive year over year growth targets. If you're pulling it off it really doesn't matter if you figured out why via a/b testing or not.

Obviously, we'd prefer stronger YOY growth without knowing the exact causes than weaker YOY growth because we wasted resources on a/b testing to get precise answers to why we're succeeding.

Beyond that, I wasn't referring to trending growth via GA to see if you're making good decisions. You can trend conversion rate by channel to see if you're getting better over time. Obviously, if conversion is going up and to the right for most channels you're making good decisions.

It's unfortunate to dismiss another view point as you're frustrated or inexperienced while ignoring the actual points made.

The biggest point being #1: Resources are finite. Time spent on A/B testing cannot be spent elsewhere. How you use your time determines your growth trajectory. A/B testing is a growth tactic, but it's not always the best use of your resources.

There is absolutely no way to solve this problem. You'll always have a trade off to make when it comes to how you spend your resources and time. You shouldn't neglect that cost when deciding if a/b testing is necessary for your organization.

Btw, keep in mind, I admit we still do some a/b testing. It's just not an key component of our growth strategy.

I agree it might not always be the best use of resources, this depends on a wide array of factors within your business.

What I mean is that don't see it anywhere as a hidden cost in the true sense of that word as you know it takes resources and time, but more so about wrong priorities and business value.

As for the other points, they are so easy to solve:

>>Lost conversions

Trade off you need to make beforehand and need to be aware of with the end goal in mind which is to improve your main conversion metrics.

>>Performance

Most decent conversion platforms use fallbacks and also your way of setting up experiments makes a big difference.

>>Confusion

Letting other departments know what you're doing and what's running, might be harder in big corps though.

>>Speed

This goes for everything in a company I think (new feature/product releases, quick fix vs. proper solution etc.). I personally don't mind spending x amount of time to improve y % in conversion rate, so comes down to setting good KPIs and testing what really is important to your business.

>>Focus

Can be done with a proper conversion team that has all these roles covered (bit hard to set up but possible) or making the conversion department a client within the company. Also the teams mentioned work for other departments as well and improving conversion is a win across the board not just for the conversion team.

Agree completely. :)

Disagree. The main reason we abandoned a/b testing is it's a major resource drain. Getting better at a/b testing would be another resource drain. Time is finite & how you use it determines where you end up.

The point of the article isn't to say a/b testing doesn't work or shoudn't be done. It's simply to say many companies are mis-using their resources by following the standard thinking on a/b testing.

We could spend another month figuring out how to get better at a/b testing or we could spend our time on another activity that produces a higher return on time spent. It turns out that's what we did and the return has been much better.

The post wasn't about the "futility" of a/b testing. It's about the costs of a/b testing that are rarely considered. There's plenty of "pro" a/b testing advice in the world that leads startups to mistakenly waste resources on it.

A solid a/b testing process doesn't just happen without time & effort. You need someone smart analyzing user behavior and forming hypotheses as well as a talented design team to develop concepts that test them.

We simply acknowledged the costs of doing a/b testing well and decided our resources are better spent elsewhere.

At least Google Reader still works :-)