Measuring Customer Satisfaction in a World of Pseudonymity

Disclaimer: this entire post is about a question to which I don’t know the answer.

The Big Question

As an incubator in the crypto space, we have been mulling over the following question:

How can we effectively measure customer satisfaction while retaining privacy and pseudonymity?

As iteration maximalists focused on quickly identifying whether there may be product market fit, this is a paramount question to at least attempt to answer.

And it’s a question we still don’t know the answer to. Below are a few thoughts we’ve touched on internally, along with a couple companies I’ve come across that are working on helping to solve this question for early-stage DeFi projects.

What is Fair Game?

First of all, what is the right amount of analytics for a new DeFi project? This is a touchy subject with projects like Metamask collecting more data than necessary, and yet it’s an essential conversation to have as an industry.

It’s important for startups to be able to learn from user interaction, and when done right it is a win-win for the responsive project and its increasingly happy users.

But sadly there is less privacy in DeFi than most DeFi users think, and running comprehensive analytics may reveal more info about users than they would be comfortable with.

Is it fair game, for example, to run on- and off-chain analysis that bundles wallets that are likely to be controlled by the same user, and build a profile based on any connected data? All of the data is on-chain, or public, and yet this would likely be argued as ‘doxxing’ that user.

In my opinion, it’s somewhere in the middle: between zero user tracking and collecting anonymized interactions.

When will the lines be clearer?

• When better privacy is baked into settlement layers (for example paying for ENS via Aztec)
• When users have more control, and can opt in and out of sharing data granularly and securely (i.e. with zero-knowledge proofs)

Some Early Tools

Abstraction Layers

Let’s say a DeFi project has decided to attempt to measure user satisfaction. Where do the founders begin? There are some crude tools they can start with.

First of all, if the data is on-chain, anyone can build custom analytics for their dapp. The leader of course is still Dune, with dashboards for most leading DeFi projects. The Dune API (still in private beta) will be a big moment for a lot of teams.

We’ve seen some great attempts at cohort analyses on Dune, such as on this dashboard by @kunal:

Others like Footprint attempt to abstract the SQL with filtering within the UI. In addition, there are a multitude of recently-funded blockchain ETL projects focused on building abstraction layers for the multi-chain cryptoverse and I expect many will be launching throughout 2023.

Coherent is another example, providing an API for retrieving decoded and human-readable blockchain events, making it easy for developers and even tech-savvy marketers to quickly put pieces together and generate insights into what’s happening on-chain.

Ceramic Network

One of the more ambitious projects working on decentralizing the data backend, Ceramic is a network of nodes that validates append-only data “streams” stored on IPFS and IPLD, with authentication controlled through support of various DID methods.

Content is stored on IPFS (using IPLD), and streams can reference other streams within the Ceramic Network global namespace.

The Cerscan explorer is worth a quick look and highlights some of the early apps leveraging Ceramic.

Cookie3 - User (Wallet) Profiling API

Cookie3 aggregates wallet actions and holdings across multiple chains in order to build user personas for any wallet. Projects can tap into their API to customize the experience for each user when connected, or to learn more about their user base.

The product is still in beta.

Approaches We’re Taking (or Have at Least Considered)

Limiting Incentives

Bootstrapping user growth with incentives makes it challenging to know if there is a product market fit. In an ideal world, incentives are introduced once a fit has been found and it is time to pour gasoline on the fire (do not try this at home!).

This can be difficult in DeFi when attempting to launch an autonomous system with governance in place, but in 99% of cases this is not the case, with projects retaining governance controls in order to maintain flexibility as they develop. Tokens or other incentive mechanisms are often introduced too early when product market fit is still an open question.

And incentives do not just include token mining. Reputation farming, expectations of a potential future airdrop, and the beta surge (people just trying the product out to keep up with what’s being developed) are other examples of potential feedback skewing inputs.

For one Tinkering Society prototype, our approach included cutting down the scope of a product to “single-player mode” in an attempt to eliminate social incentives (reputation points). We believe the multi-player mode will ultimately be a significant portion of the value proposition, but want to ensure there is a strong value proposition for the primary product use case before supplementing it with secondary value props.

Maintaining Flexibility

Staying open to pivoting is challenging in DeFi. Testing and auditing extend the dev cycle and increase the investment in code. Limiting perfectionism early on is ideal for most projects in the blockchain space.

Methods to postpone steps in the process include a testnet iteration phase, bounties during testnet phase, capping open interest or total deposits early on, or a degen-only launch with disclaimers. Note that this is difficult if you are launching a truly autonomous protocol, or one that intends to be protocol-to-protocol (i.e. a primitive or building block).

Direct Customer Feedback

It’s hard to get feedback in a market in which the idea of pseudonymity and limited sharing is ingrained.

We’ve seen many projects opt to just directly pay people in select or targeted social circles (i.e. Telegram groups) for live feedback sessions.

Gitcoin has done a great job with this, giving out merch for live Zoom sessions where interviewees speak their thoughts out loud while exploring the website.

Watch out for selection bias!

Don’t Lock Customers In

We had a debate about whether to provide a yearly subscription for an upcoming product. Getting early users to pay for a year upfront is a strong signal that they think your product will be valuable to them.

But it also locks them in. If they don’t end up using the product much or stop liking it due to the direction of development, their vote of confidence is now lost. In our case, we made the decision to only have a monthly subscription option so that users can tell us when they no longer feel the product is a must-have (by unsubscribing).

Give Users a Good Reason to Leave

Another extreme loyalty test (that Gaurav has been advocating for!) is to give customers a reason to stop using your product. Trying to convince them to leave will tell you if the product is a must-have or just a product they like but don’t love.

Focus on the Quantitative: Establish That You Have a Product Market Fit

Okay, so we think we’ve minimized incentives that might skew our customer satisfaction input, and we’ve reduced our product’s features to test the main value proposition in isolation.

Next we want to prove we’ve found a product market fit so that we can start adding back in all the great features or incentive mechanisms we have planned.

So we need a few core metrics that test our hypotheses. This is likely unique for every product, but could be retention of paid users (i.e. cohort analysis), minutes spent using the product, successful on-chain actions, daily average users, and so on.

Some formalized method’s we have discussed are the Sean Ellis Test for product market fit, Rahul Vohra’s four step guide to establishing a fit, and Brian Belfour’s focus on flattening the retention curve and then shifting it up.

Can we retrieve the data necessary to employ the above web2 satisfaction and retention tests? I think it’s possible for most apps and dapps — the data is typically there. It just may require more thoughtful measurements or specialized data people. The emergence of companies like Block119 validate this thesis, working with founders to uncover insights. I wonder if we see forensic analytics firms also pivot into the more productive user insights space?

Conclusion

The blockchain space, and DeFi in particular, can make it challenging to gather user feedback and measure product market fit.

This means it’s essential to maintain the flexibility to pivot while attempting to isolate variables and strategically seek user feedback.

We are learning as we go, and would love to hear what approaches you all are taking to get closer to answering the big question of “do we have a product market fit”?