Let’s say you’re a startup somewhere around Series A or B stage. Maybe you have a data person on staff, maybe you have a team member who’s stretching and helping out for now, or maybe you’re looking for your first data hire.
A lot of vendors want your money. But time and money are important, and you don’t want to burn either.
We’ll walk through how to decide what you need and how to shop for tools.
Who is this not for
But first, let’s clarify who this is not for.
- Ecommerce companies. You’re probably better off using TripleWhale or something more directly suited to your industry.
- Companies in spaces with serious regulations like healthcare or government. You may have limitations for compliance, and so this may still be helpful but not enough to make an informed decision.
Set your goals
Especially in the AI age, you want to be intentional with your vision for how data and analytics fit into your company.
Gather the right stakeholders in a room and ask:
- How much do we want to spend on tooling?
- How much time investment do we want to put in to see results?
- How much risk tolerance do we have for new tools and workflows vs. conventional approaches?
How are you going to implement this?
Your choices may naturally vary based on the makeup of your current team.
Ask:
- Do you have any data team members on staff? If so, how technical are they?
- Will someone in product, engineering, or another team take part of their time to set tooling up?
- Or do you expect to work with external consultants?
What you need
Consensus wisdom would say there are four “musts” in tooling:
- Data warehouse to store your data and run SQL queries
- ETL (aka data movement or data pipeline) tool to pull data into your warehouse
- Data transformation tool to get the data into the shape you need for analytics
- Data visualization (aka BI) tool to run dashboards and self-service queries
Will any of these change from must-have in the next year due to AI? Maybe! But we’ll talk more about that as we go along.
Data warehouse
What it is
You don’t want to run analytics queries on your production database and risk outages. Many teams will start with a replica database (so if you use Postgres, you can just spin up another Postgres instance). If your data is small enough and you don’t have someone full-time in data, I’d stick with this.
If you want to make a bigger investment in data, or your data is becoming big enough that queries take too long to run, you should look into a data warehouse for analytics. The most popular ones include Snowflake, Google BigQuery, Databricks, and Amazon Redshift. These are designed specifically to handle intense analytical queries.
How to shop
A big factor is which cloud your company uses. If you’re on AWS or Azure, consider Snowflake. If you’re on GCP (or are a big Google Analytics customer), consider Google BigQuery.
If your use cases lean more to data science and machine learning than analytics, Databricks may be a better option. I haven’t spent much time with it but I hear it is geared towards larger orgs.
If you’re on AWS and either 1) don’t like Snowflake or 2) planning to have your warehouse running around the clock, you could consider Amazon Redshift. It’s not my favorite — it requires more admin work — but generally performs about as well if you manage it well.
Let’s briefly touch on pricing models.
- Snowflake
- Snowflake separates pricing for storage (the data you have) and compute (running queries on it). This makes for more predictable costs, though not necessarily less expensive. In fact, a cottage industry has emerged for firms like select.dev to cut Snowflake costs.
- Snowflake’s compute pricing is based on the size of the compute cluster (using tee shirt sizes from X-Small to 6X-Large). You pay for the time that the compute cluster is in use.
- Google BigQuery
- BigQuery also charges separately for storage and compute. However, compared to Snowflake, it offers two compute pricing models — on-demand and capacity.
- On-demand is what is typically associated with BigQuery. You’re charged based on the amount of data scanned by queries. This is not strictly better or worse for customers, but can be less predictable. It also potentially penalizes you for quickly writing an optimized query.
- Capacity is closer to the Snowflake model. You pay for the time you are using it. However, there is just one size of compute.
- Databricks
- Somewhat similar to Snowflake, you’re charged based on usage. However, storage is billed directly through your cloud provider instead of Databricks.
- Amazon Redshift
- With Redshift, you pay for a cluster, so your storage and compute are combined.
ETL (aka data movement or data pipeline) tool
What it is
Let’s say you go with Snowflake as your data warehouse. Now you need to figure out how to get data into Snowflake, whether that’s from your database or SaaS tools like Salesforce, Stripe, and Zendesk.
You don’t need a data engineer to do this anymore. ETL (extract, transform, and load) tools such as Fivetran, Airbyte, and Estuary offer what you can think of as “data pipelines as a service.”
Countless companies use Salesforce, so you don’t need to write a data pipeline from scratch. You can set up a data pipeline into your data warehouse in minutes with these tools, mitigating the risk of hiring a data engineer. You may decide later on to bring in a data engineering team for more complex, bespoke use cases.
One of the nice advantages to buying is it’s nice to have someone invested in solving edge and corner cases, which is harder to justify if you have an in-house data engineer juggling several pipelines.
How to shop
I’ve been mainly using Fivetran the past six years, but there are a few good options on the market. Ask your team:
- How much data do you have? Most startups are not at petabytes of data, so the cost of an ETL tool is going to be much less than a data engineer on staff.
- What data sources will you need to pull from? Check out the Fivetran and Airbyte docs to see if all your sources are covered by existing conenctors.
- Want to try to skip this step and have an engineer ship with AI? dlt or Meltano + Dagster.
- How real-time do you need the data ETL to update? Most internal use cases are fine with updating once a day or every few hours. If you really need near-real-time, you may want something else moving your data.
Data transformation tool
What it is
You have your data warehouse and you have data pipelines into it.
You could get straight to writing queries and creating dashboards. But many teams find that the data coming in is not quite ready for that. You’ll be repeating a lot of the same filters, renaming, etc.
Data transformation tools like dbt and SQLMesh come in here. What they do is give you a version-controlled repository of SQL scripts that you can run on a schedule. So if you have a table of leads and want to score them, you can write the scoring logic in dbt and set it to rerun and update at whatever interval you’d like. You can also add tests to catch any bad data.
LLMs are pretty good at working with dbt, so you can get moving quickly whether you’re starting from scratch or migrating from an old stack.
Both dbt and SQLMesh are free and open-source, so your Head of Finance can breathe a sigh of relief. You will need to figure out hosting, though. dbt Cloud is free for the first user and $100 per user on their Starter plan. That’s where I’d start.
How to shop
I’d recommend starting with dbt, but this can be the last one you set up. Focus on getting data into your warehouse first.
Data visualization or BI tool
What it is
You have a data warehouse, data pipelines, and now reporting-ready datasets. Now you’re ready to query and visualize.
Business intelligence (BI) tools enable point-and-click data visualization and dashboard building. Tableau and Looker were two of the top choices for many years, though I feel like they’ve atrophied after being acquired by Salesforce and Google, respectively.
Lightdash, Omni, and Sigma are newer and well-regarded options. If you’re trying to start lean, you could consider Hex, a notebooking tool that is a dashboarding tool second but I’ve found does good enough for the price.
These latter tools seem to be doing a much better job of thinking AI-natively, rather than just adding AI features. The era of point-and-click dashboards is over, and it is much more efficient to define dashboards in code so that AI agents can build and iterate faster than we can.
How to shop
This is the place where cost will flex the most based on your preferences. I recommend thinking about who is going to be using the tools, how many seats do you need, and how technical/self-sufficient are your users?
What you maybe need
Agentic coding tool
Claude Code, Codex, Cursor, and more. They’re having a moment. And a lot of the hype is justified.
As AI models improve, they are better and better equipped to handle analytics problems. I’ve found that with good problem definition and prompting, Claude Code can often successfully complete tasks on the first try.
Teams using these tools effectively will be able to move faster.
Related, some tools I’m keeping an eye on that are more specifically geared towards data: paradime, nao, and Oxygen.
Reverse ETL tool
It sounds silly when you say it. But just like you have ETL tools moving data into your warehouse, there are tools to help you move data out. There’s been some consolidation — Fivetran (ETL tool) acquired Census (reverse ETL tool), which helps. Hightouch is also very popular, and has leaned into the agentic marketing angle.
Product analytics tool
If you want detailed product and web analytics, you likely want to instrument event tracking with a tool like Mixpanel or Amplitude. You can then analyze the data directly in those tools, or move it into your warehouse to sit alongside other data.
Experimentation tool
If you’re running experiments and A/B tests, you could consider tools like Eppo.
Orchestration tool
If you want to run data pipelines, data transformations, or AI/ML workflows on a schedule, you’ll want an orchestration tool. At a high level, these tools define those jobs as code in Python and help you run them. If your existing team is very technical, you might start here early on, but otherwise most teams grow into needing these. Dagster, Airflow, and Prefect are some of the most popular.
Managed services
Don’t want to shop for several tools? You can consider managed services that will spin up some of the technologies mentioned above, such as Definite.
What you probably don’t need (right now)
Data lake or lakehouse architecture
These are buzzwords and for good reason, but not something you need when you’re just starting out.
Data catalog
These are collections of metadata about your data assets. Maybe you get one of these later, but not needed when you’re just starting.
Observability tools
You want to stay on top of when there are data quality issues, before your stakeholders get upset or lose trust in the data. Observability tools like Metaplane help with this. I’d probably just start with dbt tests (AI can help you write them) and revisit this later.
Conclusion
Hopefully this rundown helped you understand what tools you should be looking at and how everything fits together.
Have more questions and want to chat further? Drop me a line at jacob@riverboat.ai.