In spring 2026, Riverboat Consulting partnered with Disco, the AI-powered commerce media platform on an agentic analytics solution in Hex.
Disco helps retailers monetize every transactional touchpoint from checkout through delivery, and helps advertisers reach high-LTV customers at moments of peak intent. It started in the Shopify ecosystem but has since expanded to other platforms, resulting in fast growth in both data collection and appetite for insights.
Background
The Disco data team was midway through migrating reporting data sources from Engineering-owned tables to data-owned tables. Doing so streamlined away numerous joins, filters, unpacking JSON, and other logic that should not take place at the dashboard level. Their stack included Snowflake for warehousing, dbt for transformation, and Hex for analysis and visualization.
Disco is an AI-forward company and wanted to shift workflows to be more agentic. Some stakeholders were already using Hex AI agents, but the data team knew that there was a lot of work to do to optimize the agents’ context. Previously, agents would frequently use the wrong tables, filters, and metric definitions in answering questions.
Our top goals were to design Disco’s agent context architecture, implement the first phases of our design, and facilitate knowledge transfer to the data team for continued scaling.
What We Built
Context Files
The Hex AI agent harness draws from several sources. However, the two types of context files that would lead to the best performance improvement vs. where we started at were the workspace context file and individual domain guide files.
The workspace context file is read in every chat (similar to an AGENTS.md or CLAUDE.md file) and is a set of instructions and information that should be relevant to as many chats as possible. Domain guides then allow agents to go deeper on a subject with the level of rigor a human analyst would want. We chose to start with four domain guides representing frequently-analyzed business topics.
We knew that these files would naturally evolve over time, either because we tuned what would lead to optimal agent performance or because the business needs changed. Letting go of the idea that the first versions needed to be perfect allowed us to move and iterate quickly.
Agent Skills
Although we started with four domain guides, we knew this number would increase over time. We wanted to prioritize an implementation that would scale and be manageable for the team after our engagement concluded.
Since we had a desired structure and template for our domain guides, we created a Claude skill hex-guide where team members could activate an interactive workflow to collect the necessary information to synthesize into a Hex domain guide, and have Claude apply the template to ensure consistency.
Disco elected a monorepo design where Hex context files (workspace context and domain guides) and dbt project were in the same Git repository, so our shared agent skills were also stored there so that all team members could access.
These skills saved hours in original development but also makes maintenance and broad changes much easier in the future. Peter Li, Head of Data & ML at Disco, said, “The agentic development workflows are crucial in a fast-moving startup and a lean data team.”
Validation and Testing Framework
Just as we would run data validation queries if we changed a dbt model or dashboard, we needed to confirm that the context we provided Hex agents actually helped them answer user questions in the way we wanted.
Hex currently has exciting observability tools in its Context Studio, which helps admins understand how agents performed in real chats with stakeholders. We used this extensively but also wanted to implement evals, which are tests we give AI agents to measure performance. These would give us much more real-time feedback on how well our system is working without waiting for stakeholders to test themselves.
As of this writing, Hex’s eval features are still in development, so we built our own framework. We said as a north star, we’d be able to run all our evals everyday and monitor scoring trends. However, it was simpler to start with running relevant evals when updating domain guides.
When creating a new domain guide, we’d create 4-5 eval questions that represented common question types for that topic, including handling risks such as vague prompts or known pitfalls. Each question would also list the criteria of what we wanted to see or not see. When we were ready to test, we created new Hex Threads using the proposed domain guide and one eval question at a time. We’d then copy/paste the output into a Claude Code session with our agent skill activated. The agent would apply our rubric and score the question. Once all questions were scored, Claude would update our pull request with a cleanly formatted table of eval results for future reference.
Semantic Model and dbt Metadata
Semantic layers are crucial for the success of AI agents. They show how the data fits together and should be queried. Had Disco not already implemented theirs before beginning to use AI agents, we may have started here before the context files. However, we were able to expand coverage to more tables and metrics, reducing edge cases and the instances where the agent had to think harder or guess to produce an answer.
Additionally, we leveraged Hex’s integration with dbt Cloud to sync dbt metadata to Hex. This eliminated the need to store table and column descriptions in multiple places. As with most fledgling dbt projects, column descriptions and tests were sparse, so we had Claude review previous pull requests and run exploratory data analysis queries to backfill.
Back-of-House Snowflake Configurations
We also wanted to make it easier for Hex AI agents to find the data in Snowflake. We made two quick win changes to enable this. First, we split out dbt outputs into separate schemas to isolate staging and intermediate tables from mart tables. Then, we applied Snowflake schema filtering in Hex so that agents would only see mart and reporting tables. This helps them find what they’re looking for faster and reduce the chance they end up querying the wrong dataset.
Knowledge Base
We documented our project plan and learnings in Notion as the engagement went on to ensure all team members were up to date. Additionally, we recorded about a dozen Loom videos screensharing our exact development workflows, where to find certain settings and configurations in Hex, and anything else the data team would need to have continued success after our engagement concluded.
Outcomes
Stakeholders were very excited about our updates in the data stakeholder Slack channel and reported performance improvements almost immediately after the workspace context addition. The usage data corroborated the vibes — average chat volume increased from 71 to 158 chats per week in the six weeks after publishing the workspace context guide.
Qualitative reviews of chats were promising. Agents were consistently invoking the domain guide context files and following SOPs as intended. While we still saw instances where agents got stuck or made incorrect assumptions, we could usually categorize it as an area where 1) we needed to provide even deeper context to the agent 2) the user could have prompted better 3) the question was so specific we’d prefer the agent get stuck and loop in a member of the data team, and then take action accordingly. This was a great improvement from just weeks before and provides significant momentum for the team.
Additionally, our efforts sparked conversations about the long-term vision for linking analysis to actual action in the business, instead of potentially siloing that information. The team is actively prototyping new in-house tooling on that front.
Peter said, “The improvements in our Hex instance from Jacob’s efforts were remarkable. From both the design and implementation phases it was clear we were in good hands. We’re excited about what exciting next steps this enables for agentic analytics and workflows at Disco.”
We made all this progress in just six weeks. In the AI age, it’s crucial to move fast. If your company is looking for energizing force for your data practice, whether its getting started or already in motion, please get in touch.