Skip to content

fix(gh-390): show each dbt layer's last build in data status - #397

Open
db-tycoon-stephen wants to merge 6 commits into
v0.2.4from
fix/gh-390-status-last-build
Open

db-tycoon-stephen wants to merge 6 commits into
v0.2.4from
fix/gh-390-status-last-build

Conversation

@db-tycoon-stephen

@db-tycoon-stephen db-tycoon-stephen commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

What this does

tycoon data status said "last build never" for every staging, intermediate and marts panel, even when tycoon data history listed the builds. Status now matches nodes by their dbt unique_id and reads build times in the right time zone.

Mental model

  • Input: the manifest's models per layer, and the dbt_runs / dbt_nodes rows written by capture_dbt.
  • Change: status looked nodes up by bare model name (stg_orders), but dbt_nodes.node_name has always held the unique_id (model.<project>.<name>). It now passes the unique_id. Fixing that exposed a second problem: started_at is stored as local wall-clock time in a naive TIMESTAMP, and the freshness label read it as UTC. The query now casts to TIMESTAMPTZ in the same DuckDB session time zone, which recovers the real instant.
  • Outcome: each layer shows the age of its most recent successful build ("2h ago"), matching data history. Before, it was always "never". With only the name fix, a build 2h old would have shown as "6h ago" in New York.

dbt_runs.success being NULL (#341) is not the cause; this query never reads it. #341 stays open for data history.

Linked issue

Closes #390

Type of change

  • Bug fix
  • New feature
  • Docs
  • Refactor / internal
  • Other:

Checklist

  • pre-commit run --all-files passes
  • Tests pass locally, and the offline csv-import pipeline still runs end to end (this is the CI gate — if it breaks, the release is blocked)
  • Added or updated tests where it made sense
  • Updated CHANGELOG.md if this is user-facing
  • Updated docs / README if behavior or commands changed

Notes for reviewers

  • Upgrade safety: ledgers written by 0.2.2 and earlier need no migration. They already store unique_ids, their NULL success is never read, and their local-time started_at is what the cast expects. A test captures a real dbt-shaped run_results.json to cover this.
  • The time zone test runs in a subprocess per zone (UTC, America/New_York, Asia/Tokyo), because DuckDB fixes its session time zone when the process starts.
  • Known limit: if the machine's time zone changes between a build and data status, the age is off by the difference. Storing UTC at capture time would fix that, but it's a writer change and is left out here.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

data status looked up each layer's last build by bare model name, but
capture_dbt has always written dbt_nodes.node_name as the dbt unique_id
(model.<project>.<name>). Nothing ever matched, so every staging,
intermediate and marts panel said "last build never" even when data
history listed the builds. dbt_runs.success being NULL (gh-341) is not
involved: the query never reads that column.

Matching on unique_id exposed a second error on the same line.
capture_dbt writes a UTC-aware datetime into the naive started_at
TIMESTAMP column, which DuckDB stores as local wall-clock time, and
_freshness_label then treated that value as UTC. Casting back to
TIMESTAMPTZ in the reading session recovers the real instant, so the age
no longer drifts by the local UTC offset. Existing ledgers read
correctly without a migration.

Closes #390

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

PR size check

mode: warn | file limit: 8

Files changed: 3 counted / 8 allowed

  • CHANGELOG.md
  • src/tycoon/commands/status.py
  • tests/test_status_last_build.py

Result: ✅ within limits.

@github-actions

Copy link
Copy Markdown

PR title check

mode: warn

fix(gh-390): show each dbt layer's last build in data status

Result: ✅ title matches the convention.

Resolve the CHANGELOG.md conflict: this change's entry joins v0.2.3's [Unreleased] section, and its link references join the sorted list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@db-tycoon-stephen db-tycoon-stephen self-assigned this Oct 13, 2026
@Mercy2codesgit

Copy link
Copy Markdown

@db-tycoon-stephen Thanks for the fix. While reviewing this area, I had a follow-up thought from both a user experience and data reliability perspective.

Could we consider extending tycoon data status to show the outcome of the latest dbt build attempt alongside the last successful build time?

For example, if the latest build fails but the previous successful build was two hours ago, the status could make both facts visible.

From a business perspective, this could help users identify potentially stale data without having to inspect run history separately. For analytics and attribution workflows, where decisions depend on complete and up-to-date data, knowing that a recent build failed helps users assess whether the available output can still be trusted.

Would this be a useful enhancement to consider for the backlog, perhaps as a separate issue from this PR?

@db-tycoon-stephen

Copy link
Copy Markdown
Contributor Author

Agreed that showing the latest attempt alongside the last success belongs in data status. It is out of scope here, and the ledger cannot answer it yet: failed runs never reach dbt_runs (#377) and dbt_runs.success is always NULL (#341). Both are in v0.2.4.

I opened #433 for the enhancement, blocked on those two. Could you add anything to it that you would want from the status line, such as whether the failed attempt's error summary should show inline?

@db-tycoon-stephen db-tycoon-stephen modified the milestones: v0.2.3, v0.2.4 Oct 21, 2026
@db-tycoon-stephen
db-tycoon-stephen changed the base branch from v0.2.3 to v0.2.4 October 27, 2026 10:28
db-tycoon-stephen and others added 3 commits October 30, 2026 15:52
The bullet landed inside the released 0.2.3 section when the branch
was merged forward; move it to a new [Unreleased] block like every
other v0.2.4 pull request.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…_dbt

The test for ledgers with a NULL dbt_runs.success leaned on capture_dbt
leaving the column NULL, which gh-341 fixes. On a tree with both fixes
the assertion read (1, 1) and the test failed for a reason unrelated to
data status. Nulling the column after capture keeps the test's point,
that the layer query never reads success, independent of how capture_dbt
fills it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
db-tycoon-stephen added a commit that referenced this pull request Nov 2, 2026
`data layers` always said "last build never" after a green build: it
matched bare model names against `dbt_nodes.node_name`, which holds the
dbt unique id (the gh-390 shape that PR #397 fixes in `data status`).
The lookup moves into one shared helper,
`tycoon.observability.dbt_builds.latest_successful_build`, that takes
unique ids and casts `started_at` back to the real instant, so #397 can
adopt it in `status.py` rather than keeping a second copy. The
never-called duplicate query and the unmounted Typer app are deleted,
and the module uses `load_config()` like the other commands.

`data health` is dropped: it was `layers` with a Fivetran table and no
health counts behind it. It returns once a freshness capture exists.

`data sources explore <name>` fixes four faults that kept its output
empty: sample rows read `cursor.description` after the connection had
closed; the sample came from `_dlt_loads` (alphabetically first) rather
than the user's table; health summed `dlt_runs.rows_loaded`, a column
that lives on `dlt_rows_by_table`; and the metadata DB was read from
the working directory instead of the project root. The unused `raw_db`
parameter is gone with it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@db-tycoon-stephen
db-tycoon-stephen requested review from JesuFemi-O and Mercy2codesgit and removed request for JesuFemi-O November 3, 2026 01:17

@Mercy2codesgit Mercy2codesgit left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the work on this PR.

I reviewed the implementation and its regression coverage. The changes correctly address the model identification issue by matching dbt nodes using their unique_id rather than their bare model names. The timestamp handling also addresses the timezone discrepancy when calculating build freshness.

I particularly appreciate the tests covering:

  • Successful builds across staging, intermediate, and marts layers.
  • Layers with no recorded builds.
  • Compatibility with existing ledger records where dbt_runs.success is NULL.
  • Consistent build-age reporting across different timezones.

I ran the new regression tests and the related existing test suites locally. All 55 tests passed.

The implementation looks good to me, and I haven't identified any blocking code issues.

One outstanding item: The CHANGELOG.md merge conflict should be resolved before merging.

Overall, good work on this fix. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

2 participants