Repository navigation
fix(gh-390): show each dbt layer's last build in data status - #397
db-tycoon-stephen wants to merge 6 commits into
Conversation
data status looked up each layer's last build by bare model name, but capture_dbt has always written dbt_nodes.node_name as the dbt unique_id (model.<project>.<name>). Nothing ever matched, so every staging, intermediate and marts panel said "last build never" even when data history listed the builds. dbt_runs.success being NULL (gh-341) is not involved: the query never reads that column. Matching on unique_id exposed a second error on the same line. capture_dbt writes a UTC-aware datetime into the naive started_at TIMESTAMP column, which DuckDB stores as local wall-clock time, and _freshness_label then treated that value as UTC. Casting back to TIMESTAMPTZ in the reading session recovers the real instant, so the age no longer drifts by the local UTC offset. Existing ledgers read correctly without a migration. Closes #390 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR size checkmode: Files changed: 3 counted / 8 allowed
Result: ✅ within limits. |
PR title checkmode:
Result: ✅ title matches the convention. |
Resolve the CHANGELOG.md conflict: this change's entry joins v0.2.3's [Unreleased] section, and its link references join the sorted list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@db-tycoon-stephen Thanks for the fix. While reviewing this area, I had a follow-up thought from both a user experience and data reliability perspective. Could we consider extending For example, if the latest build fails but the previous successful build was two hours ago, the status could make both facts visible. From a business perspective, this could help users identify potentially stale data without having to inspect run history separately. For analytics and attribution workflows, where decisions depend on complete and up-to-date data, knowing that a recent build failed helps users assess whether the available output can still be trusted. Would this be a useful enhancement to consider for the backlog, perhaps as a separate issue from this PR? |
|
Agreed that showing the latest attempt alongside the last success belongs in I opened #433 for the enhancement, blocked on those two. Could you add anything to it that you would want from the status line, such as whether the failed attempt's error summary should show inline? |
The bullet landed inside the released 0.2.3 section when the branch was merged forward; move it to a new [Unreleased] block like every other v0.2.4 pull request. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…_dbt The test for ledgers with a NULL dbt_runs.success leaned on capture_dbt leaving the column NULL, which gh-341 fixes. On a tree with both fixes the assertion read (1, 1) and the test failed for a reason unrelated to data status. Nulling the column after capture keeps the test's point, that the layer query never reads success, independent of how capture_dbt fills it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`data layers` always said "last build never" after a green build: it matched bare model names against `dbt_nodes.node_name`, which holds the dbt unique id (the gh-390 shape that PR #397 fixes in `data status`). The lookup moves into one shared helper, `tycoon.observability.dbt_builds.latest_successful_build`, that takes unique ids and casts `started_at` back to the real instant, so #397 can adopt it in `status.py` rather than keeping a second copy. The never-called duplicate query and the unmounted Typer app are deleted, and the module uses `load_config()` like the other commands. `data health` is dropped: it was `layers` with a Fivetran table and no health counts behind it. It returns once a freshness capture exists. `data sources explore <name>` fixes four faults that kept its output empty: sample rows read `cursor.description` after the connection had closed; the sample came from `_dlt_loads` (alphabetically first) rather than the user's table; health summed `dlt_runs.rows_loaded`, a column that lives on `dlt_rows_by_table`; and the metadata DB was read from the working directory instead of the project root. The unused `raw_db` parameter is gone with it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Mercy2codesgit
left a comment
There was a problem hiding this comment.
Thank you for the work on this PR.
I reviewed the implementation and its regression coverage. The changes correctly address the model identification issue by matching dbt nodes using their unique_id rather than their bare model names. The timestamp handling also addresses the timezone discrepancy when calculating build freshness.
I particularly appreciate the tests covering:
- Successful builds across staging, intermediate, and marts layers.
- Layers with no recorded builds.
- Compatibility with existing ledger records where
dbt_runs.successis NULL. - Consistent build-age reporting across different timezones.
I ran the new regression tests and the related existing test suites locally. All 55 tests passed.
The implementation looks good to me, and I haven't identified any blocking code issues.
One outstanding item: The CHANGELOG.md merge conflict should be resolved before merging.
Overall, good work on this fix. Thanks!
What this does
tycoon data statussaid "last build never" for every staging, intermediate and marts panel, even whentycoon data historylisted the builds. Status now matches nodes by their dbtunique_idand reads build times in the right time zone.Mental model
dbt_runs/dbt_nodesrows written bycapture_dbt.stg_orders), butdbt_nodes.node_namehas always held theunique_id(model.<project>.<name>). It now passes theunique_id. Fixing that exposed a second problem:started_atis stored as local wall-clock time in a naiveTIMESTAMP, and the freshness label read it as UTC. The query now casts toTIMESTAMPTZin the same DuckDB session time zone, which recovers the real instant.data history. Before, it was always "never". With only the name fix, a build 2h old would have shown as "6h ago" in New York.dbt_runs.successbeing NULL (#341) is not the cause; this query never reads it. #341 stays open fordata history.Linked issue
Closes #390
Type of change
Checklist
pre-commit run --all-filespassescsv-importpipeline still runs end to end (this is the CI gate — if it breaks, the release is blocked)CHANGELOG.mdif this is user-facingNotes for reviewers
unique_ids, their NULLsuccessis never read, and their local-timestarted_atis what the cast expects. A test captures a real dbt-shapedrun_results.jsonto cover this.data status, the age is off by the difference. Storing UTC at capture time would fix that, but it's a writer change and is left out here.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.