feat(data): add error handling for ODS layer and improve partial data support
Add materialized views to capture parsing errors from browser, location, device, and geo raw staging tables and route them to dedicated error tables in the ODS layer. Refactor DDS refresh logic to handle partial data arrivals where device and geo events may arrive independently by using a unified click_id source with LEFT JOINs. Add TRUNCATE command to prevent duplicate data accumulation in DQ summary table.
This commit is contained in:
1 parent
6bbb26b9b3
commit
e44b988d76
3 files changed
+128
-6
No files matched your search
@@ -40,6 +40,8 @@ PARTITION BY toYYYYMM(check_date)
|
||||
ORDER BY (check_date, layer, table_name, check_name);
|
||||
|
||||
-- Truncate and refill DQ summary
|
||||
TRUNCATE TABLE dm.dq_summary;
|
||||
|
||||
INSERT INTO dm.dq_summary
|
||||
SELECT
|
||||
today() AS check_date,
|
||||
|
||||
Reference in new issue
Block a user