Battle Scars

Open a tap and water comes out. The first three posts in this series were about building the infrastructure behind that tap – a YAML-driven ingestion platform for a Lakehouse Bronze layer: a fixed engine plus N entity configuration files, metadata that governs metadata, a scheduler that turns freshness into a contract. The design is sound. The tests pass. The inspector signs off.

Then winter comes.

The pipes freeze. A gasket cracks. Someone upstream closes a valve nobody documented. A Spark runtime upgrade quietly breaks dates that have been loading fine for two years. A load finishes PartialSuccess and now you have half a day of data with no clean way to know which half. Post 3 ended with a warning: a process crashes at 2:00 am without releasing its lock and silently blocks an entire domain until someone notices at 9:00 am. This post starts there – and adds three more failure modes that arrived the same way: without warning, in production, on a Thursday.

None of this was on the diagram. All of it showed up in production.

This post is about four scars from real Lakehouse implementations – the ones that show up regardless of source, scale, or team. Not because scars are clever – they aren’t. But because the test of a metadata-driven platform is not whether it avoids them. It’s whether each one lands back in the contract – a new YAML field, a new partition strategy, an audit column, a runbook – instead of a one-off patch buried in a notebook.

The Japanese have a name for that distinction: kintsugi – repairing broken pottery with gold lacquer instead of invisible adhesive, so the crack stays visible and becomes part of the object’s history. Every scar in this post is either gold (a declared YAML field, a documented partition strategy, a runbook in Git) or invisible adhesive (a patch in a notebook that holds until it doesn’t, and nobody remembers it’s there).

An architecture without scars isn’t good design. It’s low usage. Boring is earned, not designed – and this post is about the earning.

A brief detour: the control schema

Three control tables appear in the scars below – operational metadata generated by the platform itself, never hand-edited. Post 6 covers the full catalog; for the four scars, these three are the cast:

• control.NotebookLog – one row per notebook execution (one notebook per LoadKey = {Domain}_{Source}, ingesting all entities declared in that YAML). Columns: RunID, LoadKey, Status (Success

  Failed

  PartialSuccess), RowCount, StartTime, EndTime, ErrorMessage. The notebook’s output payload – persisted alongside the run – carries entity-level detail: total_entities, per-status counts (updated / no_changes / errors), and per-entity error breakdowns. Written by every ingestion notebook on completion.

• control.SchedulerLocks – one row per load. Lock state, owner RunID, acquire/release timestamps, release reason. Detailed schema in Scar #3.
• control.VolumeCheckQuarantine – anomalies detected by the volume-check post-ingestion task. Used implicitly in Scar #2’s PartialSuccess diagnosis.

These three live in a control schema separate from the Bronze data schemas (Sales_Salesforce, etc.). Why: governance access lists for control data diverge from those for business data, and a separate schema makes that boundary easy to enforce.

Battle Scar #1 – When Spark versions disagree about dates

The symptom. A load that had been running without incident for months starts failing:

INCONSISTENT_BEHAVIOR_CROSS_VERSION.READ_ANCIENT_DATETIME

The cause. Spark 3.x – the Fabric runtime version – applies the Proleptic Gregorian calendar strictly. For Date columns read from Parquet, spark.sql.parquet.datetimeRebaseModeInRead defaults to EXCEPTION: any value before 1582-10-15 (the Gregorian calendar reform cutoff) triggers READ_ANCIENT_DATETIME on read. Earlier Spark runtimes read these values without complaint. The runtime upgraded; the data didn’t.

The offending values weren’t legacy historical dates – they were corruption. A Date column carried 0008-01-01 for some rows, where the source system intended 2008-01-01. Somewhere upstream a four-digit year had been truncated or mis-parsed into a single digit, written into Parquet, and propagated for months without anyone noticing – because every runtime before this one accepted year-8 dates silently. Spark 3.x doesn’t.

The tempting fix. Patch the notebook. Catch the exception, null out the offending columns, move on. Forty-five minutes of work, problem gone.

The fix we chose. A new direct field in the entity’s YAML – declared at the entity level, applied when the engine reads Parquet files previously staged in storage:

parquet_date_fix:
  enabled: true
  columns: ["ACTDATE", "CREATEDATE"]
  cutoff_date: "1900-01-01"
  action: "NULL"

The workaround is now declared, not hidden. Every YAML that inherits this problem declares it explicitly. A new engineer reading Finance_LegacySystem.yml sees parquet_date_fix: enabled: true and immediately understands this source has a known compatibility quirk. There is no try/except buried in a notebook for someone to stumble over in six months.

The gotcha inside the fix. In YAML, action: NULL without quotes is parsed as Python null by most Python parsers (PyYAML and anything built on it, including Yamale). The engine reads it as None and throws action 'None' not implemented. The correct form is action: "NULL" – a quoted string. One missing pair of quotes, perfectly valid YAML syntax, completely wrong behavior. The contract has its own type rules, and they are not always obvious.

What happens to the nullified values. The engine logs every nullified value to an audit table: RunID, column, original value, row identifier. If someone later asks whether CREATEDATE was really nullified for a specific record, the answer is a query, not a shrug. The cutoff is deliberately more conservative than Spark’s threshold. Spark rejects pre-1582 Date values; we null anything pre-1900. The reason is business: in this domain – enterprise transactional sources – a Date earlier than 1900 is corruption or an “unknown” sentinel, not history. A literal year-8 record is wrong. So is a record claiming the entity was created in 1850. Pulling the cutoff to 1900 turns the threshold from “Spark will fail” into “business has decided these values are unrecoverable” – and auditing them as nullified is more useful than letting them through. The one risk: if cutoff_date is set too high – 2000-01-01 instead of 1900-01-01 – legitimate business dates in the 20th century get nullified. The threshold is a judgment call that needs sign-off from someone who knows what those dates mean.

The lesson. Every compatibility workaround ends up in the contract. The engine doesn’t accumulate special cases for individual sources. The YAML grows a flag that says “this source has this known issue, apply this known treatment, log it this way.” When the upstream system is eventually fixed, flipping enabled: false is a one-line PR and the workaround disappears cleanly.

Visible, reversible, auditable. Three properties that patches in notebooks never have.

One detail that closes the loop with Post 2: adding parquet_date_fix required updating the Yamale schema. The field didn’t exist when the .schema file was written – any entity deploying it would have failed L4 validation before reaching the engine. One PR added parquet_date_fix as an optional block to the schema file; template_version incremented from "1.0" to "1.1". YAML tolerates new sections gracefully – a notebook that doesn’t know about parquet_date_fix simply ignores it. But template_version isn’t about YAML tolerance. It ensures the engine reads the same contract that the schema validates and the deployed files declare. A version bump makes the change visible in the audit trail and gives the engine a hook to apply version-specific logic when needed. Entities that needed the fix declared it; entities that didn’t were untouched. The platform absorbed a new requirement with exactly the friction Post 2 designed for: one schema PR, validation before every deploy.

Runbook CFG-DATE-FIX-01 covers flag activation and impact auditing. Retirement criteria and detection steps in Post 5.

Battle Scar #2 – Partial failures and idempotency by SnapshotDate

The symptom. A batch of several dozen domain-source loads runs across the schedule. Most notebooks complete cleanly. A handful return PartialSuccess – the notebook ran, but one of the entities inside it hit a network blip, a source timeout, or an intermittent driver error. PartialSuccess is a control-table status (control.NotebookLog), distinct from Fabric’s native pipeline statuses (Succeeded | Failed | Cancelled) which would mark the orchestration Succeeded regardless of internal failures.

Why this is worse than total failure. Total failure is clean. You retry, you recover. Partial failure leaves you with a half-loaded day of data. Now you have to know which entities failed, which ones succeeded, whether retrying would double-load the successes or only pick up the failures. The answer depends on whether your pipeline was designed for this question. Most aren’t.

The pattern that saved us: idempotency by SnapshotDate. Every ingestion is partitioned by SnapshotDate. A rerun for the same day replaces that partition – not appends, not merges, not deduplicates. Overwrites. Rerunning the whole batch is safe because the successes get rewritten identically and the failures get retried – provided the source produces the same data on retry. That holds for most batch extractions; it doesn’t hold for sources with rolling windows or stateful APIs, where a retry mid-window may pull different data than the original run. No rollback procedure, no “skip these, include those” orchestration:

df.write.mode("overwrite") \
  .option("partitionOverwriteMode", "dynamic") \
  .partitionBy("SnapshotDate") \
  .saveAsTable(...)

The engine validates partition layout against the YAML before every write; an existing table with a different layout fails with a pointer to runbook CFG-PARTITION-FIX-01.

Diagnosis stays simple. One query against control.NotebookLog tells you which three loads ended with PartialSuccess; the notebook’s output payload carries the per-entity breakdown – counts and error details for each entity inside the notebook. You don’t need to dig through Fabric’s native pipeline logs. Your own control tables were built for exactly this question.

The lesson. Idempotency isn’t a feature you add later. It’s a partition-key decision you make on day one. Get it right, and partial failures become retry problems – cheap. Get it wrong, and they become consistency problems – expensive. The difference between the two is roughly the difference between “rerun the pipeline” and “call a meeting.”

The honest boundaries. Two boundaries are worth naming. First, idempotency by overwrite depends on the source producing the same data on retry – rolling windows, stateful APIs, and sources that re-publish past data with corrections break that assumption. The partition gets reproduced cleanly, but its semantic content changes. Second, the Delta guarantees only hold while the contract is intact: an external process touching parquet files directly, an append mode left in by accident, a VACUUM with retention below the active log range – all of these put the table into states where DELETE WHERE SnapshotDate = '{date}' followed by a rerun is the recovery path. These aren’t edge cases of partition overwrite; they’re consequences of bypassing the contract that makes partition overwrite work.

Runbook OPS-IDEMPOTENCY-01 covers PartialSuccess diagnosis, the safe rerun procedure, and the recovery paths when the Delta contract was bypassed.

Battle Scar #3 – When the guard lock outlives the guard

The symptom. The scheduler refuses to trigger a load. control.SchedulerLocks shows the load is locked. The process that acquired the lock crashed two hours ago. The lock is still there.

Why we have locks at all. Two scheduler runs can’t touch the same load simultaneously. If a load is mid-flight when the scheduler evaluates again, bad things happen: duplicate writes, racing metadata updates, orphaned batches. The lock is a single-writer guarantee. Post 3 introduced this consequence as an inherent property of the scheduler design.

control.SchedulerLocks holds one row per load:

-- control.SchedulerLocks (one row per LoadKey)
LoadKey              STRING     -- {Country}_{Domain}_{Source}
IsLocked              BOOLEAN    -- true while load is running
LockAcquiredTime      TIMESTAMP
LockOwnerRunID        STRING     -- PipelineRunID holding the lock
LockOwnerExecutionID  STRING     -- ExecutionID holding the lock
LockReleasedTime      TIMESTAMP  -- null while locked
ReleasedReason        STRING     -- null while locked; required on manual release
LastUpdated           TIMESTAMP

Why the lock outlived the writer. The process didn’t release cleanly – network interrupt, Fabric session timeout, cluster eviction. The cleanup code was in a finally block that never ran. The lock is now a zombie.

The escape hatch. Before executing, confirm in the pipeline monitor that no execution is currently active for this load. Releasing a lock while the process is still running creates exactly the race condition the lock was supposed to prevent.

UPDATE control.SchedulerLocks
SET IsLocked = false,
    LockReleasedTime = current_timestamp(),
    ReleasedReason = :reason
WHERE LoadKey = :load_key
  AND IsLocked = true
  AND LockOwnerRunID = :run_id;

The release utility binds :load_key, :run_id, and :reason as parameters – not string-formatted values. The operator supplies :reason interactively when invoking the utility. The LockOwnerRunID filter ensures you’re releasing the specific lock you diagnosed, not a lock another process may have acquired between your diagnosis and your update.

This is deliberately manual. Deliberately loud. ReleasedReason is required, not optional – if someone unlocks a load, they say why. Six months later, when someone queries the lock history, the reason is there in plain text.

How long before a lock is a zombie? In our scenario, four hours was the practical threshold – if a lock has been held longer than that with no active execution showing in the pipeline monitor, it didn’t release cleanly. Under four hours, the process might still be running on unusual volume. The threshold is a scheduler parameter – calibrate it against your slowest normal load. The principle of manual release with an audit trail isn’t a parameter.

Why not auto-release with a timeout? We considered it. The risk: a load legitimately runs long – unusual volume, slow source – the timeout fires, the lock releases, the scheduler fires again. Now you have two concurrent loads. Manual release with a human in the loop is slower, but the frequency of zombie locks (rare, not routine) makes the cost acceptable. For something that happens once a month, certainty is worth the extra step.

The lesson. Distributed locks need explicit escape hatches, and the escape hatch has to leave an audit trail. An unlock without a reason is technical debt with a timestamp. An unlock with a reason is a log entry a future engineer can use to decide whether the original lock was acquired correctly in the first place.

Runbook OPS-LOCK-01 covers zombie lock diagnosis (4h threshold), active-process verification, and manual release with required ReleasedReason.

Battle Scar #4 – The YAML in Git is not the YAML running

The symptom. A PR was merged that modified Sales_Salesforce.yml. The pipeline ran that night. The change didn’t take effect. Everyone reviewed the diff, approved the PR, confirmed the merge – and the pipeline behaved as if the change never happened.

The cause. The YAML in Git was updated. The YAML deployed to the Lakehouse – the one the engine actually reads at runtime – was the old one. The GitOps flow stopped at “merge.” The deployment was manual. And manual deployments are, eventually, forgotten.

Why the deploy wasn’t automated from day one. It should have been. The honest answer: other priorities, and “merge → manual upload” worked for the overwhelming majority of cases. Scar #4 is about the exceptions. Post 1 states automated deployment as a starting principle – “The Lakehouse contains deployed artifacts, not editable ones” – because we learned it from this very scar.

The fix – a process before a tool.

1. PR with the YAML change.
2. Pre-deploy Yamale validation (from Post 2).
3. Manual upload to Lh_Metadata/Files/config/ with a deployment checklist.
4. Smoke test with entities_filter – the engine supports running a subset before committing to a full load:

   entities_filter = ["Opportunities"]  # only this entity
   # after validation:
   entities_filter = None  # all entities (Python None, not the string "none")
   

5. Full run only after the filtered run confirms the change took effect.
6. Runbook CFG-SYNC-01 documents all of this.

Automating the deploy was the eventual goal – PR merge triggering an automatic copy to the Lakehouse folder, with the commit hash tagged in the deployed file. The process came first; the automation is still pending. Tools enforce process; they don’t create it – and process without tools is what we still rely on, scar and all.

DMBOK Ch. 6 §4.3 (“Script Usage for All Changes”) frames the principle the slow way: every database change should be scripted, version-controlled, and reviewable. We learned it the fast way – by living a deploy gap and writing the runbook that closes it.

The lesson – and the hardest one. “Versioned in Git” is not the same as “running in production” until there’s a deployment mechanism you can point to. GitOps without CD creates a category of drift that is harder to detect than environment drift, because both sides look correct individually. The Git version is correct. The deployed version is correct – it just happens to be the previous one. When something breaks, the first question – “is the YAML I’m looking at the one that ran?” – cannot be answered confidently. The fix is boring: a deploy script, a commit-hash tag in the config file. The cost of not having it is enormous.

Config drift has more than one dimension. The deployed/Git gap is the most visible, but there are others: entities enabled in YAML with no recent execution, entities enabled with no Delta table in the Lakehouse. Both can be spotted with a simple cross-reference: join deployed YAMLs against control.NotebookLog filtered to the last N days to find entities that are configured but never ran; join against the Lakehouse catalog to find entities that are configured but have no Delta table. All three share the same root – the YAML describes a world that isn’t the one actually running. The full consistency audit pattern is material for Post 6 (Living Metadata).

Runbook CFG-SYNC-01 covers the pre-deploy sequence: validation and the entities_filter smoke test. Deployment checklist and consistency audit triggers included.

Scars as teachers

Each scar above started as a production incident. Each one ended as a YAML field, a partition strategy, an audit column, or an entry in a runbook. None of them appear in the architecture diagrams. All of them are load-bearing.

Post 1 said Bronze should be boring. Post 2 said metadata governs metadata. Post 3 said the scheduler makes freshness a contract. This post adds one more: every scar leaves behind either an artifact or folklore. The artifact is the gold – a YAML flag, a partition strategy, a runbook. Folklore is the invisible adhesive – what the senior engineer knows and the junior engineer learns by breaking things at 3:00 am.

An architecture without scars is not one that got everything right on the first try. It’s one that hasn’t been stressed yet. The moment it meets real load, upstream changes, or a runtime surprise it didn’t plan for, it will start accumulating scars like every system that runs in production. The only question is which kind.

Back to kintsugi. The philosophy is direct: breakage and repair are part of the object’s history, not something to hide or disguise. The repaired piece is more honest than the one that arrived intact. It carries its history on the surface.

A data platform is no different. The patch buried in a notebook imitates the intact object: the crack exists but nobody sees it. parquet_date_fix: enabled: true in the YAML is the gold – the incompatibility is visible, declared, part of the contract. Runbook OPS-LOCK-01 is the gold – the scar has a name and a procedure; every execution leaves a reason in the log. These aren’t engineering embarrassments. They’re evidence of a system that learned something in production and brought it to the surface.

Artifacts don’t happen by themselves. The YAML field captures the workaround. The partition strategy captures the failure mode. The runbook captures what a human should do when it happens again – what to check when the lock is stuck, when the deploy didn’t propagate, when a column’s dates stopped parsing. The contract captures what the engine should do differently. Both matter. Neither survives unless someone writes it down.

What’s next

Post 5 – Runbooks as Infrastructure: Every scar above left behind not just a YAML field but a human-facing instruction: what the on-call engineer does when it happens again at 3:00 am. Those instructions are infrastructure, not documentation – versioned next to the engine, tested by the 3:00 am test, referenced by stable IDs from the control tables that paged the engineer in the first place. Why runbooks in Confluence rot, why runbooks in Git don’t, and what separates a runbook you can hand to someone without context from one that only the author can execute.

Post 6 – Living Metadata: Every scar in this post generated data. parquet_date_fix logs nullified values. SchedulerLocks records unlock reasons. notebooklog captures every partial failure. The control tables are full – but if nobody reads them, they’re dead metadata. How logs become reports, reports become decisions, and three different audiences (ops, quality, governance) end up consuming the same underlying data for three different purposes.

This is the fourth post in the series “YAML Metadata-Driven Ingestion.” The patterns described here have evolved across several enterprise Lakehouse implementations and are platform-agnostic, though our reference platform is Microsoft Fabric.