Parse, Validate and Transform

What this concept is

A data pipeline is a sequence of transformations with a contract at each boundary. Here the stages are bounded bytes, decoded text, parsed fields, validated records and a calculated report. Validation protects the next stage from assumptions that have not yet been established.

Why it matters here

This is your first small import tool. Separating stages lets you say whether an error came from encoding, structure or a domain rule, instead of treating every failure as an empty data set.

Concrete example

Imagine three records: Python,25, Math,0, Reading,15. The CSV structure is valid, but zero violates the allowed range. Under our whole-input rejection policy the tool produces no successful summary. Reporting count 2 and total 40 would silently discard an invalid record and misrepresent the source.

In contrast, a header-only file is explicitly valid and produces count 0, total 0 and an empty topic map. These two situations need different outcomes even though neither should produce a positive total.

Common confusion / misconception

Do not confuse cleanup with permission to invent missing values. Stripping surrounding whitespace is an agreed transformation; changing two into 2 is not. Do not catch every exception and return an empty list. Programmer defects and rejected inputs are different evidence.

How to use it

Read at most the documented limit plus one byte, so oversize input can be detected without an unlimited read. Decode strictly, parse with a library, then validate every record before aggregating. Raise an informative error with a record location. Keep input untouched and write reports to a separate destination only after success.

Check yourself

Why read 4097 bytes when the limit is 4096? The extra byte distinguishes a full allowed file from one that exceeds the limit. The parser rejects the latter; this is not permission to truncate it and accept the prefix.

Mini drill or application

Create one bad input for each stage: oversize bytes, invalid UTF-8, malformed CSV and an out-of-range duration. Predict which stage rejects it. Preserve the file bytes before and after running the tool.

Read this only if stuck

Use Python CSV for parsing behavior. The staged rejection policy is the lab contract, not a feature the CSV parser automatically provides.