The invisible first three bytes

A client's validation step started rejecting every file from a new partner with the same error: required column order_id missing. The header clearly said order_id. Opening the file in an editor showed nothing unusual.

$ head -c 16 partner-export.csv | xxd
00000000: efbb bf6f 7264 6572 5f69 642c 6375 7374  ...order_id,cust

The first three bytes, EF BB BF, are a UTF-8 byte order mark. Some spreadsheet tools add it when saving "CSV UTF-8". Most viewers hide it, so the first column is actually named \ufefforder_id, which does not match anything.

The fix

In Python, opening the file with encoding="utf-8-sig" strips the mark if it is present and does nothing if it is not, so it is safe to use everywhere:

with open(path, newline="", encoding="utf-8-sig") as fh:
    reader = csv.DictReader(fh)

We made this the default in eotl-check 0.4.1. It is a one-word change, and the kind of thing worth fixing once in a shared tool rather than in every importer separately.

← All notes