We construct an orthogonal basis for the space of $m \times n$ matrices with row and column sums equal to zero. This vector space corresponds to the affine space naturally associated with the Birkhoff polytope, contingency tables and Latin squares. We also provide orthogonal bases for the spaces underlying magic squares and Sudoku boards. Our construction combines the outer (i.e., tensor or dyadic) product on vectors with certain rooted, vector-labeled, binary trees. Our bases naturally respect the decomposition of a vector space into centrosymmetric and skew-centrosymmetric pieces; the bases can be easily modified to respect the usual matrix symmetry and skew-symmetry as well. We'll finish the talk with a discussion of potential applications.