The measurements, and which way each difference runs
A4 is defined metrically: 210 millimetres wide by 297 millimetres tall, which in the PDF point unit comes to approximately 595 by 842 points. US Letter is defined in inches: 8.5 by 11, which is exactly 612 by 792 points, since a point is 1/72 of an inch. Those are the four numbers the whole problem reduces to.
Compare them in the right direction and the relationship is easy to remember: A4 is narrower and taller than Letter. Narrower by roughly 17 points, a little under a quarter of an inch, and taller by about 50 points, close to seven tenths of an inch. Neither dimension is a superset of the other, which is the crux of the incompatibility — there is no orientation or rotation in which one size simply contains the other.
Geographically the split is straightforward. A4 is the standard almost everywhere in the world, part of the ISO 216 paper series. Letter is the standard in the United States and Canada, alongside its relatives Legal and Tabloid. So a document crossing between North America and anywhere else is very likely crossing a paper size boundary at the same time, usually without anybody mentioning it.