Oh bugger, I should have added that I'm cross-compiling from amd64 to ARM. I think that is important and probably would help clear things up.
And so for more completeness:
Build machine:
config.log
$ ./configure --prefix=/arm-linux-gnueabihf --build=x86_64-cross-linux-gnu --host=arm-linux-gnueabihf --without-included-regex --quiet CFLAGS=-O3 -pipe -mfpu=neon -mtune=cortex-a9 -march=armv7-a -mfloat-abi=hard CXXFLAGS=-O3 -pipe -mfpu=neon -mtune=cortex-a9 -march=armv7-a -mfloat-abi=hard LDFLAGS=-L/arm-linux-gnueabihf/lib -Wl,-rpath -Wl,/arm-linux-gnueabihf/lib CPPFLAGS=-I/arm-linux-gnueabihf/include
What I ended up doing was just sed-ing configure to wipe out Wcast-align.