It looks as though the bug is that the prompt() function does its own checking, returning EISDIR unless x->recursive is set, and this code wasn't updated with the addition of -d. I've attached a patch which fixes this bug and adds a test of that code path. The fixes can also be retrieved from https://github.com/rkd91/coreutils_rm_di_patch.
I haven't submitted patches for coreutils before; if there's somewhere else I should post this patch to, I'm happy to be pointed there.
Best,
Rob