Subject: bug#20526: BUG: text file is detected as binary

From: Paul Eggert <[email-masked]>
Date: Thu, 07 May 2015 13:07:20 -0700

On 05/07/2015 10:47 AM, Sebastian Pöhn wrote:
>
> Thanks for this fast feedback. Your explanation sounds very 
> reasonable. As you may have noticed this a makefile out of openwrt 
> with is mainlined there.
>
> 1) I downgraded to grep 2.20. Issue is gone with the same environment. 
> So this is in my eyes a regression.
>

Not really, as Openwrt is relying on undefined behavior.  The spec for 
grep has never defined what grep does when you feed it binary data that 
is not properly encoded for the current locale.  Different versions of 
grep (and we're not just talking GNU grep here, but other 
implementations) do different things.  Some grep implementations dump 
core.  These behaviors all conform to the spec.  (Well, GNU grep isn't 
supposed to dump core, but older versions of GNU grep are buggy and will 
dump core sometimes anyway, so you'll need good luck with them.)

> 2) I will also open a report at fedora, maybe the use some strange 
> setting in building the new packet.
>

Nowadays most people are using UTF-8, so I suggest encoding the 
Makefiles in UTF-8 and specifying a UTF-8 locale when you build. Another 
possibility is the attached hack (I haven't tried it).  The most 
conservative course would be to insist that Makefiles be ASCII, although 
....

> 3) I will send a short notice to openwrt asking if they think it is 
> fine to use ë or ö. I personally have a strong opinion on that ;)
>

Don't blame you a bit.
diff --git a/include/scan.mk b/include/scan.mk
index c2a8f7e..2a83924 100644
--- a/include/scan.mk
+++ b/include/scan.mk
@@ -55,7 +55,7 @@ endif
 
 $(FILELIST): $(OVERRIDELIST)
 	rm -f $(TMP_DIR)/info/.files-$(SCAN_TARGET)-*
-	$(call FIND_L, $(SCAN_DIR)) $(SCAN_EXTRA) -mindepth 1 $(if $(SCAN_DEPTH),-maxdepth $(SCAN_DEPTH)) -name Makefile | xargs grep -HE 'call $(GREP_STRING)' | sed -e 's#^$(SCAN_DIR)/##' -e 's#/Makefile:.*##' | uniq | awk -v of=$(OVERRIDELIST) -f include/scan.awk > $@
+	LC_ALL=C; export LC_ALL; $(call FIND_L, $(SCAN_DIR)) $(SCAN_EXTRA) -mindepth 1 $(if $(SCAN_DEPTH),-maxdepth $(SCAN_DEPTH)) -name Makefile | xargs grep -HE 'call $(GREP_STRING)' | sed -e 's#^$(SCAN_DIR)/##' -e 's#/Makefile:.*##' | uniq | awk -v of=$(OVERRIDELIST) -f include/scan.awk > $@
 
 $(TMP_DIR)/info/.files-$(SCAN_TARGET).mk: $(FILELIST)
 	( \