GNU bug report logs - #15483
POSIXLY_CORRECT documentation vis a vis some simple EREs

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 15483 in the body.
You can then email your comments to 15483 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Glenn Golden <gdg <at> zplane.com>
To: bug-grep <at> gnu.org
Subject: POSIXLY_CORRECT documentation vis a vis some simple EREs
Date: Sat, 28 Sep 2013 11:52:38 -0600

--
Regarding EREs having leading repetition operators, e.g. '*xyz':

Section 9.5.3 of

    http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html 

supplies the grammar for POSIX-conforming EREs. From the notes at the very
bottom:

    -----------------------------------------------------------------------
    The ERE grammar does not permit several constructs that previous
    sections specify as having undefined results:

	[ ... ]

        * One or more ERE_dupl_symbols appearing first in an ERE, or [ ... ]

    Implementations are permitted to extend the language to allow these.
    Conforming applications cannot use such constructs. 
    -----------------------------------------------------------------------


To my eyes, the last sentence seems to say that a conforming implementation
must not accept EREs like '*xyz'. But egrep [grep 2.14] does accept them,
even with POSIXLY_CORRECT defined, e.g.

   $ export POSIXLY_CORRECT=1
   $ echo 'abcdefghi' | egrep --color=auto '*def'

matches 'def'.  In contrast, POSIX regex(3) rejects such EREs with "invalid
preceding regular expression".

Not sure whether this is a POSIX conformance issue or not; it depends on the
intended semantics of POSIXLY_CORRECT.

To my eyes, the man page is a bit ambiguous, since it first says that it
"behaves as POSIX.2 requires", but then goes on to list only some specific
behaviors related to option processing.  It wasn't clear to me whether listing
the option-related behavior was intended to limit the scope of the
POSIXLY_CORRECT-ness to only those aspects, or if they were listed just
because they are (for example) often confusing to users, hence worthwhile to
call out explicitly.

In summary, there are a few questions/branches to this:

   1. If POSIXLY_CORRECT is intended to be conforming only in the specific
      respects listed, I'd suggest that the name of the associated envar be
      changed to reflect that (e.g., something like POSIXLY_CORRECT_OPTS),
      and also to change the man page text to read something like:

       POSIXLY_CORRECT_OPTS
	  If set, grep conforms with POSIX.2 with regard to the following
          option processing behaviors: [ description of option behaviors ]

   2. If POSIXLY_CORRECT is intended to mean 'fully conforming in all respects'
      then it seems like the present behavior is in technical violation.

   3. If (2) is the case, and the decision is made to change the behavior of
      grep accordingly, it might be worthwhile to also change the doc for
      POSIXLY_CORRECT to something like this:

       POSIXLY_CORRECT
	  If set, grep conforms with POSIX.2 in all respects.  In particular,
          [ description of option-related behaviors and/or other behaviors
	    that are deemed worthwhile to call out explicitly ]

   4. If (2) is the case, but the decision is made not to change the behavior
      of grep (i.e. accept the non-conformance) it might be wortwhile to
      change the doc for POSIXLY_CORRECT to something like this:

       POSIXLY_CORRECT
	  If set, grep conforms with POSIX.2 in almost all respects.  In
          particular, [ description of option-related behaviors and/or other
          behaviors that are deemed worthwhile to call out explicitly ]. But
          it does not conform precisely regarding ERE's like '*xyz' [ and
          whatever other ways are known to be non-conforming. ]

To pre-answer an expected question, asked of a submitter (Roman Donchenko) in
a similar POSIX violation bug report (#37737): 

    Are you encountering this problem in a real-world usecase, or are you
    simply reporting a violation of the standard?

My response is essentially the same as Roman gave: I am reporting it only as a
violation, but otoh, the POSIX-mandated behavior makes a lot more sense to me
than the current behavior, since expressions like '*xyz' are almost always user
error; the intent is usually '.*xyz'.  So if such expressions were rejected by
egrep, it would IMO be a behavioral improvement for users (like, ummm... me)
who chronically mis-remember how '*' is interpreted by bash vs. grep. 

Message #8 received at 15483 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: gdg <at> zplane.com
Cc: 15483 <at> debbugs.gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Sat, 28 Sep 2013 17:17:50 -0600

[Message part 1 (text/plain, inline)]

On 09/28/2013 11:52 AM, Glenn Golden wrote:
>         * One or more ERE_dupl_symbols appearing first in an ERE, or [ ... ]
> 
>     Implementations are permitted to extend the language to allow these.
>     Conforming applications cannot use such constructs. 

POSIX does not require implementations to diagnose undefined behavior,
and the intent of POSIXLY_CORRECT is only to change behavior where we do
not comply with the requirements by default.

> Not sure whether this is a POSIX conformance issue or not; it depends on the
> intended semantics of POSIXLY_CORRECT.

It's definitely NOT a POSIX conformance issue.  The question can still
be raised on whether it is a consistency issue with other GNU
applications on how POSIXLY_CORRECT is normally used, and whether it is
a documentation bug.

>    1. If POSIXLY_CORRECT is intended to be conforming only in the specific
>       respects listed, I'd suggest that the name of the associated envar be
>       changed to reflect that (e.g., something like POSIXLY_CORRECT_OPTS),
>       and also to change the man page text to read something like:
> 
>        POSIXLY_CORRECT_OPTS
> 	  If set, grep conforms with POSIX.2 with regard to the following
>           option processing behaviors: [ description of option behaviors ]

No, for two reasons.  1. The name POSIXLY_CORRECT is already entrenched,
and it's too late to add a new variable name for marginal value.  2.
POSIXLY_CORRECT only changes behavior if our default is saner than what
POSIX requires, but where you can request the POSIX behavior even though
it is arguably not as nice as the default.

> 
>    2. If POSIXLY_CORRECT is intended to mean 'fully conforming in all respects'
>       then it seems like the present behavior is in technical violation.

Fully conforming is pretty broad.  POSIX does not require grep to
diagnose where your usage of grep is undefined.  It only requires that
if you stick to the subset of regular expressions that are well-defined
by POSIX, then the behavior matches what POSIX states.  We can do
whatever we think makes the most sense with input that is beyond the
subset required by POSIX, and still be a POSIX-compliant implementation
while doing so.

> 
>    3. If (2) is the case, and the decision is made to change the behavior of
>       grep accordingly, it might be worthwhile to also change the doc for
>       POSIXLY_CORRECT to something like this:

I don't see the need to change any behavior.  We are already POSIX
conforming, since POSIX already states that your input string is
non-compliant.

> 
>    4. If (2) is the case, but the decision is made not to change the behavior
>       of grep (i.e. accept the non-conformance) 

What non-conformance?  You have to prove that we are mishandling input
that is well-defined by POSIX before we have non-conformance.  But POSIX
doesn't require us to do anything in particular with undefined behavior
- we can reject it, we can silently ignore it, we can give it useful
semantics as an extension, or any number of other things, and still be
fully compliant.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #11 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Glenn Golden <gdg <at> zplane.com>
To: bug-grep <at> gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Sun, 29 Sep 2013 11:25:52 -0600

--
Eric Blake <eblake <at> redhat.com> Sat, 28 Sep 2013 17:17:50 -0600:
>
> the intent of POSIXLY_CORRECT is only to change behavior where we do
> not comply with the requirements by default.
>

OK, that's a good start at clarifying the intent of POSIXLY_CORRECT, and glad
to see it.  IMO it would be worthwhile for us to hash this out further, if
you're not against doing so, since it has (to me) always been a bit ambiguous
as to what it implies in specific tools. Maybe others would benefit from it too.

From your response, it seems that the central question is whether there is or
is not a conformance issue. So in what follows I will leave out any discussion
of how one might deal with non-conformance (via envars, etc, as mentioned in
my original posting and your response) and just concentrate solely on the
conformance question.

To summarize our views: I suspect (but am not sure) that there is a compliance
issue; you emphatically believe there is not:

>
> It's definitely NOT a POSIX conformance issue.
>

You've made your case in part by the following:

>
> You have to prove that we are mishandling input that is well-defined by
> POSIX before we have non-conformance.
>

I agree with the above, but only if you replace "have to" with "it is
sufficient to".  "Have to" implies that it is _necessary_ to prove mishandling
in order to demonstrate non-conformance.  I disagree, and claim that it is only
sufficient, not necessary.  In other words, showing mishandling of a POSIXly
well-defined input is one way to demonstrate non-conformance, but not the only
way.

I'll try now to make the case that there is another way to show non-conformance,
and I'll try to be more careful than I was in the original post. You're going
to hate me for what follows, so let me preface it with a few observations from
my own experience with IEEE standards so you'll hopefully hate me less. :)

The intent of being as punctilious as I'm about to be is not to be an
asshole -- though I suspect you may disagree :) --  but to minimize
contaminating the discussion with personal opinions, personal preferences,
or playing of word games. Just a step by step logical progression with an
earnest intent to get to the bottom of an issue on which two reasonable people
disagree.  Having spent many years involved in IEEE standards activity
(802, nowhere near POSIX) my experience has been that this even though
exercises like this may seem ridiculous and time-wasting, the plus side is
that when you finally finish, you wind up either coming to a mutually agreeable
conclusion which can be logically justified in view of the standard and common
sense, or you wind up identifying sloppy language or logical faults within the
standard.  Either of those is a useful result.  So please bear with me.
(But by all means enjoy it for its amusement value. :)

In what follows, 'grep' means GNU grep 2.14 (but probably applies to many
earlier versions as well).

Begin by asking some questions, answering each as best I can in view of the
standard, and using ordinary reasoning:

   1. Is grep a computer program?  			yes

	Self-evident

   2. Does grep perform some desired function?		yes

	Self-evident

   3. Is grep an 'application' as defined by POSIX:	yes

	3.17 Application: A computer program that performs some
	desired function

   4. Does grep 'use' the ERE '*xyz'			yes

	Regardless of the state of POSIXLY_CORRECT, grep -E accepts '*xyz' as
        a valid input, performs a well-defined operation based upon it, and
        produces a well-defined result. I can't come up with any non-tortuous
        interpretation of the word "use" that allows one to say with a straight
        face that "grep cannot use the ERE '*xyz'". So my conclusion is that
	grep _can_ use '*xyz' (and can do so regardless of POSIXLY_CORRECT).
	

   5. Is grep a 'conforming application'		no

	Per the final sentence of 9.5.3, "conforming applications cannot use
	[constructs like '*xyz']".  By a well known rule of logic -- modus
	tollens, to be pretentious --  if A implies B, then !B implies !A.
	Define statements A and B as follows:

	    Statement A:  "application X is conforming"
	    Statement B:  "application X cannot use ERE '*xyz'"

	Implication: (Application X can use '*xyz') implies (application X
	    is not conforming);

	Conclusion: grep is a non-conforming application.

Now let's ask what is probably the more relevant question: Can an application
which is non-conforming be part of a conforming implementation?

And here, I am at a loss for any satisfying answer which does not either become
quickly bogged down in silliness or does violence to common sense. This is why
I say I'm not sure if there is a conformance issue or not.  Really, it seems
like sloppy language.

On one hand, the standard seems to want to draw a line between "programs which
are part of an implementation" and "programs which are applications". But on
the other hand, it defines "utility program" and "application" in ways which
makes it impossible to deny that a program like grep is both a component of
"an implementation" and simultaneously "an application".

So I turn it over to you and hope for a detailed reasoning similar to the
above, either refuting that grep is a non-conforming application, or accepting
it and addressing the question of conforming application vs. conforming
implementation.

Message #16 received at 15483-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: gdg <at> zplane.com, 15483-done <at> debbugs.gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Sun, 29 Sep 2013 16:53:21 -0700

Glenn Golden wrote:
> Per the final sentence of 9.5.3, "conforming applications cannot use
> 	[constructs like '*xyz']"

This is making the incorrect assumption that 'grep'
internally must be implemented as a strictly conforming
POSIX application.  POSIX does not require that, and the
rest of your conclusions therefore do not follow.

Eric explained the intent of POSIXLY_CORRECT pretty well.
Occasionally people ask for a different feature, where
a GNU application diagnoses the use of any extension to
POSIX.  The need for such a feature is less, though, and
the hassle is greater, and so it's typically not worth
the aggravation.

As this does not seem to be a bug in grep I'm going to
take the liberty of marking it 'done'.

Message #19 received at 15483 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: gdg <at> zplane.com
Cc: 15483 <at> debbugs.gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 30 Sep 2013 08:21:30 -0600

[Message part 1 (text/plain, inline)]

On 09/29/2013 11:25 AM, Glenn Golden wrote:
> --
> Eric Blake <eblake <at> redhat.com> Sat, 28 Sep 2013 17:17:50 -0600:
>>
>> the intent of POSIXLY_CORRECT is only to change behavior where we do
>> not comply with the requirements by default.
>>
> 
> OK, that's a good start at clarifying the intent of POSIXLY_CORRECT, and glad
> to see it.  IMO it would be worthwhile for us to hash this out further, if
> you're not against doing so, since it has (to me) always been a bit ambiguous
> as to what it implies in specific tools. Maybe others would benefit from it too.

You are welcome to submit documentation patches to clarify this as you
see fit (I'm probably a bit too biased with the current status quo to
write such a patch myself, as I'm liable to overlook what seems obvious
to me but not so obvious to a newcomer).

>    3. Is grep an 'application' as defined by POSIX:	yes
> 
> 	3.17 Application: A computer program that performs some
> 	desired function

What are the desired functions of GNU grep? I claim that ONE of the
desired functions is to be usable as an implementation that can serve in
a strictly-conforming POSIX environment with known behavior.  I also
claim that another desired function of GNU grep is to provide useful
extensions beyond the bare minimum specified by POSIX.

Next, it is required that the implementation of grep be limited to
solely a strictly conforming POSIX application, in order to provide its
functionality of being used in a strictly conforming environment?
Heavens no.  Just because you CAN write a program in a strictly
conforming environment does not imply that you MUST write a program in
that limited environment, and we'd much rather write grep to take
advantage of extensions outside the realm of POSIX.  Doing so makes our
life easier as maintainers, and makes grep more useful in its target
audience of GNU/Linux where people have come to expect having extensions
enabled by default.

Case in point: have you ever typed 'grep --help' to see what grep can
do, or 'grep --version' to see what build you are using?  Well, such a
command line is not permissible of a strictly conforming use of grep
(ie. POSIX states that it is undefined behavior to pass a command line
argument that has a length larger than 2 and which starts with two
dashes).  And we implement it in grep via the non-POSIX getopt_long()
function (initially exported by glibc, but made available via the gnulib
project for use on other platforms where libc has not made that
extension).  So our handling of your non-compliant input is best done by
using a non-compliant function within our code.

>    4. Does grep 'use' the ERE '*xyz'			yes

Only insofar as GNU grep has decided to give defined semantics, as an
extension, in its handling of what POSIX has declared as undefined.  But
if you only ever pass strictly-conforming input to grep, you will never
know what grep would have done with your input.  And if you pass '*xyz'
to grep, you are already outside the realm of strict conformance, so you
no longer has POSIX as your argument on what should happen, and
therefore, POSIXLY_CORRECT should have no bearing on that effect.

> 
> Now let's ask what is probably the more relevant question: Can an application
> which is non-conforming be part of a conforming implementation?

Yes, insofar as it's behavior when limited to conforming input is
likewise conforming.  POSIX specifically permits extensions.  If ALL
implementations were REQUIRED to reject '*xyz', then the behavior of
'*xyz' would be well-defined, and then you could talk about conformance
issues if GNU grep didn't reject it.  But by very definition, POSIX
cannot define what happens on undefined input, and the very reason that
POSIX documents some things as undefined is precisely to allow for
extensions.

In all of this, you seem to be asking for a mode of operation in grep
that explicitly rejects all POSIX-undefined behavior as an immediate
error.  GNU sed has tried to implement such an approach - if you use
'sed --posix', that tries as hard as possible to gracefully reject
anything that is not strictly conforming; and which is a different mode
than 'POSIXLY_CORRECT=1 sed' where extensions are still left enabled and
the only changes are when behavior is not compliant by default.  If you
wanted to, you could submit a patch to implement 'grep --posix' which
explicitly rejected all undefined POSIX behavior (well, all except for
the fact that the very command line to enter such a mode is via an
extension to POSIX).  But I'm not going to write such a patch myself (I
see very little benefit to be gained compared to the cost of maintaining
the patch).

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #22 received at 15483-done <at> debbugs.gnu.org (full text, mbox):

From: Glenn Golden <gdg <at> zplane.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 15483-done <at> debbugs.gnu.org, bug-grep <at> gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 30 Sep 2013 09:01:27 -0600

--
Paul Eggert <eggert <at> cs.ucla.edu> [2013-09-29 16:53:21 -0700]:
> Glenn Golden wrote:
> > Per the final sentence of 9.5.3, "conforming applications cannot use
> > 	[constructs like '*xyz']"
> 
> This is making the incorrect assumption that 'grep' internally must be
> implemented as a strictly conforming POSIX application.
>

Disagree strongly that any such assumption is being made. The reasoning 
step that you excerpt from above is strictly deductive; no assumptions of
any kind are present.

>
> POSIX does not require that,
>

Agree.

>
> and the rest of your conclusions therefore do not follow.
>

Permit me to disagree and to substantiate that disagreement in detail, but at
a later time.  I will not be able to respond further for a few days.

> 
> Eric explained the intent of POSIXLY_CORRECT pretty well.  Occasionally
> people ask for a different feature, where a GNU application diagnoses the
> use of any extension to POSIX.  The need for such a feature is less, though,
> and the hassle is greater, and so it's typically not worth the aggravation.
> 

I completely understand, and am sympathetic with the need to realistically
assess cost/benefit on any code change.  It's always easy for users to suggest
something that sounds "so simple", but can wind up as a great deal of work
to implement behind the scenes, open unintended cans of worms, etc.  Been
there, done that many times, and I fully sympathize.

Let me add tangentially that I think you guys (and all the gnu maintainers)
do a superb job and I respect your time and effort.  I realize I'm being a
bit of a gadfly on this particular issue, but as I said, I think something
constructive can come from exploring it carefully in a bit of detail.

I would hope when the discussion ends you will be able to say, "boy, that
guy was a verbose pain in the ass but he was being civil and raised some good
points that were beneficial to have in the ML." Actually I hope for more,
that some clarifying language changes can wind up being made in the
POSIXLY_CORRECT description, but let's save specifics on that for later in
the discussion.

>
> As this does not seem to be a bug in grep I'm going to take the liberty
> of marking it 'done'.
>

OK.  Do you object to continuing the discussion on the thread, even though
the ticket is closed? Or is there a more appropriate venue? 

Message #28 received at 15483 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: gdg <at> zplane.com
Cc: 15483-done <at> debbugs.gnu.org, Paul Eggert <eggert <at> cs.ucla.edu>,
 15483 <at> debbugs.gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 30 Sep 2013 11:27:11 -0600

[Message part 1 (text/plain, inline)]

On 09/30/2013 09:01 AM, Glenn Golden wrote:

>>
>> As this does not seem to be a bug in grep I'm going to take the liberty
>> of marking it 'done'.
>>
> 
> OK.  Do you object to continuing the discussion on the thread, even though
> the ticket is closed? Or is there a more appropriate venue? 

Continuing here is just fine.  We can reopen the ticket if needs be, if
we end up with a patch to documentation as a result of the conversation.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #34 received at 15483 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
Cc: 15483 <at> debbugs.gnu.org, gdg <at> zplane.com
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 30 Sep 2013 11:32:07 -0600

[Message part 1 (text/plain, inline)]

On 09/30/2013 08:21 AM, Eric Blake wrote:

> Case in point: have you ever typed 'grep --help' to see what grep can
> do, or 'grep --version' to see what build you are using?  Well, such a
> command line is not permissible of a strictly conforming use of grep
> (ie. POSIX states that it is undefined behavior to pass a command line
> argument that has a length larger than 2 and which starts with two
> dashes).

Since we're playing the role of pedantry here, let me amend that:

POSIX states that it is undefined behavior to pass a command line
argument that has a length larger than 2 and which starts with two
dashes, IFF you did not first pass the '--' end-of-options argument.
(The behavior of 'grep -- --help' is well-defined: search for lines of
stdin that match the regular expression "--help".)

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #37 received at 15483-done <at> debbugs.gnu.org (full text, mbox):

From: Glenn Golden <gdg <at> zplane.com>
To: 15483-done <at> debbugs.gnu.org, bug-grep <at> gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 7 Oct 2013 10:53:18 -0600

--
OK, if Wednesday is Prince Spaghetti Day, I guess Monday must be POSIX Pedantry
Day.  Seems to be a pattern developing...  :)

Anyway...  In the discussion below, the following notation is used, just to
avoid overloading certain words like "strictly" with meanings that seemed
(to me) ambiguous in some earlier posts:

  P		The set of EREs accepted by the grammar defined in clause 9.5.3
  x = P		Regex x is a member of P
  x > P		Regex x is an extension to P (hence not a member of P)
  grep		GNU grep, 2.14
  'blah'	Words/phrases well-defined within POSIX (single-quotes used).

Paul Eggert <eggert <at> cs.ucla.edu> [2013-09-29 16:53:21 -0700]:
> Glenn Golden wrote:
> > Per the final sentence of 9.5.3, "conforming applications cannot use
> >     [constructs like '*xyz']"
>
> This is making the incorrect assumption that 'grep' internally must be
> implemented as a strictly conforming POSIX application.
>

I looked over the reasoning step that you excerpted from above, and honestly
see no sign of anything that could be construed as an assumption of any sort,
about grep or anything else.

What's going on in that step is a straightforward analysis of a sentence from
the standard (last sentence of clause 9.5.3).

The sentence is transformed into a logically equivalent form, and a free
variable "X" is introduced. Finally, "grep" is substituted for X, and the
deductive conclusion --  following from the transformed sentence and prior
deductive steps -- is that grep, when considered as an 'application', cannot
be considered to be a 'conforming application'.

In short, it seems to me straightforward deductive reasoning, free of any
assumptions at all.

If you still don't buy this, can you be more explicit about what led you to
conclude that this assumption was being made?  I'm not being intentionally
obtuse just to avoid your argument. If there's an argument there I'd like to
understand it, but honestly just don't see it.

In any case... Here's perhaps a better way to make the same point about the
final sentence of 9.5.3: Go at it initially without even considering grep;
purely from an application-writer's perspective.  This prevents complicating
the discussion with grep's [debatably] dual status as both a system 'utility'
and simultaneously as an 'application'. Bear with me, as I think this will
get to the core of the issue:

Suppose we decide to write a simple database application. It accepts a regex
as user-supplied input, extracts records from the database matching the regex,
then reformats the selected records in some fancy way, which becomes the
application's output.

To handle the input queries, we choose the extended regex engine from gnulib.
This engine accepts extended EREs like '*xyz' and others from the bullet list
in 9.5.3 which are disallowed by the reference grammar.  So, if a user queries
our app with an RE like '*xyz', the app interprets it in accordance with GNU
extended regex rules, and returns the matching records which we then format
and barf out.

Now we ask: Can our app, as described above, be certified as a 'conforming
application' (2.2.2)?

To answer, we need to examine all the 'shall' clauses in 2.2.2 to be sure we
comply with those. Let's say we do.  Now we also need an interpretation of the
last sentence of 9.5.3, which places a restriction on what a 'conforming
application' may do, and be sure that we _don't_ do that.

Unfortunately, the standard seems to offer no insight as to what is meant or
intended by "cannot use".  One interpretation, which seems non-tortuous to me,
is as follows:

    * Conforming applications must not accept x > P as input.

Or stated in a more practically actionable way:

    * If presented with an RE x > P as input, a conforming application must
      diagnose it as fatal (i.e., declare it in error and refuse to continue.)

Note that I'm not claiming that the above interpretation is Right; only that
it seems reasonable.

I checked the opengroup/IEEE site

    http://get.posixcertified.ieee.org/PR/

for any existing problem reports or interpretations on this clause. I found
none, but the site presently has numerous broken links, and it wasn't clear
the search feature was working properly. (I reported the issue to the site
admin like a good boy. :)

So before continuing with this line of reasoning, let me ask: Do you agree
with this interpretation?

If so, I'll continue.  If not, please explain a preferred interpretation. 
Understanding this is central to choosing text I'd submit for approval in
a doc patch (per Eric's suggestion to do so).

Message #43 received at 15483 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: gdg <at> zplane.com
Cc: 15483-done <at> debbugs.gnu.org, 15483 <at> debbugs.gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 07 Oct 2013 11:05:09 -0600

[Message part 1 (text/plain, inline)]

On 10/07/2013 10:53 AM, Glenn Golden wrote:

> The sentence is transformed into a logically equivalent form, and a free
> variable "X" is introduced. Finally, "grep" is substituted for X, and the
> deductive conclusion --  following from the transformed sentence and prior
> deductive steps -- is that grep, when considered as an 'application', cannot
> be considered to be a 'conforming application'.

So what?  No one cares whether grep is a 'conforming application', so
long as it can be used correctly as part of making a conforming
environment. Our counterargument is that we KNOW that grep in not a
conforming application, and WE DON'T CARE.  What we DO care about is
that in a conforming environment (where POSIXLY_CORRECT is set in the
environment), then grep behaves as required by POSIX for all
well-defined semantics, and that extension semantics are unaffected by
POSIXLY_CORRECT.

> 
> In short, it seems to me straightforward deductive reasoning, free of any
> assumptions at all.
> 
> If you still don't buy this, can you be more explicit about what led you to
> conclude that this assumption was being made?  I'm not being intentionally
> obtuse just to avoid your argument. If there's an argument there I'd like to
> understand it, but honestly just don't see it.

You're arguing about grep being non-conforming as if it mattered, but
our counterargument is that it does not matter.

> 
> To handle the input queries, we choose the extended regex engine from gnulib.
> This engine accepts extended EREs like '*xyz' and others from the bullet list
> in 9.5.3 which are disallowed by the reference grammar.

Being disallowed by the reference grammar does NOT imply that such regex
must cause an error.  POSIX is perfectly clear that implementations may
provide extensions that give non-error meaning to any regex that is not
strictly defined in the narrow subset required by POSIX.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #49 received at 15483-done <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: gdg <at> zplane.com
Cc: 15483-done <at> debbugs.gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 07 Oct 2013 11:33:32 -0600

[Message part 1 (text/plain, inline)]

On 10/07/2013 10:53 AM, Glenn Golden wrote:

> Unfortunately, the standard seems to offer no insight as to what is meant or
> intended by "cannot use".  One interpretation, which seems non-tortuous to me,
> is as follows:
> 
>     * Conforming applications must not accept x > P as input.

Not true.  Rather, the intent of POSIX is that if an application is
presented x > P as input, then the application is no longer bound by the
rules of POSIX, and undefined behavior will occur.  That is, rather than
make your application worry about the diagnosis of x > P, you can
instead transfer the blame to the user that is passing you bogus input.

Remember, POSIX also says that strlen(NULL) is undefined behavior.  In
some implementations, it succeeds (generally with the value 0); in
others it fails by causing SIGSEGV.  But the whole point is that it is
undefined - the burden is on the user to not pass invalid input in the
first place, and not on libc to react in any particular manner to that
bad input.  The burden of strictly conforming is on the caller to not
provide bad input in the first place, not on what the receiver must (or
must not) do with input that is out of spec.

> 
> Or stated in a more practically actionable way:
> 
>     * If presented with an RE x > P as input, a conforming application must
>       diagnose it as fatal (i.e., declare it in error and refuse to continue.)

That is one possible action to take on undefined behavior, and might
even be a desirable behavior, but it is NOT required behavior, and does
not happen to be the current behavior of grep unless someone writes a
patch.  Remember, 'sed --posix' is an extension to POSIX where GNU sed
tries hard to diagnose extension inputs, and we would not be opposed to
such a patch for grep if it is easy to maintain; but I am also
personally unlikely to write such a patch.

If you still worry about what POSIX intends or not, it is probably
better to ask the actual POSIX list (<austin-group-l <at> opengroup.org>), as
they would actually be able to do something about it.  This list is
probably not the most effective place to argue about what POSIX intended.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #52 received at 15483 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: gdg <at> zplane.com
Cc: 15483 <at> debbugs.gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 07 Oct 2013 11:11:37 -0700

On 10/07/13 09:53, Glenn Golden wrote:
> grep, when considered as an 'application'

There is no requirement that grep must
be implemented as a POSIX application.
'grep' can be written entirely in machine code.
As far as POSIX is concerned,
'grep' could even be a mechanical Turk,
i.e., not implemented via computer at all.

As a practical matter, users generally prefer
implementations that support useful extensions
to POSIX to implementations that support
only POSIX and nothing more.  Supporting a strict-POSIX
option for 'grep' sounds like it wouldn't be worth the
hassle for us.  Perhaps I'm wrong, and I could be
proven wrong if someone wrote an easy-to-maintain
patch that implements strict-POSIX behavior for 'grep'.

At any rate, it's not an urgent matter, as there's
no POSIX-conformance issue here, and little practical
benefit to any such change.

Message #55 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Glenn Golden <gdg <at> zplane.com>
To: bug-grep <at> gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 7 Oct 2013 15:24:47 -0600

--
Eric Blake <eblake <at> redhat.com> [2013-10-07 11:05:09 -0600]:
> 
> So what?  No one cares whether grep is a 'conforming application',
>

Disagree strongly that "no one" cares.

>
> Our counterargument is that we KNOW that grep in not a conforming application,
> and WE DON'T CARE.
>

From your perspective as developers/maintainers, I understand that _you_ don't
care -- nor should you -- whether grep is a conforming application, or even
an application at all for that matter: Grep was not _designed_ to be an
application within the POSIX framework;  you're not _viewing_ it as an
application; and you're not _maintaining_ it as an application. It's a system
component, and as such, part of a POSIX 'implementation', and it is entirely
conforming as such.  No argument. I agree and support that point of view
entirely. (And btw, the doc patch draft that I had planned to submit also
makes your points very clearly and explicitly, so as to leave no doubt in
the readers' mind that simply because grep is not a conforming application
does not imply that there is any "defect" in grep.)  Totally agree with your
point of view on that.

But!

I completely disagree that _no one_ cares.  As you said of yourself, you're
very close to all this and things which appear obvious to you as maintainers
may not be so obvious to others outside that realm. I claim that users in the
position of writing applications which use grep as a building block certainly
_do_ care about whether grep is or is not a conforming application, and
therefore this "duality" should certainly be called out in the doc, because
it has real relevance in the real world.

That's the case I was in the process of making in the prior post, but wanted
to find out first whether you agreed or not that grep was a non-conforming
application. It matters.

> 
> You're arguing about grep being non-conforming as if it mattered, but
> our counterargument is that it does not matter.
> 

To you it does not. To users writing applications and concerned about POSIX
conformance, I believe it does. I'll continue making that case if you wish,
but now I'm so sure you wish. :) Let me know.

>
> POSIX is perfectly clear that implementations may provide extensions that
> give non-error meaning to any regex that is not strictly defined in the
> narrow subset required by POSIX.
>

I agree entirely.  But the key word above is 'implementations'. It is not
clear that it is true for 'conforming applications' in all cases.

To me, the language contrast in 9.5.3. between "permitted" (pertaining to
'implementations') vs. "cannot use" (pertaining to 'conforming applications')
strongly suggests that in this section of the spec they specifically intended
to distinguish between "behavior which is permitted by internal components of
a POSIX system" [i.e. 'implementations', which certainly may use extended EREs 
and/or diagnose them or not as they see fit] vs. "behavior which is permitted
by conforming applications" [and hence does not fall under the rubric of
'implementation', and which "cannot use" ERE's > P.]  To me, the very presence
of the two distinct sentences is a red flag that they were trying to draw the
readers' attention to a distinction in conformance behavior between the two
entities ('implementations' and 'conforming applications') in regard to EREs.

Just got your follow-up email as I was writing this, which talks about the
above, and suggests bringing it up on the POSIX list. I will do so and spare
you any further noise until I get some response there.

Let me just conclude here by saying that I will be honestly surprised to find
that that they intended the phrase "cannot use x" (for x > P) to include the
interpretation "you may use x if you want, since such x is outside the scope
of POSIX to begin with." To me, it seems peculiar that they would write
"cannot use" so specifically, and it also seems like circular reasoning. If it
is true as you say that

    "...the intent of POSIX is that if an application is presented x > P as
     input, then the application is no longer bound by the rules of POSIX,
     and undefined behavior will occur."

then why was the "cannot use" sentence specifically placed into that section?

By your explanation above, there is no need for that sentence, because for
x > P, *anything* the standard says about an application's behavior is
irrelevant; after all, "the application is no longer bound by the rules of
POSIX".  So why in the world put an explicit sentence into a standard which
constrains an application's behavior, but applies *only* in the case (x > P)
in which the application "is no longer bound by the rules [of the standard]"? 
Doing so places the standard in the very strange position of making a specific
statement about a situation which, by your explanation above, is outside its
scope to make any statement about! :)

I don't get it.

On the other hand, the interpretation earlier suggested -- "cannot use" ==
"must reject" -- offers a straightforward ho-hum reason for why the writers
may have included that specific sentence: Because they desired to specify
how 'conforming applications' -- not 'implementations' -- should behave
specifically in the face of an input ERE which is undefined. One clean,
straightforward, and parsimonious explanation for the phrase "cannot use"
is that that it means "must reject", rather than "you can do what you want".

But, of course, that's just my own opinion. I may well be wrong. You guys
certainly know this spec far better than I, and my take is based only on work
with other IEEE standards. (In which case, as an aside, having chaired a few
subcommittees therein which often dealt with language pedantry like this,
I would never allow "cannot use" to be interpreted in the way that you have
described.) But, that's just me, and it doesn't take into account what you
guys know about POSIX's historical intent that I certainly don't.

Your example of strlen(NULL) does not seem to add clarity to this topic,
because (afaict) there is no specific sentence in the standard regarding
strlen(NULL) which is analogous to the very specific sentence in 9.5.3 calling
out explicitly that conforming applications "cannot use" x > P. The very fact
that that explicit sentence is present in 9.5.3 suggests to me that it was
placed there for a reason, and that one "reasonable reason" might have been
to explicitly say, "conforming apps must not accept such constructs".

Anyway... I will bring this up with the POSIX folks as you suggested. And
again, I apologize for being a fussbudget about this, but I think we have now
homed in on the basic disagreement, and it seems to me essential to understand
it.  Let's see if we can get some clarity from the POSIX ML on it.

Thanks for listening and responding in any case. I do appreciate your time and
will follow up when I learn more from the POSIX ML. (And btw, for sanity/stress
sake: Perhaps worthwhile to view this entire conversation with a certain degree
of humor which the extreme sport of pedantry deserves :) and not get too
irritated about it.  In the end, it may in fact turn out that I have been
wasting your time, but I assure you I am not *trying* to do so. :) I hope you
will eventually smile at all this in the spirit in which it was intended.)

Message #58 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Glenn Golden <gdg <at> zplane.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: 15483-done <at> debbugs.gnu.org, bug-grep <at> gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 14 Oct 2013 11:21:52 -0600

--
Yes boys and girls, it's Monday, time again for our next episode of...
POSIX Pedantry...

We left our intrepid heros poised on the edge of conformance eternity, with
Eric suggesting that Glenn raise various questions on the POSIX mailing list...

I did so, initially just asking about the interpretation of "cannot use" and
its history in the spec.  Once they responded to that, we got into more detail
on the background and motivation that led to that question, i.e. what we've
been discussing here:

    http://news.gmane.org/gmane.comp.standards.posix.austin.general

The topic subject is "Seeking interpretation of 'cannot use' in Section 9.5.3"
dated 2013-10-09 19:00:05 GMT.

Two people responded: Mark Ziegast of Shware Systems, and Geoff Clare from
OpenGroup.

Here's a summary of the discussion, as best I understand it: ["POSCOR" below is
shorthand for POSIXLY_CORRECT.]

 * Regarding the initial question about the semantics of "cannot use": Their
   interpretation seems to accord with the straightforward semantic that I had
   suggested here, which amounts to: If you accept x > P as input and do
   anything with it other than rejecting it, then you are "use"ing it; hence
   "cannot use" is effectively synonymous with "must reject".

   On this topic, neither Mark nor Geoff expressed any sense of unease or
   murkiness about that interpretation. 

 * Regarding the semantics of the last sentence of XBD 9.5.3 in its entirety,
   in view of my question asking about the history of how it came to be worded
   as it is: After examining the history, Geoff's take is that the leading
   phrase "Conforming applications..." is just plain wrong, and has been since
   c. 2001.  In concert with the straightforward interpretation of "cannot
   use", it effectively declares -- as I've been opining -- that any 
   application which fails to reject extended EREs as input is unconditionally
   non-conforming. 

   Mark agrees it is wrong, but disagreed mildly with Geoff as to how it ought
   to be fixed.  Geoff suggested simply adding the qualifier "strictly".
   Mark's approach [2013-10-12 09:50:10 GMT] is to be more specific, stating
   explicitly that apps which expose x > P constructs to the user, while not
   strictly or generally conforming, can be conforming in the more restricted
   sense of XBD 2.2.3, as long as they meet the associated documentary
   requirement.  (As I understand it, the documentary requirement is to list
   the x > P constructs accepted by the application.) If not documented in
   this way, then the app is considered non-conforming.

Assuming you accept the above as authoritative for our purposes here, here
are some follow-on observations:

 - Since "cannot use" has the straightforward interpretation I'd suggested,
   my earlier conclusions based on that interpretation are valid.  However,
   those conclusions are now totally irrelevant because the remaining text of
   the offending sentence has been wrong since 2001. :) There is amusement
   value here, I hope you're chuckling. 
   
 - The new (corrected) text for the last sentence of 9.5.3 essentially
   inverts its original meaning, rendering it sensible: The original text
   said that no application accepting x > P could be conforming, period.
   The corrected text implies that "almost any" application accepting x > P
   _can be_ conforming [per XBD 2.2.3], provided that it documents the set
   of x > P which it accepts. (And, of course, apps accepting x > P cannot
   claim strict conformance, but there was never any debate about that.)

 - As to how the above pertains to GNU grep, IMO: Well, the present [wrong]
   text, in concert with the interpretation of "cannot use" implies clearly
   that every application based on GNU grep since c. 2001 has been non-
   conforming, unless it had explicitly pre-filtered those x > P and rejected
   them before handing them to grep, which of course no one ever does.

   But!

   That historic (and present) non-compliance is now irrelevant, since it
   isn't what the spec intended to say. :)

   Nevertheless, in my own defense as a fussbudget, it does justify the
   reasoning that led to this thread in the first place, including the claim
   that there is at minimum a doc issue [w.r.t. POSCOR] and perhaps a
   conformance issue [w.r.t. the original, wrong, text].

   No need to dwell on this (unless you wish to) since it will soon be
   obviated by the corrected spec text. But at least I feel now like a
   vindicated fussbudget rather than a plain vanilla fussbudget.

As to where to go from here: Once the corrected language makes it to draft
form (from which it can be cited in the man page) I'll take it as an action
item to submit a doc patch against grep.1.

The patch will discuss the conformance issue in detail, including the app vs.
imp duality, and we can hash it out here. But honestly I doubt you'll find
much to argue with, as it simply states the facts.  It carefully makes the
point which you guys have made here that there is no "defect" in GNU grep's
implementation simply because it does not reject x > P when POSCOR is set,
and explains why that is so.

But it will also state, per the [corrected] spec, that apps based on GNU grep
cannot be strictly or generally conforming unless they explicitly pre-filter
x > P (since GNU grep presently has no means to do that) but can be conforming
per XBD 2.2.3 provided that they list those x > P accepted by GNU grep.
(Prior to the spec correction, that last phrase ["...but can be conforming..."]
was not possible to write; and that was the core issue that brought this
thread up in the first place.)

In short, the reader will come away with a clear understanding of how the
use of GNU grep in his application relates to its POSIX conformance.

That doc patch, along with the spec change, will, IMO, be a constructive and
positive outcome of the entire conversation. I hope that you agree, and I tip
my hat to you both in supporting/putting up with a discussion in which a
contentious issue was hashed out constructively to positive result.

Message #64 received at 15483 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: gdg <at> zplane.com, Eric Blake <eblake <at> redhat.com>
Cc: 15483 <at> debbugs.gnu.org
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 14 Oct 2013 10:53:13 -0700

On 10/14/13 10:21, Glenn Golden wrote:
> I'll take it as an action item to submit a doc patch against grep.1.

I wouldn't bother with grep.1, as it's not the real documentation.
Instead, I suggest a patch against the grep manual, which is
in Texinfo format.

I suggest adding something to the paragraph that talks about
POSIXLY_CORRECT.

Message #67 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Glenn Golden <gdg <at> zplane.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 15483 <at> debbugs.gnu.org, bug-grep <at> gnu.org, Eric Blake <eblake <at> redhat.com>
Subject: Re: bug#15483: POSIXLY_CORRECT documentation vis a vis some simple
 EREs
Date: Mon, 14 Oct 2013 12:39:44 -0600

--
Paul Eggert <eggert <at> cs.ucla.edu> [2013-10-14 10:53:13 -0700]:
> 
> I wouldn't bother with grep.1, as it's not the real documentation.
> Instead, I suggest a patch against the grep manual, which is
> in Texinfo format.
> 
> I suggest adding something to the paragraph that talks about POSIXLY_CORRECT.
>

Good idea, I'll concentrate the efforts there.

However, since the man page is clearly deficient wrt documentation of
_application_ conformance, I will also submit a short patch to grep.1 which
simply suggests that the reader refer to the info(1) doc for more detailed
discussion of conformance issues.  That should cover all bases.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.