Subject: Re: bug#23019: parse-partial-sexp doesn't output the full state needed for its continuance.

From: Stefan Monnier <[email-masked]>
Date: Fri, 18 Mar 2016 00:49:07 -0400

> Do this by adding two new fields to the parser state: the syntax of the last
> character scanned, and the last end of comment scanned.  This should make the
> parser state complete.

Thanks.  I like the "syntax of the last character scanned", but I don't
understand the reasoning behind "last end of comment scanned".  Why is
this relevant?  Is it in case the "last character scanned" was a "slash
ending a comment" so as to avoid treating "*/*" as both a comment closer and
a subsequent opener?

If so, I'm not sure I like it.  It sounds to me like there's a chance
it's actually incomplete (e.g. it doesn't address the similar problem
when the "last character scanned" is an end of a string which also
happens to be a valid first-char of a comment-starter), and even if it
isn't, it "feels ad-hoc" to me.

Would it be difficult to do the following instead:
- get rid of element 11.
- change element 10 so it's nil if the last char was an "end of
  something".  Another way to look at it, is that the element 10 should
  only be non-nil if the "next lexeme" might start on that
  previous character.

I also have a side question: IIUC your patch makes the 5th element
redundant (can be replaced with a test whether "last char syntax" was
"escape"), is that right?

        Stefan

> Also document element 9 of the parser state.  Also refactor the code a bit.

> * src/syntax.c (struct lisp_parse_state): Add two new fields.
> (internalize_parse_state): New function, extracted from scan_sexps_forward.
> (back_comment): Call internalize_parse_state.
> (forw_comment): Return the syntax of the last character scanned to the caller.
> (Fforward_comment, scan_lists): New dummy variables, passed to forw_comment.
> (scan_sexps_forward): Remove a redundant state parameter.  Access all `state'
> information via the address parameter `state'.  Remove the code which converts
> from external to internal form of `state'.  Access buffer contents only from
> `from' onwards.  Reformulate code at the top of the main loop correctly to
> recognize comment openers when starting in the middle of one.  Call
> forw_comment with extra argument (for return of final syntax value).
> (Fparse_partial_sexp): Document elements 9, 10, 11 of the parser state in the
> doc string.  Clarify the doc string in general.  Call
> internalize_parse_state.  Take account of the new elements when consing up the
> output parser state.

> * doc/lispref/syntax.texi: (Parser State): Document element 9 and the new
> elements 10 and 11.  Minor wording corrections (remove reference to "trivial
> cases").
> (Low Level Parsing): Minor corrections

> diff --git a/doc/lispref/syntax.texi b/doc/lispref/syntax.texi
> index d5a7eba..67a00d7 100644
> --- a/doc/lispref/syntax.texi
> +++ b/doc/lispref/syntax.texi
> @@ -791,10 +791,10 @@ Parser State
>  @subsection Parser State
>  @cindex parser state
 
> -  A @dfn{parser state} is a list of ten elements describing the state
> -of the syntactic parser, after it parses the text between a specified
> -starting point and a specified end point in the buffer.  Parsing
> -functions such as @code{syntax-ppss}
> +  A @dfn{parser state} is a list of (currently) twelve elements
> +describing the state of the syntactic parser, after it parses the text
> +between a specified starting point and a specified end point in the
> +buffer.  Parsing functions such as @code{syntax-ppss}
>  @ifnottex
>  (@pxref{Position Parse})
>  @end ifnottex
> @@ -851,15 +851,21 @@ Parser State
>  this element is @code{nil}.
 
>  @item
> -Internal data for continuing the parsing.  The meaning of this
> -data is subject to change; it is used if you pass this list
> -as the @var{state} argument to another call.
> +The list of the positions of the currently open parentheses, starting
> +with the outermost.
> +
> +@item
> +The @var{syntax-code} (@pxref{Syntax Table Internals}) of the last
> +buffer position scanned, or @code{nil} if no scanning has happened.
> +
> +@item
> +The position after the previous end of comment, or @code{nil} if the
> +scanning has not passed a comment end.
>  @end enumerate
 
>    Elements 1, 2, and 6 are ignored in a state which you pass as an
> -argument to continue parsing, and elements 8 and 9 are used only in
> -trivial cases.  Those elements are mainly used internally by the
> -parser code.
> +argument to continue parsing.  Elements 9 to 11 are mainly used
> +internally by the parser code.
 
>    One additional piece of useful information is available from a
>  parser state using this function:
> @@ -898,11 +904,11 @@ Low-Level Parsing
 
>  If the fourth argument @var{stop-before} is non-@code{nil}, parsing
>  stops when it comes to any character that starts a sexp.  If
> -@var{stop-comment} is non-@code{nil}, parsing stops when it comes to the
> -start of an unnested comment.  If @var{stop-comment} is the symbol
> +@var{stop-comment} is non-@code{nil}, parsing stops after the start of
> +an unnested comment.  If @var{stop-comment} is the symbol
>  @code{syntax-table}, parsing stops after the start of an unnested
> -comment or a string, or the end of an unnested comment or a string,
> -whichever comes first.
> +comment or a string, or after the end of an unnested comment or a
> +string, whichever comes first.
 
>  If @var{state} is @code{nil}, @var{start} is assumed to be at the top
>  level of parenthesis structure, such as the beginning of a function
> diff --git a/src/syntax.c b/src/syntax.c
> index 249d0d5..11b1ff0 100644
> --- a/src/syntax.c
> +++ b/src/syntax.c
> @@ -153,6 +153,9 @@ struct lisp_parse_state
>      ptrdiff_t comstr_start;  /* Position of last comment/string starter.  */
>      Lisp_Object levelstarts; /* Char numbers of starts-of-expression
>  				of levels (starting from outermost).  */
> +    int prev_syntax; /* Syntax of previous character scanned, or Smax. */
> +    ptrdiff_t prev_comment_end; /* Position after end of last closed
> +                                   comment, or -1. */
>    };
>  
>  /* These variables are a cache for finding the start of a defun.
> @@ -176,7 +179,8 @@ static Lisp_Object skip_syntaxes (bool, Lisp_Object, Lisp_Object);
>  static Lisp_Object scan_lists (EMACS_INT, EMACS_INT, EMACS_INT, bool);
>  static void scan_sexps_forward (struct lisp_parse_state *,
>                                  ptrdiff_t, ptrdiff_t, ptrdiff_t, EMACS_INT,
> -                                bool, Lisp_Object, int);
> +                                bool, int);
> +static void internalize_parse_state (Lisp_Object, struct lisp_parse_state *);
>  static bool in_classes (int, Lisp_Object);
>  static void parse_sexp_propertize (ptrdiff_t charpos);
 
> @@ -911,10 +915,11 @@ back_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	}
>        do
>  	{
> +          internalize_parse_state (Qnil, &state);
>  	  scan_sexps_forward (&state,
>  			      defun_start, defun_start_byte,
>  			      comment_end, TYPE_MINIMUM (EMACS_INT),
> -			      0, Qnil, 0);
> +			      0, 0);
>  	  defun_start = comment_end;
>  	  if (!adjusted)
>  	    {
> @@ -2314,7 +2319,9 @@ in_classes (int c, Lisp_Object iso_classes)
>     into *CHARPOS_PTR and the corresponding bytepos into *BYTEPOS_PTR.
>     Else, return false and store the charpos STOP into *CHARPOS_PTR, the
>     corresponding bytepos into *BYTEPOS_PTR and the current nesting
> -   (as defined for state.incomment) in *INCOMMENT_PTR.
> +   (as defined for state->incomment) in *INCOMMENT_PTR.  The
> +   SYNTAX_WITH_FLAGS of the last character scanned in the comment is
> +   stored into *last_syntax_ptr.
 
>     The comment end is the last character of the comment rather than the
>     character just after the comment.
> @@ -2326,7 +2333,7 @@ static bool
>  forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	      EMACS_INT nesting, int style, int prev_syntax,
>  	      ptrdiff_t *charpos_ptr, ptrdiff_t *bytepos_ptr,
> -	      EMACS_INT *incomment_ptr)
> +	      EMACS_INT *incomment_ptr, int *last_syntax_ptr)
>  {
>    register int c, c1;
>    register enum syntaxcode code;
> @@ -2346,6 +2353,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>  	  *incomment_ptr = nesting;
>  	  *charpos_ptr = from;
>  	  *bytepos_ptr = from_byte;
> +          *last_syntax_ptr = syntax;
>  	  return 0;
>  	}
>        c = FETCH_CHAR_AS_MULTIBYTE (from_byte);
> @@ -2415,6 +2423,7 @@ forw_comment (ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t stop,
>      }
>    *charpos_ptr = from;
>    *bytepos_ptr = from_byte;
> +  *last_syntax_ptr = syntax;
>    return 1;
>  }
 
> @@ -2436,6 +2445,7 @@ between them, return t; otherwise return nil.  */)
>    EMACS_INT count1;
>    ptrdiff_t out_charpos, out_bytepos;
>    EMACS_INT dummy;
> +  int dummy2;
 
>    CHECK_NUMBER (count);
>    count1 = XINT (count);
> @@ -2499,7 +2509,7 @@ between them, return t; otherwise return nil.  */)
>  	}
>        /* We're at the start of a comment.  */
>        found = forw_comment (from, from_byte, stop, comnested, comstyle, 0,
> -			    &out_charpos, &out_bytepos, &dummy);
> +			    &out_charpos, &out_bytepos, &dummy, &dummy2);
>        from = out_charpos; from_byte = out_bytepos;
>        if (!found)
>  	{
> @@ -2659,6 +2669,7 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
>    ptrdiff_t from_byte;
>    ptrdiff_t out_bytepos, out_charpos;
>    EMACS_INT dummy;
> +  int dummy2;
>    bool multibyte_symbol_p = sexpflag && multibyte_syntax_as_symbol;
 
>    if (depth > 0) min_depth = 0;
> @@ -2755,7 +2766,8 @@ scan_lists (EMACS_INT from, EMACS_INT count, EMACS_INT depth, bool sexpflag)
>  	      UPDATE_SYNTAX_TABLE_FORWARD (from);
>  	      found = forw_comment (from, from_byte, stop,
>  				    comnested, comstyle, 0,
> -				    &out_charpos, &out_bytepos, &dummy);
> +				    &out_charpos, &out_bytepos, &dummy,
> +                                    &dummy2);
>  	      from = out_charpos, from_byte = out_bytepos;
>  	      if (!found)
>  		{
> @@ -3119,7 +3131,7 @@ the prefix syntax flag (p).  */)
>  }
>  
>  /* Parse forward from FROM / FROM_BYTE to END,
> -   assuming that FROM has state OLDSTATE (nil means FROM is start of function),
> +   assuming that FROM has state STATE (nil means FROM is start of function),
>     and return a description of the state of the parse at END.
>     If STOPBEFORE, stop at the start of an atom.
>     If COMMENTSTOP is 1, stop at the start of a comment.
> @@ -3127,12 +3139,11 @@ the prefix syntax flag (p).  */)
>     after the beginning of a string, or after the end of a string.  */
 
>  static void
> -scan_sexps_forward (struct lisp_parse_state *stateptr,
> +scan_sexps_forward (struct lisp_parse_state *state,
>  		    ptrdiff_t from, ptrdiff_t from_byte, ptrdiff_t end,
>  		    EMACS_INT targetdepth, bool stopbefore,
> -		    Lisp_Object oldstate, int commentstop)
> +		    int commentstop)
>  {
> -  struct lisp_parse_state state;
>    enum syntaxcode code;
>    int c1;
>    bool comnested;
> @@ -3148,7 +3159,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
>    Lisp_Object tem;
>    ptrdiff_t prev_from;		/* Keep one character before FROM.  */
>    ptrdiff_t prev_from_byte;
> -  int prev_from_syntax;
> +  int prev_from_syntax, prev_prev_from_syntax;
>    bool boundary_stop = commentstop == -1;
>    bool nofence;
>    bool found;
> @@ -3165,6 +3176,7 @@ scan_sexps_forward (struct lisp_parse_state *stateptr,
>  do { prev_from = from;				\
>       prev_from_byte = from_byte; 		\
>       temp = FETCH_CHAR_AS_MULTIBYTE (prev_from_byte);	\
> +     prev_prev_from_syntax = prev_from_syntax;  \
>       prev_from_syntax = SYNTAX_WITH_FLAGS (temp); \
>       INC_BOTH (from, from_byte);		\
>       if (from < end)				\
> @@ -3174,88 +3186,38 @@ do { prev_from = from;				\
>    immediate_quit = 1;
>    QUIT;
 
> -  if (NILP (oldstate))
> -    {
> -      depth = 0;
> -      state.instring = -1;
> -      state.incomment = 0;
> -      state.comstyle = 0;	/* comment style a by default.  */
> -      state.comstr_start = -1;	/* no comment/string seen.  */
> -    }
> -  else
> -    {
> -      tem = Fcar (oldstate);
> -      if (!NILP (tem))
> -	depth = XINT (tem);
> -      else
> -	depth = 0;
> -
> -      oldstate = Fcdr (oldstate);
> -      oldstate = Fcdr (oldstate);
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      /* Check whether we are inside string_fence-style string: */
> -      state.instring = (!NILP (tem)
> -			? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
> -			: -1);
> -
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      state.incomment = (!NILP (tem)
> -			 ? (INTEGERP (tem) ? XINT (tem) : -1)
> -			 : 0);
> -
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      start_quoted = !NILP (tem);
> +  depth = state->depth;
> +  start_quoted = state->quoted;
> +  prev_prev_from_syntax = Smax;
> +  prev_from_syntax = state->prev_syntax;
 
> -      /* if the eighth element of the list is nil, we are in comment
> -	 style a.  If it is non-nil, we are in comment style b */
> -      oldstate = Fcdr (oldstate);
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      state.comstyle = (NILP (tem)
> -			? 0
> -			: (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
> -			   ? XINT (tem)
> -			   : ST_COMMENT_STYLE));
> -
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      state.comstr_start =
> -	RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
> -      oldstate = Fcdr (oldstate);
> -      tem = Fcar (oldstate);
> -      while (!NILP (tem))		/* >= second enclosing sexps.  */
> -	{
> -	  Lisp_Object temhd = Fcar (tem);
> -	  if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
> -	    curlevel->last = XINT (temhd);
> -	  if (++curlevel == endlevel)
> -	    curlevel--; /* error ("Nesting too deep for parser"); */
> -	  curlevel->prev = -1;
> -	  curlevel->last = -1;
> -	  tem = Fcdr (tem);
> -	}
> +  tem = state->levelstarts;
> +  while (!NILP (tem))		/* >= second enclosing sexps.  */
> +    {
> +      Lisp_Object temhd = Fcar (tem);
> +      if (RANGED_INTEGERP (PTRDIFF_MIN, temhd, PTRDIFF_MAX))
> +        curlevel->last = XINT (temhd);
> +      if (++curlevel == endlevel)
> +        curlevel--; /* error ("Nesting too deep for parser"); */
> +      curlevel->prev = -1;
> +      curlevel->last = -1;
> +      tem = Fcdr (tem);
>      }
> -  state.quoted = 0;
> -  mindepth = depth;
> -
curlevel-> prev = -1;
curlevel-> last = -1;
 
> -  SETUP_SYNTAX_TABLE (prev_from, 1);
> -  temp = FETCH_CHAR (prev_from_byte);
> -  prev_from_syntax = SYNTAX_WITH_FLAGS (temp);
> -  UPDATE_SYNTAX_TABLE_FORWARD (from);
> +  state->quoted = 0;
> +  mindepth = depth;
> +
> +  SETUP_SYNTAX_TABLE (from, 1);
 
>    /* Enter the loop at a place appropriate for initial state.  */
 
> -  if (state.incomment)
> +  if (state->incomment)
>      goto startincomment;
> -  if (state.instring >= 0)
> +  if (state->instring >= 0)
>      {
> -      nofence = state.instring != ST_STRING_STYLE;
> +      nofence = state->instring != ST_STRING_STYLE;
>        if (start_quoted)
>  	goto startquotedinstring;
>        goto startinstring;
> @@ -3266,10 +3228,10 @@ do { prev_from = from;				\
>    while (from < end)
>      {
>        int syntax;
> -      INC_FROM;
> -      code = prev_from_syntax & 0xff;
 
>        if (from < end
> +          && (state->prev_comment_end == -1
> +              || prev_from >= state->prev_comment_end)
>  	  && SYNTAX_FLAGS_COMSTART_FIRST (prev_from_syntax)
>  	  && (c1 = FETCH_CHAR (from_byte),
>  	      syntax = SYNTAX_WITH_FLAGS (c1),
> @@ -3280,32 +3242,37 @@ do { prev_from = from;				\
>  	  /* Record the comment style we have entered so that only
>  	     the comment-end sequence of the same style actually
>  	     terminates the comment section.  */
> -	  state.comstyle
> +	  state->comstyle
>  	    = SYNTAX_FLAGS_COMMENT_STYLE (syntax, prev_from_syntax);
>  	  comnested = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax)
>  		       | SYNTAX_FLAGS_COMMENT_NESTED (syntax));
> -	  state.incomment = comnested ? 1 : -1;
> -	  state.comstr_start = prev_from;
> +	  state->incomment = comnested ? 1 : -1;
> +	  state->comstr_start = prev_from;
>  	  INC_FROM;
>  	  code = Scomment;
>  	}
> -      else if (code == Scomment_fence)
> -	{
> -	  /* Record the comment style we have entered so that only
> -	     the comment-end sequence of the same style actually
> -	     terminates the comment section.  */
> -	  state.comstyle = ST_COMMENT_STYLE;
> -	  state.incomment = -1;
> -	  state.comstr_start = prev_from;
> -	  code = Scomment;
> -	}
> -      else if (code == Scomment)
> -	{
> -	  state.comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
> -	  state.incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
> -			     1 : -1);
> -	  state.comstr_start = prev_from;
> -	}
> +      else
> +        {
> +          INC_FROM;
> +          code = prev_from_syntax & 0xff;
> +          if (code == Scomment_fence)
> +            {
> +              /* Record the comment style we have entered so that only
> +                 the comment-end sequence of the same style actually
> +                 terminates the comment section.  */
> +              state->comstyle = ST_COMMENT_STYLE;
> +              state->incomment = -1;
> +              state->comstr_start = prev_from;
> +              code = Scomment;
> +            }
> +          else if (code == Scomment)
> +            {
> +              state->comstyle = SYNTAX_FLAGS_COMMENT_STYLE (prev_from_syntax, 0);
> +              state->incomment = (SYNTAX_FLAGS_COMMENT_NESTED (prev_from_syntax) ?
> +                                 1 : -1);
> +              state->comstr_start = prev_from;
> +            }
> +        }
 
>        if (SYNTAX_FLAGS_PREFIX (prev_from_syntax))
>  	continue;
> @@ -3357,18 +3324,21 @@ do { prev_from = from;				\
>  	     middle of it.  We don't want to do that if we're just at the
>  	     beginning of the comment (think of (*) ... (*)).  */
>  	  found = forw_comment (from, from_byte, end,
> -				state.incomment, state.comstyle,
> -				(from == BEGV || from < state.comstr_start + 3)
> +				state->incomment, state->comstyle,
> +				(from == BEGV || from < state->comstr_start + 3)
>  				? 0 : prev_from_syntax,
> -				&out_charpos, &out_bytepos, &state.incomment);
> +				&out_charpos, &out_bytepos, &state->incomment,
> +                                &prev_from_syntax);
>  	  from = out_charpos; from_byte = out_bytepos;
> -	  /* Beware!  prev_from and friends are invalid now.
> -	     Luckily, the `done' doesn't use them and the INC_FROM
> -	     sets them to a sane value without looking at them. */
> +	  /* Beware!  prev_from and friends (except prev_from_syntax)
> +	     are invalid now.  Luckily, the `done' doesn't use them
> +	     and the INC_FROM sets them to a sane value without
> +	     looking at them. */
>  	  if (!found) goto done;
>  	  INC_FROM;
> -	  state.incomment = 0;
> -	  state.comstyle = 0;	/* reset the comment style */
> +	  state->incomment = 0;
> +	  state->comstyle = 0;	/* reset the comment style */
> +          state->prev_comment_end = from;
>  	  if (boundary_stop) goto done;
>  	  break;
 
> @@ -3396,16 +3366,16 @@ do { prev_from = from;				\
 
>  	case Sstring:
>  	case Sstring_fence:
> -	  state.comstr_start = from - 1;
> +	  state->comstr_start = from - 1;
>  	  if (stopbefore) goto stop;  /* this arg means stop at sexp start */
curlevel-> last = prev_from;
> -	  state.instring = (code == Sstring
> +	  state->instring = (code == Sstring
>  			    ? (FETCH_CHAR_AS_MULTIBYTE (prev_from_byte))
>  			    : ST_STRING_STYLE);
>  	  if (boundary_stop) goto done;
>  	startinstring:
>  	  {
> -	    nofence = state.instring != ST_STRING_STYLE;
> +	    nofence = state->instring != ST_STRING_STYLE;
 
>  	    while (1)
>  	      {
> @@ -3419,7 +3389,7 @@ do { prev_from = from;				\
>  		/* Check C_CODE here so that if the char has
>  		   a syntax-table property which says it is NOT
>  		   a string character, it does not end the string.  */
> -		if (nofence && c == state.instring && c_code == Sstring)
> +		if (nofence && c == state->instring && c_code == Sstring)
>  		  break;
 
>  		switch (c_code)
> @@ -3442,7 +3412,7 @@ do { prev_from = from;				\
>  	      }
>  	  }
>  	string_end:
> -	  state.instring = -1;
> +	  state->instring = -1;
curlevel-> prev = curlevel->last;
>  	  INC_FROM;
>  	  if (boundary_stop) goto done;
> @@ -3461,25 +3431,99 @@ do { prev_from = from;				\
>   stop:   /* Here if stopping before start of sexp. */
>    from = prev_from;    /* We have just fetched the char that starts it; */
>    from_byte = prev_from_byte;
> +  prev_from_syntax = prev_prev_from_syntax;
>    goto done; /* but return the position before it. */
 
>   endquoted:
> -  state.quoted = 1;
> +  state->quoted = 1;
>   done:
> -  state.depth = depth;
> -  state.mindepth = mindepth;
> -  state.thislevelstart = curlevel->prev;
> -  state.prevlevelstart
> +  state->depth = depth;
> +  state->mindepth = mindepth;
> +  state->thislevelstart = curlevel->prev;
> +  state->prevlevelstart
>      = (curlevel == levelstart) ? -1 : (curlevel - 1)->last;
> -  state.location = from;
> -  state.location_byte = from_byte;
> -  state.levelstarts = Qnil;
> +  state->location = from;
> +  state->location_byte = from_byte;
> +  state->levelstarts = Qnil;
>    while (curlevel > levelstart)
> -    state.levelstarts = Fcons (make_number ((--curlevel)->last),
> -			       state.levelstarts);
> +    state->levelstarts = Fcons (make_number ((--curlevel)->last),
> +                                state->levelstarts);
> +  state->prev_syntax = prev_from_syntax;
>    immediate_quit = 0;
> +}
> +
> +/* Convert a (lisp) parse state to the internal form used in
> +   scan_sexps_forward.  */
> +static void
> +internalize_parse_state (Lisp_Object external, struct lisp_parse_state *state)
> +{
> +  Lisp_Object tem;
> +
> +  if (NILP (external))
> +    {
> +      state->depth = 0;
> +      state->instring = -1;
> +      state->incomment = 0;
> +      state->quoted = 0;
> +      state->comstyle = 0;	/* comment style a by default.  */
> +      state->comstr_start = -1;	/* no comment/string seen.  */
> +      state->levelstarts = Qnil;
> +      state->prev_syntax = Smax;
> +      state->prev_comment_end = -1;
> +    }
> +  else
> +    {
> +      tem = Fcar (external);
> +      if (!NILP (tem))
> +	state->depth = XINT (tem);
> +      else
> +	state->depth = 0;
> +
> +      external = Fcdr (external);
> +      external = Fcdr (external);
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      /* Check whether we are inside string_fence-style string: */
> +      state->instring = (!NILP (tem)
> +                         ? (CHARACTERP (tem) ? XFASTINT (tem) : ST_STRING_STYLE)
> +                         : -1);
> +
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->incomment = (!NILP (tem)
> +                          ? (INTEGERP (tem) ? XINT (tem) : -1)
> +                          : 0);
 
> -  *stateptr = state;
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->quoted = !NILP (tem);
> +
> +      /* if the eighth element of the list is nil, we are in comment
> +	 style a.  If it is non-nil, we are in comment style b */
> +      external = Fcdr (external);
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->comstyle = (NILP (tem)
> +                         ? 0
> +                         : (RANGED_INTEGERP (0, tem, ST_COMMENT_STYLE)
> +                            ? XINT (tem)
> +                            : ST_COMMENT_STYLE));
> +
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->comstr_start =
> +	RANGED_INTEGERP (PTRDIFF_MIN, tem, PTRDIFF_MAX) ? XINT (tem) : -1;
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->levelstarts = tem;
> +
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->prev_syntax = NILP (tem) ? Smax : XINT (tem);
> +      external = Fcdr (external);
> +      tem = Fcar (external);
> +      state->prev_comment_end = NILP (tem) ? -1 : XINT (tem);
> +    }
>  }
 
>  DEFUN ("parse-partial-sexp", Fparse_partial_sexp, Sparse_partial_sexp, 2, 6, 0,
> @@ -3488,6 +3532,7 @@ Parsing stops at TO or when certain criteria are met;
>   point is set to where parsing stops.
>  If fifth arg OLDSTATE is omitted or nil,
>   parsing assumes that FROM is the beginning of a function.
> +
>  Value is a list of elements describing final state of parsing:
>   0. depth in parens.
>   1. character address of start of innermost containing list; nil if none.
> @@ -3501,16 +3546,20 @@ Value is a list of elements describing final state of parsing:
>   6. the minimum paren-depth encountered during this scan.
>   7. style of comment, if any.
>   8. character address of start of comment or string; nil if not in one.
> - 9. Intermediate data for continuation of parsing (subject to change).
> + 9. List of positions of currently open parens, outermost first.
> +10. Syntax of last character scanned, or nil if no scanning has happened.
> +11. Position after end of previous comment scanned, or nil.
> +12..... Possible further internal information used by `parse-partial-sexp'.
> +
>  If third arg TARGETDEPTH is non-nil, parsing stops if the depth
>  in parentheses becomes equal to TARGETDEPTH.
> -Fourth arg STOPBEFORE non-nil means stop when come to
> +Fourth arg STOPBEFORE non-nil means stop when we come to
>   any character that starts a sexp.
>  Fifth arg OLDSTATE is a list like what this function returns.
>   It is used to initialize the state of the parse.  Elements number 1, 2, 6
>   are ignored.
> -Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
> - If it is symbol `syntax-table', stop after the start of a comment or a
> +Sixth arg COMMENTSTOP non-nil means stop after the start of a comment.
> + If it is the symbol `syntax-table', stop after the start of a comment or a
>   string, or after end of a comment or a string.  */)
>    (Lisp_Object from, Lisp_Object to, Lisp_Object targetdepth,
>     Lisp_Object stopbefore, Lisp_Object oldstate, Lisp_Object commentstop)
> @@ -3527,15 +3576,17 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
>      target = TYPE_MINIMUM (EMACS_INT);	/* We won't reach this depth.  */
 
>    validate_region (&from, &to);
> +  internalize_parse_state (oldstate, &state);
>    scan_sexps_forward (&state, XINT (from), CHAR_TO_BYTE (XINT (from)),
>  		      XINT (to),
> -		      target, !NILP (stopbefore), oldstate,
> +		      target, !NILP (stopbefore),
>  		      (NILP (commentstop)
>  		       ? 0 : (EQ (commentstop, Qsyntax_table) ? -1 : 1)));
 
>    SET_PT_BOTH (state.location, state.location_byte);
 
> -  return Fcons (make_number (state.depth),
> +  return
> +    Fcons (make_number (state.depth),
>  	   Fcons (state.prevlevelstart < 0
>  		  ? Qnil : make_number (state.prevlevelstart),
>  	     Fcons (state.thislevelstart < 0
> @@ -3553,11 +3604,18 @@ Sixth arg COMMENTSTOP non-nil means stop at the start of a comment.
>  				  ? Qsyntax_table
>  				  : make_number (state.comstyle))
>  			       : Qnil),
> -			      Fcons (((state.incomment
> -				       || (state.instring >= 0))
> -				      ? make_number (state.comstr_start)
> -				      : Qnil),
> -				     Fcons (state.levelstarts, Qnil))))))))));
> +		         Fcons (((state.incomment
> +                                  || (state.instring >= 0))
> +                                 ? make_number (state.comstr_start)
> +                                 : Qnil),
> +			   Fcons (state.levelstarts,
> +                             Fcons (state.prev_syntax == Smax
> +                                    ? Qnil
> +                                    : make_number (state.prev_syntax),
> +                               Fcons (state.prev_comment_end == -1
> +                                      ? Qnil
> +                                      : make_number (state.prev_comment_end),
> +                                      Qnil))))))))))));
>  }
>  
>  void

>> Stefan

> -- 
> Alan Mackenzie (Nuremberg, Germany).