Subject: [PATCH v1] calc: Allow strings with higher character codes
From: "Jacob S. Gordon" <[email-masked]>
To: [email-masked]
Date: Tue, 20 May 2025 21:34:02 -0400
List-Id: <debbugs-submit.debbugs.gnu.org>

Tags: patch

Hello all,

Please find below a feature proposal for strings in `calc', and a first
draft of a patch attached to this message.

Motivation
==========

Suppose you're working with Unicode code points in `calc', and you end
up with the following vector on the stack. You'd like to know what a
string composed of these character codes would look like, so you toggle
`calc-display-strings' (`d "') and … nothing happens.

,----
| 1:  [383, 117, 99, 99, 101, 383, 115]
`----

Later, in an `org-mode' file you have the following table with a list of
dates in the first column. Since [formulas] can be any algebraic
expression understood by `calc', and `calc' [understands dates], you try
to insert a Unicode character for rows where the first column is in the
past. When you evaluate the formula (`C-c C-c' on the `#+TBLFM:' line)
`calc' stops short of displaying the string.

,----
| | Date             | Past?           |
| |------------------+-----------------|
| | [2025-05-01 Thu] | string([10003]) |
| | [2026-05-01 Fri] |                 |
| #+TBLFM: $2 = if($1 < now(), string("✓"), string(""))
`----

Both of these problems are due to the fact that some or all of the
character codes are outside the `Latin-1' (8-bit) range. If we replace
this hard-coded limitation with a custom variable and increase its
value, both of these use-cases can be supported.

,----
| 1:  "ſucceſs"
`----

,----
| | Date             | Past? |
| |------------------+-------|
| | [2025-05-01 Thu] | ✓     |
| | [2026-05-01 Fri] |       |
| #+TBLFM: $2 = if($1 < now(), string("✓"), string(""))
`----

The alternative is that the user has to exit `calc' (or its syntax) and
dip into `Lisp':

,----
| (concat '(383 117 99 99 101 383 115))
`----

,----
| | Date             | Past? |
| |------------------+-------|
| | [2025-05-01 Thu] | ✓     |
| | [2026-05-01 Fri] |       |
| #+TBLFM: $2 = '(if (time-less-p (org-read-date t t $1) (current-time)) "✓" "")
`----

[formulas] <https://orgmode.org/manual/Formula-syntax-for-Calc.html>

[understands dates]
<https://www.gnu.org/software/emacs/manual/html_node/calc/Date-Forms.html>

Proposal & Impact
=================

The attached patch introduces a custom variable
`calc-string-maximum-character' (optimistically versioned for `31.1'),
which replaces a hard-coded maximum in the function
`math-vector-is-string'. This variable defaults to `0xFF' in order to
preserve the current behaviour, but otherwise can be any character up to
`(max-char)'. Since the vector contents are passed to
`math-vector-to-string', the Unicode-aware `concat' has no problem with
the higher characters:

,----
| (defun math-vector-to-string (a &optional quoted)
|   (setq a (concat (mapcar (lambda (x) (if (consp x) (nth 1 x) x))
|                           (cdr a))))
|   […])
`----

Here are the outstanding issues I've identified for discussion:

1. Since users can blow past the variable type and set
   `calc-string-maximum-character' to /anything/, I'm not sure the
   patch's error handling is enough. If a hapless user sets it to
   something invalid like a string (`"invalid"', let's say), then with
   the current patch they'll encounter at least two kinds of errors:

   a) With the following vector on the stack, executing
      `calc-display-strings' (`d "') will display `Wrong type argument:
      number-or-marker-p, "invalid"' in the minibuffer, /and/ enter a
      string display mode where the vector isn't rendered as seen in the
      second block below.

      ,----
      | 1:  [0, 1, 2]
      `----

      ,----
      | 1:  .
      `----

      Only executing `calc-display-strings' (`d "') again will toggle
      the display mode and show the original vector. This is a bad
      experience for the user, and should be mitigated by raising an
      error in `calc-display-strings' before the display mode is
      toggled.

   b) If a user tries to enter a string algebraically with
      `calc-algebraic-entry' (`''), say `string("abc")', the same
      message from the first error will appear in the minibuffer, but
      the string is not added to the stack. This is slightly cryptic,
      but not as bad an experience as the first error.

2. With a higher value of `calc-string-maximum-character', the displayed
   string could contain right-to-left or a bidirectional mixture of
   characters that could conceivably interfere with the `calc' alignment
   functions `calc-left-justify' (`d <'), `calc-center-justify' (`d ='),
   and `calc-right-justify' (`d >'). Toggling the display of the
   following vectors reveals a misalignment of the fully Arabic string
   under center justification, and misalignment of the full- and
   mixed-Arabic strings under right justification. None of these contain
   any of the funky bidirectional Unicode markers so I'm not sure if
   there's other problems lurking.

   ,----
   | 3:  [108, 101, 102, 116, 45, 116, 111, 45, 114, 105, 103, 104, 116]
   | 2:  [1605, 1606, 32, 1575, 1604, 1610, 1605, 1610, 1606, 32, 1573, 1604, 1609, 32, 1575, 1604, 1610, 1587, 1575, 1585]
   | 1:  [108, 101, 102, 116, 45, 1610, 1605, 1610, 1606]
   `----

   ,----
   | 3:  "left-to-right"
   | 2:  "من اليمين إلى اليسار"
   | 1:  "left-يمين"
   `----

   ,----
   | 3:                       "left-to-right"
   | 2:                   "من اليمين إلى اليسار"
   | 1:                         "left-يمين"
   `----

   ,----
   | 3:                                               "left-to-right"
   | 2:                                        "من اليمين إلى اليسار"
   | 1:                                                   "left-يمين"
   `----

   Also, combining diacritical marks appear as separate characters, but
   I'm not sure if this is the expected behaviour and/or related to my
   configuration.

   ,----
   | 1.  [117, 776]
   `----

   ,----
   | 1:  "ü"
   `----

3. I haven't found any internal references to `math-vector-is-string'
   that look like they could conflict with this change
   (`math-format-flat-expr-fancy', `math-compose-expr',
   `calc-kbd-query'). Existing references are mostly related to
   displaying strings from vectors, `string' or `bstring' objects, and
   composite objects involving vectors or strings, but I could use an
   extra set of eyes to confirm. Since `org-mode' uses `calc'
   expressions in tables, I might need to get their concurrence with the
   change. I'm unaware of any third-party dependencies on this function.

4. For unit tests, are there any naming conventions I should follow? I
   just stuck all of the tests in one place for `math-vector-is-string'.

Thanks for your consideration!

--
Jacob S. Gordon
[email-masked]

=========================

Please avoid sending me HTML emails and MS Office documents.
https://useplaintext.email/#etiquette
https://www.gnu.org/philosophy/no-word-attachments.html

In GNU Emacs 30.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.49,
cairo version 1.18.4)
System Description: Arch Linux

Configured using:
 'configure --with-pgtk --sysconfdir=/etc --prefix=/usr
 --libexecdir=/usr/lib --localstatedir=/var --disable-build-details
 --with-cairo --with-harfbuzz --with-libsystemd --with-modules
 --with-native-compilation=aot --with-tree-sitter 'CFLAGS=-march=x86-64
 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=3
 -Wformat -Werror=format-security -fstack-clash-protection
 -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g
 -ffile-prefix-map=/build/emacs/src=/usr/src/debug/emacs -flto=auto'
 'LDFLAGS=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro
 -Wl,-z,now -Wl,-z,pack-relative-relocs -flto=auto'
 'CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions
 -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer
 -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g
 -ffile-prefix-map=/build/emacs/src=/usr/src/debug/emacs -flto=auto''

From ffc1813155f8e23db271c01c249c163912ecf377 Mon Sep 17 00:00:00 2001
From: "Jacob S. Gordon" <[email-masked]>
Date: Mon, 19 May 2025 15:05:37 -0400
Subject: [PATCH v1] calc: Allow strings with higher character codes

The current behavior of the functions 'calc-display-strings', 'strings',
and 'bstrings' is to skip any vector containing integers outside the
Latin-1 range (0x00-0xFF).  We introduce a custom variable
'calc-string-maximum-character' to replace this hard-coded maximum, and
to allow vectors containing higher character codes to be displayed as
strings.  The default value of 0xFF preserves the existing behavior.

* lisp/calc/calc.el (calc-string-maximum-character):

Add custom variable 'calc-string-maximum-character'.

* lisp/calc/calccomp.el (math-vector-is-string):

Replace hard-coded maximum with 'calc-string-maximum-character', and the
'natnump' assertion with 'characterp'.  The latter guards against the
maximum being larger than '(max-char)', but not on invalid types of the
maximum such as strings.

* test/lisp/calc/calc-tests.el (calc-math-vector-is-string):

Add tests for 'math-vector-is-string' using different values of
'calc-string-maximum-character'.

* doc/misc/calc.texi (Quick Calculator, Strings, Customizing Calc):

Add variable definition for 'calc-string-maximum-character' and
reference thereof when discussing 'calc-display-strings'.  Generalize a
comment about string display and availability of 8-bit fonts.
---
 doc/misc/calc.texi           | 56 +++++++++++++++++++++++-------
 etc/NEWS                     | 13 ++++++-
 lisp/calc/calc.el            | 31 +++++++++++++++++
 lisp/calc/calccomp.el        | 15 +++++---
 test/lisp/calc/calc-tests.el | 67 ++++++++++++++++++++++++++++++++++++
 5 files changed, 164 insertions(+), 18 deletions(-)

diff --git a/doc/misc/calc.texi b/doc/misc/calc.texi
index 61466b55201..eda442ecb38 100644
--- a/doc/misc/calc.texi
+++ b/doc/misc/calc.texi
@@ -10179,7 +10179,7 @@ Quick Calculator
 is displayed only according to the current mode settings.  But
 running Quick Calc again and entering @samp{120} will produce the
 result @samp{120 (16#78, 8#170, x)} which shows the number in its
-decimal, hexadecimal, octal, and ASCII forms.
+decimal, hexadecimal, octal, and character forms.
 
 Please note that the Quick Calculator is not any faster at loading
 or computing the answer than the full Calculator; the name ``quick''
@@ -10836,11 +10836,11 @@ Strings
 @cindex Strings
 @cindex Character strings
 Character strings are not a special data type in the Calculator.
-Rather, a string is represented simply as a vector all of whose
-elements are integers in the range 0 to 255 (ASCII codes).  You can
-enter a string at any time by pressing the @kbd{"} key.  Quotation
-marks and backslashes are written @samp{\"} and @samp{\\}, respectively,
-inside strings.  Other notations introduced by backslashes are:
+Rather, a string is represented simply as a vector all of whose elements
+are integers in the Latin-1 range 0 to 255.  You can enter a string at
+any time by pressing the @kbd{"} key.  Quotation marks and backslashes
+are written @samp{\"} and @samp{\\}, respectively, inside strings.
+Other notations introduced by backslashes are:
 
 @example
 @group
@@ -10857,21 +10857,24 @@ Strings
 
 @noindent
 Finally, a backslash followed by three octal digits produces any
-character from its ASCII code.
+character from its code.
 
 @kindex d "
 @pindex calc-display-strings
 Strings are normally displayed in vector-of-integers form.  The
 @w{@kbd{d "}} (@code{calc-display-strings}) command toggles a mode in
 which any vectors of small integers are displayed as quoted strings
-instead.
+instead.  The display of strings containing higher character codes can
+be enabled by increasing the custom variable
+@code{calc-string-maximum-character} (@pxref{Customizing Calc}).
 
 The backslash notations shown above are also used for displaying
-strings.  Characters 128 and above are not translated by Calc; unless
-you have an Emacs modified for 8-bit fonts, these will show up in
-backslash-octal-digits notation.  For characters below 32, and
-for character 127, Calc uses the backslash-letter combination if
-there is one, or otherwise uses a @samp{\^} sequence.
+strings.  For ASCII control characters (below 32), and for the
+@code{DEL} character (127), Calc uses the backslash-letter combination
+if there is one, or otherwise uses a @samp{\^} sequence.  Control
+characters above 127 are not translated by Calc, and will show up in
+backslash-octal-digits notation.  The display of higher character codes
+will depend on your display settings and system font coverage.
 
 The only Calc feature that uses strings is @dfn{compositions};
 @pxref{Compositions}.  Strings also provide a convenient
@@ -35684,6 +35687,33 @@ Customizing Calc
 The default value of @code{calc-gregorian-switch} is @code{nil}.
 @end defvar
 
+@defvar calc-string-maximum-character
+@xref{Strings}.@*
+
+The variable @code{calc-string-maximum-character} is the maximum value
+of a vector's elements for @code{calc-display-strings}, @code{string},
+and @code{bstring} to display the vector as a string.  This maximum
+@emph{must} represent a character, i.e. it's a non-negative integer less
+than or equal to @code{(max-char)} or @code{0x3FFFFF}.  Any negative
+value effectively disables the display of strings, and for values larger
+than @code{0x3FFFFF} the display acts as if the maximum were
+@code{0x3FFFFF}.  Some natural choices (and their resulting ranges) are:
+
+@itemize
+@item
+@code{0x7F} or 127 (ASCII),
+@item
+@code{0xFF} or 255 (Latin-1, the default),
+@item
+@code{0x10FFFF} (Unicode),
+@item
+@code{0x3FFFFF} (Emacs).
+@end itemize
+
+The default value of @code{calc-string-maximum-character} is @code{0xFF}
+or 255.
+@end defvar
+
 @node Reporting Bugs
 @appendix Reporting Bugs
 
diff --git a/etc/NEWS b/etc/NEWS
index 4ad6c48d78a..58d0c18b2ba 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -2106,7 +2106,18 @@ is bound to 'C-l' in the calendar buffer.
 You can now use the mouse wheel to scroll the calendar by 3 months.
 With the shift modifier, it scrolls by one month.  With the meta
 modifier, it scrolls by year.
-
+
+** Calc
+
+*** New user option 'calc-string-maximum-character'.
+
+Previously, the 'calc-display-strings', 'string', and 'bstring'
+functions only considered integer vectors whose elements are all in the
+Latin-1 range 0-255.  This hard-coded maximum is replaced by
+'calc-string-maximum-character', and setting it to a higher value allows
+the display of matching vectors as Unicode strings. The default value is
+'0xFF' or '255' to preserve the existing behavior.
+
 * New Modes and Packages in Emacs 31.1
 
 ** New minor mode 'delete-trailing-whitespace-mode'.
diff --git a/lisp/calc/calc.el b/lisp/calc/calc.el
index a7bd671998e..a350419b320 100644
--- a/lisp/calc/calc.el
+++ b/lisp/calc/calc.el
@@ -628,6 +628,37 @@ calc-infinite-mode
 (defcalcmodevar calc-display-strings nil
   "If non-nil, display vectors of byte-sized integers as strings.")
 
+(defcustom calc-string-maximum-character #xFF
+  "Maximum value of vector contents to be displayed as a string.
+
+If a vector consists of characters up to this maximum value, the
+function `calc-display-strings' will toggle displaying the vector as a
+string.  This maximum value must represent a character (see `characterp').
+Some natural choices (and their resulting ranges) are:
+
+- `0x7F'     (`ASCII'),
+- `0xFF'     (`Latin-1', the default),
+- `0x10FFFF' (`Unicode'),
+- `0x3FFFFF' (`Emacs').
+
+Characters for low control codes are either caret or backslash escaped,
+while others without a glyph are displayed in backslash-octal notation.
+The display of strings containing higher character codes will depend on
+your display settings and system font coverage.
+
+See the following for further information:
+
+- info node `(calc)Strings',
+- info node `(elisp)Text Representations',
+- info node `(emacs)Text Display'."
+  :version "31.1"
+  :type '(choice (restricted-sexp :tag "Character Code"
+                                  :match-alternatives (characterp))
+                 (const :tag "ASCII"   #x7F)
+                 (const :tag "Latin-1" #xFF)
+                 (const :tag "Unicode" #x10FFFF)
+                 (const :tag "Emacs"   #x3FFFFF)))
+
 (defcalcmodevar calc-matrix-just 'center
   "If nil, vector elements are left-justified.
 If `right', vector elements are right-justified.
diff --git a/lisp/calc/calccomp.el b/lisp/calc/calccomp.el
index cc27e6c2025..7e1f8378d80 100644
--- a/lisp/calc/calccomp.el
+++ b/lisp/calc/calccomp.el
@@ -907,13 +907,20 @@ math-compose-rows
 		(concat " " math-comp-right-bracket)))))
 
 (defun math-vector-is-string (a)
+  "Return t if A can be displayed as a string, and nil otherwise.
+
+Elements of A must either be a character (see `characterp') or a complex
+number with only a real character part, each with a value less than or
+equal to the custom variable `calc-string-maximum-character'."
   (while (and (setq a (cdr a))
-	      (or (and (natnump (car a))
-		       (<= (car a) 255))
+	      (or (and (characterp (car a))
+		       (<= (car a)
+			   calc-string-maximum-character))
 		  (and (eq (car-safe (car a)) 'cplx)
-		       (natnump (nth 1 (car a)))
+		       (characterp (nth 1 (car a)))
 		       (eq (nth 2 (car a)) 0)
-		       (<= (nth 1 (car a)) 255)))))
+		       (<= (nth 1 (car a))
+			   calc-string-maximum-character)))))
   (null a))
 
 (defconst math-vector-to-string-chars '( ( ?\" . "\\\"" )
diff --git a/test/lisp/calc/calc-tests.el b/test/lisp/calc/calc-tests.el
index 42eb6077b04..2fd6a6be45e 100644
--- a/test/lisp/calc/calc-tests.el
+++ b/test/lisp/calc/calc-tests.el
@@ -879,5 +879,72 @@ calc-math-read-preprocess-string
   (should-error (math-read-preprocess-string nil))
   (should-error (math-read-preprocess-string 42)))
 
+(ert-deftest calc-math-vector-is-string ()
+  "Test `math-vector-is-string' with varying `calc-string-maximum-character'.
+
+All tests operate on both an integer vector and the corresponding
+complex vector.  The sets covered are:
+
+1. `calc-string-maximum-character' is a valid character. The last case
+with `0x3FFFFF' is borderline, as integers above it will not make it
+past the `characterp' test.
+2. `calc-string-maximum-character' is negative, so the test always fails.
+3. `calc-string-maximum-character' is above `(max-char)', so only the
+first `characterp' test is active.
+4. `calc-string-maximum-character' has an invalid type, which triggers
+an error in the comparison."
+  (cl-flet* ((make-vec (lambda (contents) (append (list 'vec) contents)))
+             (make-cplx (lambda (x) (list 'cplx x 0)))
+             (make-cplx-vec (lambda (contents)
+                              (make-vec (mapcar #'make-cplx contents)))))
+    ;; 1: calc-string-maximum-character is a valid character
+    (dolist (maxchar '(#x7F #xFF #x10FFFF #x3FFFFD #x3FFFFF))
+      (let* ((calc-string-maximum-character maxchar)
+             (small-chars (number-sequence (- maxchar 2) maxchar))
+             (large-chars (number-sequence maxchar (+ maxchar 2)))
+             (small-real-vec (make-vec small-chars))
+             (large-real-vec (make-vec large-chars))
+             (small-cplx-vec (make-cplx-vec small-chars))
+             (large-cplx-vec (make-cplx-vec large-chars)))
+        (should (math-vector-is-string small-real-vec))
+        (should-not (math-vector-is-string large-real-vec))
+        (should (math-vector-is-string small-cplx-vec))
+        (should-not (math-vector-is-string large-cplx-vec))))
+    ;; 2: calc-string-maximum-character is negative
+    (let* ((maxchar -1)
+           (calc-string-maximum-character maxchar)
+           (valid-contents (number-sequence 0 2))
+           (invalid-contents (number-sequence (- maxchar 2) maxchar))
+           (valid-real-vec (make-vec valid-contents))
+           (invalid-real-vec (make-vec invalid-contents))
+           (valid-cplx-vec (make-cplx-vec valid-contents))
+           (invalid-cplx-vec (make-cplx-vec invalid-contents)))
+      (should-not (math-vector-is-string valid-real-vec))
+      (should-not (math-vector-is-string invalid-real-vec))
+      (should-not (math-vector-is-string valid-cplx-vec))
+      (should-not (math-vector-is-string invalid-cplx-vec)))
+    ;; 3: calc-string-maximum-character is larger than (max-char)
+    (let* ((maxchar (+ (max-char) 3))
+           (calc-string-maximum-character maxchar)
+           (valid-chars (number-sequence (- (max-char) 2) (max-char)))
+           (invalid-chars (number-sequence (1+ (max-char)) maxchar))
+           (valid-real-vec (make-vec valid-chars))
+           (invalid-real-vec (make-vec invalid-chars))
+           (valid-cplx-vec (make-cplx-vec valid-chars))
+           (invalid-cplx-vec (make-cplx-vec invalid-chars)))
+      (should (math-vector-is-string valid-real-vec))
+      (should-not (math-vector-is-string invalid-real-vec))
+      (should (math-vector-is-string valid-cplx-vec))
+      (should-not (math-vector-is-string invalid-cplx-vec)))
+    ;; 4: calc-string-maximum-character has the wrong type
+    (let* ((calc-string-maximum-character "wrong type")
+           (contents (number-sequence 0 2))
+           (real-vec (make-vec contents))
+           (cplx-vec (make-cplx-vec contents)))
+      (should-error (math-vector-is-string real-vec)
+                    :type 'wrong-type-argument)
+      (should-error (math-vector-is-string cplx-vec)
+                    :type 'wrong-type-argument))))
+
 (provide 'calc-tests)
 ;;; calc-tests.el ends here

base-commit: 87fa5f565d70c514bd47b59bb0ef0cba8750e983
-- 
Jacob S. Gordon
[email-masked]

=========================

Please avoid sending me HTML emails and MS Office documents.
https://useplaintext.email/#etiquette
https://www.gnu.org/philosophy/no-word-attachments.html