From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 03:14:56 2020 Received: (at submit) by debbugs.gnu.org; 24 Jun 2020 07:14:56 +0000 Received: from localhost ([127.0.0.1]:37659 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnzcV-0006Qz-Ov for submit@debbugs.gnu.org; Wed, 24 Jun 2020 03:14:55 -0400 Received: from lists.gnu.org ([209.51.188.17]:40638) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnzcR-0006Qp-3O for submit@debbugs.gnu.org; Wed, 24 Jun 2020 03:14:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56062) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jnzcQ-00068X-UU for bug-gnu-emacs@gnu.org; Wed, 24 Jun 2020 03:14:50 -0400 Received: from mail-wr1-x432.google.com ([2a00:1450:4864:20::432]:33448) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jnzcP-0007yo-CO for bug-gnu-emacs@gnu.org; Wed, 24 Jun 2020 03:14:50 -0400 Received: by mail-wr1-x432.google.com with SMTP id j94so1188392wrj.0 for ; Wed, 24 Jun 2020 00:14:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version; bh=dTreTf9207uAIgiJUXZPKkSUMZNWcaOu044j+VmwQw8=; b=Bg3iGTFrlYNaooz5FDA3p71FHlbZywdcdSql2JCw8dztLflltpFbYvCeXTo0PoLpt0 A/R1fMyGBaAa7uzx2GyfzUxRXbym6lJtrxhw/5BIPmZ3UWeko3MgejIi7uUsiuS3Jgb6 a53t7q2nAUlxzBcytPWdfU5D4loUgtycNBeIPOya8H0E6qZ6TszxFPCIQW3U51OKctzh Lupq5oPxsOseAJC2MHpBVwXl/BrER/mIoJgu8RmKrlRHMUYaYsiDGG4p1y7O+d38cUYv F1VIh57PQ82YljRe4LcWAFF02bTo49z9rxK7YZGtUeNLjdiyQT5ISXY3iB8iXPN1t8nv qOlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version; bh=dTreTf9207uAIgiJUXZPKkSUMZNWcaOu044j+VmwQw8=; b=tHrQk9Y83FSztpOMjdEFssaIOEWqXivjQgYpccLLPeuvQotLHPMciV5Yyi8ewrNZwp /GfObUFIgnYEmm5eCjNKnKwvcmB4TxiMxZVDCry1KGZo/oaVeoFhzg/Cwq0o1IQxAgQZ VTVS6oUsKicQaMmH0g7ljekLYZfho0UkuIkxNTrt+gNn57GQWZZG3g72CMF6maOQjfEf ft1eAsSb/C2SKm5oo2vISEDxMF3ixJ4VR6xofzIIK6VDuk0WKFXIrYO5vTptskP+4XjP 4en5evJ0o/2BOBlD5tOQ1f6XyWnIeL2lk3UwZQ1CDiD64ZBJZIyCekJ9hV9Y+b7kabT7 PfRQ== X-Gm-Message-State: AOAM532xyRgLrtVb4zIrrfA9x6lAA+HqG3Q1DMmIFwX1OzH9kTeJ/7lc 6kWUIgch5uNM9uiZlk4KZDEnlF0X X-Google-Smtp-Source: ABdhPJxm+HBGIBsnVhiKnbzRit1yHWEFuin9CjyG50iUO0Fd0KVxhJyvj8HyA/3SC+58HNiR7Ilahg== X-Received: by 2002:a05:6000:1c7:: with SMTP id t7mr29241311wrx.14.1592982886634; Wed, 24 Jun 2020 00:14:46 -0700 (PDT) Received: from kyoncho-H87-D3H ([130.204.243.78]) by smtp.gmail.com with ESMTPSA id u9sm6918443wme.16.2020.06.24.00.14.45 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jun 2020 00:14:45 -0700 (PDT) From: Ivan Yonchovski To: bug-gnu-emacs@gnu.org Subject: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop Date: Wed, 24 Jun 2020 10:14:44 +0300 Message-ID: <875zbh0wbf.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2a00:1450:4864:20::432; envelope-from=yyoncho@gmail.com; helo=mail-wr1-x432.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) This is needed for the cases when the module is listening for extenal events and wants to call back emacs to process them. ATM this is kind of possible by using signals on linux and using WM_INPUTLANGCHANGE as described in https://nullprogram.com/blog/2017/02/14/ but this looks more a hack for a missing feature. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 10:45:07 2020 Received: (at 42028) by debbugs.gnu.org; 24 Jun 2020 14:45:07 +0000 Received: from localhost ([127.0.0.1]:38646 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo6eB-0004ou-CW for submit@debbugs.gnu.org; Wed, 24 Jun 2020 10:45:07 -0400 Received: from eggs.gnu.org ([209.51.188.92]:56192) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo6e8-0004o5-TH for 42028@debbugs.gnu.org; Wed, 24 Jun 2020 10:45:05 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:50929) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jo6e3-0003Cz-IK; Wed, 24 Jun 2020 10:44:59 -0400 Received: from [176.228.60.248] (port=2205 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jo6e2-0005zc-VB; Wed, 24 Jun 2020 10:44:59 -0400 Date: Wed, 24 Jun 2020 17:44:42 +0300 Message-Id: <83bll8ttet.fsf@gnu.org> From: Eli Zaretskii To: Ivan Yonchovski In-Reply-To: <875zbh0wbf.fsf@gmail.com> (message from Ivan Yonchovski on Wed, 24 Jun 2020 10:14:44 +0300) Subject: Re: bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop References: <875zbh0wbf.fsf@gmail.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 42028 Cc: 42028@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Ivan Yonchovski > Date: Wed, 24 Jun 2020 10:14:44 +0300 > > This is needed for the cases when the module is listening for extenal > events and wants to call back emacs to process them. ATM this is kind of > possible by using signals on linux and using WM_INPUTLANGCHANGE as > described in https://nullprogram.com/blog/2017/02/14/ but this looks > more a hack for a missing feature. It is almost trivial to let modules insert events into the event queue. The problem is that I expect a module that inserts such events to want to be called to process those events as well. Is that expectation correct? If it is, then we need to think about extending the mechanism that calls event handlers, not just about exposing the likes of kbd_buffer_store_event to modules. We should also consider how a module could define its own events. IOW, this calls for a slightly more detailed specification, before someone could sit down and code the stuff. I suggest to take a look at how existing events are handled, and propose such a detailed specification. Thanks. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 12:17:51 2020 Received: (at 42028) by debbugs.gnu.org; 24 Jun 2020 16:17:51 +0000 Received: from localhost ([127.0.0.1]:38747 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo85v-000744-0G for submit@debbugs.gnu.org; Wed, 24 Jun 2020 12:17:51 -0400 Received: from mail-lf1-f48.google.com ([209.85.167.48]:37684) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo85r-00073n-L0 for 42028@debbugs.gnu.org; Wed, 24 Jun 2020 12:17:49 -0400 Received: by mail-lf1-f48.google.com with SMTP id k15so1594206lfc.4 for <42028@debbugs.gnu.org>; Wed, 24 Jun 2020 09:17:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=JDPfoPzFYNlHD1wRkvvuJy2Bjmq5s9AdMrdSZF0YbfI=; b=RpFmfkr2fL/l8rirYd/LiGqNaummp1qR4+iPy+YNaHd5M14fLNdA9z+P3YQdCVRVjr tJvQkPc9gFW0dQakyJSZdiwNVjfWb5Vay8qtcaH48TEQK+aNx81DMS3VeG2xKd/GQIqf wUz/f76E87sFnW87dYNqochmWEZ+Gi0tx1LvVSnbqDNbKv1SXUIy6qN66B4lDH6H0rC1 uYsfO0KrFjXqrpWnSBFJNdD6qgWSFZRQabPLISmzw2KvBCwyNTRSLXcJUrcEgWjh9j2N izTfClY6Ho8EueXHHcQFlSQe70Pz/rY+VhEXOQfI0C6+Joj1xFHREgZj2LrHaalCZ9ed sdTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JDPfoPzFYNlHD1wRkvvuJy2Bjmq5s9AdMrdSZF0YbfI=; b=c11Yr2sgYter28htpxgA//N2kgGBTTaP2jQdOmKeX5z45CLFpvseiFzqkdGSH4MwPA DBgNpzrYByErhXa5lF3c/zbVnhS+wZ79hgcNmAsWSQvIlt00EtXon+N0IhpAuOIv/dWK EAdDQNu4zWFA8wBfCipjvP7tBlbrjX0XUA3Fcq9tzlptGXMqPGpm3xXFxV8ZyHkXnkOh GgnJVN1dG2BqM4zFxWa0f2gcg5D8N95hOCNtYTxDmkE3HPnTgywyj65l8dNBmk7oQo6l DQLgxa/Ku+x1mJq83aWbUYz/vAE/XRreEgcni8vCb9XuBrgm3SQNYqJZa0PgRQp5RMSI yTLA== X-Gm-Message-State: AOAM532acqXPf5R9JFPY+2bmZHanSQ4MneMwdePbWGga5q4hLPcbEQMz gZc1ftwpCXvu0lbgE3ssW8x8aQY6P3+wT9B9oBlCfA== X-Google-Smtp-Source: ABdhPJxEdu0fkr3C+vBo+FZ76QAAdfvL01uZBDVZ33ipE86FnsuIBlTS0BK/SDROjudWftHUDxZt9v7UUrrrt8FEmvc= X-Received: by 2002:a05:6512:3046:: with SMTP id b6mr15846057lfb.65.1593015461544; Wed, 24 Jun 2020 09:17:41 -0700 (PDT) MIME-Version: 1.0 References: <875zbh0wbf.fsf@gmail.com> <83bll8ttet.fsf@gnu.org> In-Reply-To: <83bll8ttet.fsf@gnu.org> From: yyoncho Date: Wed, 24 Jun 2020 19:17:30 +0300 Message-ID: Subject: Re: bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop To: Eli Zaretskii Content-Type: multipart/alternative; boundary="000000000000b5086005a8d6d0ea" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42028 Cc: 42028@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --000000000000b5086005a8d6d0ea Content-Type: text/plain; charset="UTF-8" > The problem is that I expect a module that inserts such events > to want to be called to process those events as well. Is that > expectation correct? > Yes. As a side note, this feature will be useful even for the core C part for anything that wants to perform async processing not including LispObject. Thanks, Ivan --000000000000b5086005a8d6d0ea Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

The problem is that I expect = a module that inserts such events
to want to be called to process those events as well.=C2=A0 Is that
expectation correct?=C2=A0=C2=A0

Yes. A= s a side note, this feature will be useful even for the core C part for any= thing
that wants to perform async processing not including LispOb= ject.=C2=A0

Thanks,
Ivan
--000000000000b5086005a8d6d0ea-- From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 12:29:20 2020 Received: (at 42028) by debbugs.gnu.org; 24 Jun 2020 16:29:20 +0000 Received: from localhost ([127.0.0.1]:38763 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo8H1-0007MN-LG for submit@debbugs.gnu.org; Wed, 24 Jun 2020 12:29:20 -0400 Received: from eggs.gnu.org ([209.51.188.92]:60336) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo8Gy-0007M6-7n for 42028@debbugs.gnu.org; Wed, 24 Jun 2020 12:29:18 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:52316) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jo8Gs-0008Cg-WF; Wed, 24 Jun 2020 12:29:11 -0400 Received: from [176.228.60.248] (port=4904 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jo8Gr-0007yu-NR; Wed, 24 Jun 2020 12:29:10 -0400 Date: Wed, 24 Jun 2020 19:28:54 +0300 Message-Id: <83366ktol5.fsf@gnu.org> From: Eli Zaretskii To: yyoncho In-Reply-To: (message from yyoncho on Wed, 24 Jun 2020 19:17:30 +0300) Subject: Re: bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop References: <875zbh0wbf.fsf@gmail.com> <83bll8ttet.fsf@gnu.org> X-Spam-Score: -0.3 (/) X-Debbugs-Envelope-To: 42028 Cc: 42028@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.3 (-) > From: yyoncho > Date: Wed, 24 Jun 2020 19:17:30 +0300 > Cc: 42028@debbugs.gnu.org > > As a side note, this feature will be useful even for the core C part for anything > that wants to perform async processing not including LispObject. How will it help that? The event queue is processed synchronously. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 12:47:18 2020 Received: (at 42028) by debbugs.gnu.org; 24 Jun 2020 16:47:18 +0000 Received: from localhost ([127.0.0.1]:38788 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo8YP-0007oV-PM for submit@debbugs.gnu.org; Wed, 24 Jun 2020 12:47:18 -0400 Received: from mail-lf1-f54.google.com ([209.85.167.54]:37642) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo8YM-0007oF-4k for 42028@debbugs.gnu.org; Wed, 24 Jun 2020 12:47:16 -0400 Received: by mail-lf1-f54.google.com with SMTP id k15so1643547lfc.4 for <42028@debbugs.gnu.org>; Wed, 24 Jun 2020 09:47:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Fwn0feY9rnR09OVuwGU1IxdGW+SezzB11Ycrrn+sxDI=; b=XCb0m8pkxrr+85xd656oX66f7xiyttp/JFFKJV/GdK/zH5EnsMAXfWvQ4I/mDKIN6U HJqGfu1s6D4VubrqZ4xRdrAbLJqHhLsmUZtuRhCL5nkSJcdSnkbpGzatCuEh4KIlJNLS ddmLfDb0ywARHKvIiuBnRxGzUblBJz6+bZ/tLSUBxCyv2D2QY0Pz7+pkO305Nk6T4+// Tc02wyiILGFJB+cLEpywsiuAaaIznMgFiLFZmQ0KIpYsKtuBYsSZQtQzk2mpwJBSm5vM FpawkkjzC1+uKDJBWmiDYDp5D2RZREgTj8OmFM5FHwe98/m4Eh7Z7EiioQWH9+ceQjU7 x74g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Fwn0feY9rnR09OVuwGU1IxdGW+SezzB11Ycrrn+sxDI=; b=O8x7oN4C3JjkTn3KAf/6EZRJooM8imWgx+TvT5V6l4lToxOjM7N0ttyCEoq7KRmZ11 yIT269tnu4YOdu8GmKQ6yiXvLRdwGztiTibXur6WDdjyRvU9ohNZSifEFQPMO9w3ubX5 WfdMN0D9v0XoQsxSMiIQIeS5Rg0IOeZ42UEJF2yMwVxtp0nJMzifhvPKdHtXkCpnPWm3 UGu4M1xUktkDNqJATk/cdZnBh36AuSY9zRZakWwBL6kDs1u4EQPzQCcOBm7If8zbUvIw /KQfUrNBM0so7EAG+3oCzUXBZgL+E/YvVhKQ2ntMTXCsJukVY6Y9I9UHH+4k0dvmalTE j77w== X-Gm-Message-State: AOAM530t5LpgaRL5pzpvH6XSQKZQUIrSfxbMVVpyPpJzp66zDs6Nv27w Lk/ysCZ8Uc7lrXt2CUcghvO0quhu//9/EYPiYvSm6Q== X-Google-Smtp-Source: ABdhPJywlJSeTUOfAxdhsGFiHoc+cxYYXCgAYcDKZiE7G34nb1yw4gaiKqiv7ixoVzP5yd9bupf6HQ/0cugOgqn7Cn0= X-Received: by 2002:a05:6512:303:: with SMTP id t3mr15948307lfp.159.1593017227992; Wed, 24 Jun 2020 09:47:07 -0700 (PDT) MIME-Version: 1.0 References: <875zbh0wbf.fsf@gmail.com> <83bll8ttet.fsf@gnu.org> <83366ktol5.fsf@gnu.org> In-Reply-To: <83366ktol5.fsf@gnu.org> From: yyoncho Date: Wed, 24 Jun 2020 19:46:56 +0300 Message-ID: Subject: Re: bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop To: Eli Zaretskii Content-Type: multipart/alternative; boundary="000000000000fee45f05a8d73915" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42028 Cc: 42028@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --000000000000fee45f05a8d73915 Content-Type: text/plain; charset="UTF-8" Here it is an example: If we oversimplify json parsing consist of 2 parts. 1. JSON parsing to Jansson data structures 2. Converting Jasson data structures to lisp data structures. If we assume that 1 is taking most of the time then we could write the following code: start(string, callback) -> convert string to char* -> post it to worker thread -> worker thread does the parsing -> worker thread post to main loop -> main loop converts Jansson ds to lisp ds -> main loop calls the callback passed on start The same strategy probably could be applied to Font locking with TreeSitter. Thanks, Ivan On Wed, Jun 24, 2020 at 7:29 PM Eli Zaretskii wrote: > > From: yyoncho > > Date: Wed, 24 Jun 2020 19:17:30 +0300 > > Cc: 42028@debbugs.gnu.org > > > > As a side note, this feature will be useful even for the core C part for > anything > > that wants to perform async processing not including LispObject. > > How will it help that? The event queue is processed synchronously. > --000000000000fee45f05a8d73915 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Here it is an example: If we oversimplify json parsing con= sist of 2 parts.

1. JSON parsing to Jansson data structures
2. Converting Jasson data structures to lisp data structures.
=

If we assume that 1 is taking most of the time then we = could write the following code:

start(string, call= back) -> convert string to char* -> post it to worker thread -> wo= rker thread does the parsing=C2=A0
-> worker thread post to ma= in loop -> main loop converts Jansson ds to lisp ds -> main loop
calls the callback passed on start

The same = strategy probably could be applied to Font locking with TreeSitter.=C2=A0

Thanks,
Ivan

On Wed, Jun 24, 2= 020 at 7:29 PM Eli Zaretskii <eliz@gnu.o= rg> wrote:
yyoncho@gmail.com>
> Date: Wed, 24 Jun 2020 19:17:30 +0300
> Cc: 42028@d= ebbugs.gnu.org
>
> As a side note, this feature will be useful even for the core C part f= or anything
> that wants to perform async processing not including LispObject.

How will it help that?=C2=A0 The event queue is processed synchronously.
--000000000000fee45f05a8d73915-- From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 13:08:25 2020 Received: (at 42028) by debbugs.gnu.org; 24 Jun 2020 17:08:25 +0000 Received: from localhost ([127.0.0.1]:38799 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo8sq-0008Jw-Sm for submit@debbugs.gnu.org; Wed, 24 Jun 2020 13:08:25 -0400 Received: from eggs.gnu.org ([209.51.188.92]:45064) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo8sm-0008Jh-1n for 42028@debbugs.gnu.org; Wed, 24 Jun 2020 13:08:23 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:52740) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jo8sg-0007eS-Rk; Wed, 24 Jun 2020 13:08:14 -0400 Received: from [176.228.60.248] (port=3329 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jo8sZ-0004gQ-8y; Wed, 24 Jun 2020 13:08:14 -0400 Date: Wed, 24 Jun 2020 20:07:50 +0300 Message-Id: <83zh8ss87t.fsf@gnu.org> From: Eli Zaretskii To: yyoncho In-Reply-To: (message from yyoncho on Wed, 24 Jun 2020 19:46:56 +0300) Subject: Re: bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop References: <875zbh0wbf.fsf@gmail.com> <83bll8ttet.fsf@gnu.org> <83366ktol5.fsf@gnu.org> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 42028 Cc: 42028@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: yyoncho > Date: Wed, 24 Jun 2020 19:46:56 +0300 > Cc: 42028@debbugs.gnu.org > > 1. JSON parsing to Jansson data structures > 2. Converting Jasson data structures to lisp data structures. > > If we assume that 1 is taking most of the time Does it? Did someone time these separately? > start(string, callback) -> convert string to char* -> post it to worker thread -> worker thread does the parsing > -> worker thread post to main loop -> main loop converts Jansson ds to lisp ds -> main loop > calls the callback passed on start You cannot easily post to the queue from a worker thread, because the queue cannot be posted to asynchronously. And I lack the larger picture: how would this work within a framework of some Emacs feature? E.g., how will this processed be triggered, and by what kind of trigger? > The same strategy probably could be applied to Font locking with TreeSitter. Again, I don't see the wider picture. Are you familiar with how the current JIT font-lock works? If so, can you explain which parts of that will be replaced/modified, and how? From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 13:33:43 2020 Received: (at 42028) by debbugs.gnu.org; 24 Jun 2020 17:33:43 +0000 Received: from localhost ([127.0.0.1]:38822 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo9HL-0000UI-6h for submit@debbugs.gnu.org; Wed, 24 Jun 2020 13:33:43 -0400 Received: from mail-wr1-f41.google.com ([209.85.221.41]:40158) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo9HG-0000U0-G4 for 42028@debbugs.gnu.org; Wed, 24 Jun 2020 13:33:41 -0400 Received: by mail-wr1-f41.google.com with SMTP id h5so3094995wrc.7 for <42028@debbugs.gnu.org>; Wed, 24 Jun 2020 10:33:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=references:user-agent:from:to:cc:subject:in-reply-to:date :message-id:mime-version; bh=YvC750XB6C+fj8RXf9dA3Nj/oX/5hjx/htOxY5zylo4=; b=Fk192yTb10Pb/TRMFgBnkk3jbM8HHYYGaveNqeoIGNNumSqeCpNDge+9MnlbNcC6TU 7m3ee6PPNwTWKUa3aPVBY8DghQ5ZIh+h8YlIfNBk2Or/0evcRTV1OicyNGDyVHwkGJeu isNO6mcp/jby8mZgkkVtxw5lAiWF9ottdaBRnHZr9BgQPJIYf0FCv13rsxIR8bhLceYI Fx0o9THkksO6h/5OKybOiEWlsWfkZ6EpTXiyS+fqpiGZAOF/A+kk1efEUWA48kSgkjII dXkQGdURql5vG8rNmWXkUEUdDQT+3LUKVYfARhAGvjp6rxX/y9JL8u384xtN+2uwXYTt As7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:date:message-id:mime-version; bh=YvC750XB6C+fj8RXf9dA3Nj/oX/5hjx/htOxY5zylo4=; b=a1KsT/TSgMtSWXCxM+mjrSn44ex16ffgZQXMWKwjJPB/RHTaeFiyBvPCHV3z7KjXWb HnbqFXQQ6+M5D+JSvG6cBXUVWBc4f8mnxekvX4/LVgrV7X05QO+4WERyzl0rDlGMaIaR O2ghHWOmLXLWcBOlyh4jZb7YaoqTRgNQf5HIh9Upb3a1CIxAOr/I98J4DMQE2S8UY07o LkbyJrayBp1KqcjM6diu749iFU9JhQE4U32uiSGOPdgVbLh7LZ+l63Ao/jF1sJEUy9Ik p3AVMXZgTmcflZAGEVfIEm/IbfpMiB2oEgmjKe5t6FvQlKggXrfAuUoO2E7wbMPk/KCS xjcA== X-Gm-Message-State: AOAM530M/Ln5saONRruaiBD8BRlOJoA3eAyGa7xDWSi21n51WiOtBmDn qD7EDhn5pJ7Yi2Isayy/531wDuc9 X-Google-Smtp-Source: ABdhPJztjr8G7wv4fmnnkhNgZ0zXIzjTvB9dLyxHyI2jaqH7MSScHeUNMTfM7vtAeSuzBABiV9+iSQ== X-Received: by 2002:adf:97cb:: with SMTP id t11mr32166070wrb.314.1593020012046; Wed, 24 Jun 2020 10:33:32 -0700 (PDT) Received: from kyoncho-H87-D3H ([2a00:4802:28d:9e00:8c64:bcc8:970f:40b]) by smtp.gmail.com with ESMTPSA id u20sm8590284wmc.44.2020.06.24.10.33.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jun 2020 10:33:31 -0700 (PDT) References: <875zbh0wbf.fsf@gmail.com> <83bll8ttet.fsf@gnu.org> <83366ktol5.fsf@gnu.org> <83zh8ss87t.fsf@gnu.org> User-agent: mu4e 1.3.7; emacs 27.0.91 From: Ivan Yonchovski To: Eli Zaretskii Subject: Re: bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop In-reply-to: <83zh8ss87t.fsf@gnu.org> Date: Wed, 24 Jun 2020 20:33:29 +0300 Message-ID: <874kr0gyhi.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42028 Cc: 42028@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eli Zaretskii writes: > Does it? Did someone time these separately? IDK, this was just an example. >> start(string, callback) -> convert string to char* -> post it to worker thread -> worker thread does the parsing >> -> worker thread post to main loop -> main loop converts Jansson ds to lisp ds -> main loop >> calls the callback passed on start > > You cannot easily post to the queue from a worker thread, because the > queue cannot be posted to asynchronously. AFAIK this is not possible at all - thus this feature request. > > And I lack the larger picture: how would this work within a framework > of some Emacs feature? E.g., how will this processed be triggered, > and by what kind of trigger? The goal users to be able to handle more usecases. ATM the only way to achieve that is hackish as described here https://nullprogram.com/blog/2017/02/14/ + several practical usecases. > Again, I don't see the wider picture. Are you familiar with how the > current JIT font-lock works? If so, can you explain which parts of > that will be replaced/modified, and how? AFAIK the initial tree-sitter parsing (which does not involve anything emacs related) may take 200-300ms. In other editors (e.g. vscode) this part is not happening on the UI thread but in some extenal background thread which then passes the AST to the main UI thread to do the actual fontlock. IMO this model might be replicated in emacs in the event of TS integration. Thanks, Ivan From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 14:05:19 2020 Received: (at 42028) by debbugs.gnu.org; 24 Jun 2020 18:05:19 +0000 Received: from localhost ([127.0.0.1]:38871 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo9lu-0001G8-DQ for submit@debbugs.gnu.org; Wed, 24 Jun 2020 14:05:19 -0400 Received: from eggs.gnu.org ([209.51.188.92]:32864) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo9lr-0001Ft-60 for 42028@debbugs.gnu.org; Wed, 24 Jun 2020 14:05:17 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:53321) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jo9lm-00024F-05; Wed, 24 Jun 2020 14:05:10 -0400 Received: from [176.228.60.248] (port=2864 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1jo9ll-0006EM-BL; Wed, 24 Jun 2020 14:05:09 -0400 Date: Wed, 24 Jun 2020 21:04:52 +0300 Message-Id: <83y2ocs5kr.fsf@gnu.org> From: Eli Zaretskii To: Ivan Yonchovski In-Reply-To: <874kr0gyhi.fsf@gmail.com> (message from Ivan Yonchovski on Wed, 24 Jun 2020 20:33:29 +0300) Subject: Re: bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop References: <875zbh0wbf.fsf@gmail.com> <83bll8ttet.fsf@gnu.org> <83366ktol5.fsf@gnu.org> <83zh8ss87t.fsf@gnu.org> <874kr0gyhi.fsf@gmail.com> X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 42028 Cc: 42028@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) > From: Ivan Yonchovski > Cc: 42028@debbugs.gnu.org > Date: Wed, 24 Jun 2020 20:33:29 +0300 > > > You cannot easily post to the queue from a worker thread, because the > > queue cannot be posted to asynchronously. > > AFAIK this is not possible at all - thus this feature request. Your feature just asked for a way to queue events, it didn't say anything about doing that asynchronously, not from a function invoked by the main thread. If you mean the latter, then it AFAIU would need a serious redesign of the Emacs event queue, to make it accessible from several threads running in parallel at once. > > Again, I don't see the wider picture. Are you familiar with how the > > current JIT font-lock works? If so, can you explain which parts of > > that will be replaced/modified, and how? > > AFAIK the initial tree-sitter parsing (which does not involve anything > emacs related) may take 200-300ms. In other editors (e.g. vscode) this > part is not happening on the UI thread but in some extenal background > thread which then passes the AST to the main UI thread to do the actual > fontlock. IMO this model might be replicated in emacs in the event of TS > integration. What you describe is very different from how JIT font-lock works now. Which is why I asked the question: I know (more or less) how it works in other editors, I just don't yet understand well enough how something like that would fit into the existing fontification framework. I hope there is a way of fitting it, because otherwise it would mean a serious surgery of the display engine as well, which will make the job significantly larger and harder. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 14:38:41 2020 Received: (at 42028) by debbugs.gnu.org; 24 Jun 2020 18:38:41 +0000 Received: from localhost ([127.0.0.1]:39003 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joAID-00026u-G3 for submit@debbugs.gnu.org; Wed, 24 Jun 2020 14:38:41 -0400 Received: from mail-wr1-f51.google.com ([209.85.221.51]:35681) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joAIB-00026h-NN for 42028@debbugs.gnu.org; Wed, 24 Jun 2020 14:38:40 -0400 Received: by mail-wr1-f51.google.com with SMTP id g18so3299023wrm.2 for <42028@debbugs.gnu.org>; Wed, 24 Jun 2020 11:38:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=references:user-agent:from:to:cc:subject:in-reply-to:date :message-id:mime-version; bh=wHck0mTFo/puF8WqyTp12OtABKTN47zw/abUwF2m440=; b=FrL6FMq0nQmRoO3eD9e/fmHHIkNYyVM4ncqJnxb+kZciPpDE4IjYk0STo2VF5UU7az I02IxbnzDfR0Q/llbZJISVzgNLL8+/YZNA+sMujf+LVRu8wf/94+10HDKmGkDFPoO/j2 klr2kT88IiNf/UR4OyJJF7X8Th7PAVnmc7X68J9johtxZMB4CzURK/1Rj8xKju1OReHA H172JyN1fU+N6u631mUt+AwcY36a3JnuMpfombUn7hFYst3TBqEVZaZYAuYCp+YloTxl FU31+CYqZc0jI4BfvcKSJDc9RsVdLjLfnZgz/dqvfQ4qrrzlV9MyAG/RKbrn5XEl1HAh iBCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:user-agent:from:to:cc:subject :in-reply-to:date:message-id:mime-version; bh=wHck0mTFo/puF8WqyTp12OtABKTN47zw/abUwF2m440=; b=ijsm7hw6N8k65vykqhHExIg1CyYB0hOy4fV/kEdNmrU2eotEzwun09rmMGejhL6/3J SuNqhr7aTsm6cRzhrhdAjK7Hg1m0BmcmxhFuWQqfPRDHsFKz5lIJvM/cQVfMrg5hfU2K yzkr4eqwGGlpV44ByHAxByOgI2oSRF6NGR4skDEkOCupzdv5RXVmbAUAybl6IdrsNN/K IUmhkkMUpEipJtLTEwQcL+ksdrmbAG+YdCZ4mIUqF+xKIlq0FypmCuWNkVuU9dGe6atq cp2M8ewJtumPRepcC9cb9SDoEc21y5zcRysfCqMXuwFc8Q7qPhzi5ywEl3HMp9eaSprN CFzw== X-Gm-Message-State: AOAM5305iEZ0fHQivDEcFxbUAGdtyL4c53fYkia4bqQHyNtVHF8wZdrd iHwa727s0/myKh/sB2NTTYpdIrAg X-Google-Smtp-Source: ABdhPJyRjcSxK65ntjkoEgv1McNzxIkYq9RCQLqiMu1B+36BDMr01cedEK832gQvZrGz8H2z7C1DGA== X-Received: by 2002:adf:f711:: with SMTP id r17mr33250779wrp.409.1593023913306; Wed, 24 Jun 2020 11:38:33 -0700 (PDT) Received: from kyoncho-H87-D3H ([2a00:4802:28d:9e00:8c64:bcc8:970f:40b]) by smtp.gmail.com with ESMTPSA id p25sm8946806wmg.39.2020.06.24.11.38.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jun 2020 11:38:32 -0700 (PDT) References: <875zbh0wbf.fsf@gmail.com> <83bll8ttet.fsf@gnu.org> <83366ktol5.fsf@gnu.org> <83zh8ss87t.fsf@gnu.org> <874kr0gyhi.fsf@gmail.com> <83y2ocs5kr.fsf@gnu.org> User-agent: mu4e 1.3.7; emacs 27.0.91 From: Ivan Yonchovski To: Eli Zaretskii Subject: Re: bug#42028: [Feature Request] 27.0.91; Provide the ability dynamic modules to post events in emacs event loop In-reply-to: <83y2ocs5kr.fsf@gnu.org> Date: Wed, 24 Jun 2020 21:38:31 +0300 Message-ID: <87eeq4fgwo.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 42028 Cc: 42028@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Eli Zaretskii writes: > > Your feature just asked for a way to queue events, it didn't say > anything about doing that asynchronously, not from a function invoked > by the main thread. If you mean the latter, then it AFAIU would need > a serious redesign of the Emacs event queue, to make it accessible > from several threads running in parallel at once. Yes, sorry about that. I didn't mention it explicitly in the bug report it was described in more details in the linked article. > What you describe is very different from how JIT font-lock works now. > Which is why I asked the question: I know (more or less) how it works > in other editors, I just don't yet understand well enough how > something like that would fit into the existing fontification > framework. I hope there is a way of fitting it, because otherwise it > would mean a serious surgery of the display engine as well, which will > make the job significantly larger and harder. IMO we are good here. At least on lsp-mode side we were able to implement delayed semantic fontification as a result of async source. But in lsp-mode case we are reseiving the messages from the server on the UI thread. Thanks, Ivan