Subject: bug#28159: closed (Re: Closing bug #28159? Updater needs to support HTTP(S) servers)

From: [email-masked] (GNU bug Tracking System)
Date: Thu, 30 Apr 2020 21:16:02 +0000

Your bug report

#28159: Updater needs to support HTTP(S) servers

which was filed against the guix package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
28159: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=28159
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: Closing bug #28159? Updater needs to support HTTP(S) servers
From: Ludovic Courtès <[email-masked]>
To: Brice Waegeneire <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Thu, 30 Apr 2020 23:14:59 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Hi Brice,

Brice Waegeneire <[email-masked]> skribis:

> It looks like now most of the major updaters that relied on FTP (GNU,
> kernel.org, KDE and Gnbome) now support HTTP(S). I think we can close
> this
> bug.

Yup.  There’s still the ‘gnu-ftp’ and the ‘xorg’ updaters which,
according to ‘guix refresh --list-updaters’, account for 2.2% of the
packages.  We can change them later when it becomes necessary.

Closing, thank you!

Ludo’.

Subject: Updater needs to support HTTP(S) servers
From: Hartmut Goebel <[email-masked]>
To: bug-guix <[email-masked]>
Date: Sun, 20 Aug 2017 14:06:02 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Hi,

our updater currently only supports FTP servers, but more and more
projects shutdown the FTP service and provide HTTP(S) servers only (e.g
the Linux kernel). For other projects, the main distribution point has
changed to HTTP and the mirrors still providing FTP at lagging (e.g.
KDE, see [1]).

A common case is to simply use Apache to serve the directories, but it
will deliver a HTML view on the directory contents (using mod_autoindex
[3]).

In [2] Ludo wrote:

    So we need a way to list the latest releases somehow.  If they publish
    JSON, XML, or some other structured info format, that’s fine too.  But
    HTTP alone is not good: we’d have to infer the information from HTML
    pages, which sounds fragile.

IMHO we can not expect project and mirror sites to provide these
additional data. Most projects simply will not do since this would
require the server to generate some data-files n the fly.

OTOH, I assume the delivered directory index pages to be well-formed
(X)HTML. Thus parsing the HTML should be quite simple: We only need to
pattern-match "<A>" tags, or – if guile has some decent one – a 
xml/html-parser use this to query the data.

Only relative links without slash (except a trailing one) have to be
handled. Links with a trailing slash can be assumed to be a directories.
(Since auto-index only works if URL is pointing to a directory and the
directory is marked by a training slash we can assume the generated
links for directories will all have the trailing slash.) At least this
would be a good start which could be refined if necessary.

Please note tha I'm not suggesting to write a general-purpose parser,
but aiming for auto-index html-pages only.

Some things I already found out:

  * Directory-listings generated by mod_autoindex can be provided as a
    simple list by passing the query-parameter "F=0" in the URL [4].
    There are other query parameters for sorting and pattern matching.
  * nginx's "ngx_http_autoindex_module" [6] seem to not use query
    parameters, but can be configured (on the server-side) to provide
    the content as XML or json. The "fancy_index" module [7] si
    documented to "Allow choosing to sort elements", but [7] does not
    state how and if "fancy" can be switched off.
  * Lighttp supports some of these options [5].

[1] http://lists.gnu.org/archive/html/guix-devel/2017-05/msg00237.html
[2] http://lists.gnu.org/archive/html/guix-devel/2017-05/msg00292.html
[3] https://httpd.apache.org/docs/2.4/mod/mod_autoindex.html
[4] https://httpd.apache.org/docs/2.4/mod/mod_autoindex.html#query
[5]
https://redmine.lighttpd.net/projects/1/wiki/Docs_ModDirlisting#Table-sorting
[6] http://nginx.org/en/docs/http/ngx_http_autoindex_module.html
[7] https://www.nginx.com/resources/wiki/modules/fancy_index/

-- 

Regards
Hartmut Goebel

| Hartmut Goebel          | [email-masked]               |
| www.crazy-compilers.com | compilers which you thought are impossible |