GNU bug report logs - #73304
Python in relocatable guix pack leads to wrong sys.path

Previous Next

To reply to this bug, email your comments to 73304 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> protonmail.com>
To: bug-guix <at> gnu.org
Subject: Python in relocatable guix pack leads to wrong sys.path
Date: Mon, 16 Sep 2024 19:23:04 +0000

When I add both python and a python package (seems like any) to a
relocatable guix pack, the resulting python in the merged profile seems
to be broken. Specifically its `sys.path` contains missing paths.

To replicate:
```
mkdir -p /tmp/b && cd /tmp/b
tar xzvf $(guix pack -RR python python-pyflakes -S /profile=)`
```

Then try executing the python from merged profile, and see sys.path output
--8<---------------cut here---------------start------------->8---
./profile/bin/python3
>>> import sys
>>> print(sys.path)
['', '/tmp/b/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python310.zip', '/tmp/b/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10', '/tmp/b/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10/lib-dynload', '/tmp/b/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10/site-packages']
--8<---------------cut here---------------end--------------->8---

Now if you instead use the realpath of this python
--8<---------------cut here---------------start------------->8---
$(realpath ./profile/bin/python3)
>>> import sys
>>> print(sys.path)
['', '/tmp/b/gnu/store/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python310.zip', '/tmp/b/gnu/store/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10', '/tmp/b/gnu/store/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10/lib-dynload', '/tmp/b/gnu/store/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10/site-packages']
--8<---------------cut here---------------end--------------->8---

As can be observed, the first instance misses "gnu/store" in the path,
and instead points to a path that does not exist. I have not
investigated, at least not yet, where this discrepancy comes from.
I was not able to replicate this with anything else than relocatable
pack. So my guess would be something strange happens when making
bwrap/fakechroot etc.

Regards,
Rutherther

Message #8 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> protonmail.com>
To: 73304 <at> debbugs.gnu.org
Subject: Re: Python in relocatable guix pack leads to wrong sys.path
Date: Mon, 16 Sep 2024 19:28:51 +0000

As a workaround, the profile in gnu/store can be used instead
of the one exposed by -S symlink. Specifically, getting
`realpath ./profile` and using that as the profile instead
of `./profile` directly resolves the issue.

Message #11 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Rutherther <rutherther <at> protonmail.com>
Cc: Josselin Poiret <dev <at> jpoiret.xyz>, 73304 <at> debbugs.gnu.org,
 Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Mon, 30 Sep 2024 23:08:29 +0200

Hello,

Rutherther <rutherther <at> protonmail.com> skribis:

> When I add both python and a python package (seems like any) to a
> relocatable guix pack, the resulting python in the merged profile seems
> to be broken. Specifically its `sys.path` contains missing paths.
>
> To replicate:
> ```
> mkdir -p /tmp/b && cd /tmp/b
> tar xzvf $(guix pack -RR python python-pyflakes -S /profile=)`
> ```
>
> Then try executing the python from merged profile, and see sys.path output
>
> ./profile/bin/python3
>>>> import sys
>>>> print(sys.path)
> ['', '/tmp/b/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python310.zip', '/tmp/b/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10', '/tmp/b/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10/lib-dynload', '/tmp/b/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10/site-packages']
>
>
> Now if you instead use the realpath of this python
>
> $(realpath ./profile/bin/python3)
>>>> import sys
>>>> print(sys.path)
> ['', '/tmp/b/gnu/store/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python310.zip', '/tmp/b/gnu/store/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10', '/tmp/b/gnu/store/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10/lib-dynload', '/tmp/b/gnu/store/pwajwfiygrkmag8fm1wzn3cf362j7sri-python-3.10.7R/lib/python3.10/site-packages']
>
> As can be observed, the first instance misses "gnu/store" in the path,
> and instead points to a path that does not exist.

Commit d5e0180805f52ef38a03ff9d6c11dd6174e3526b fixed a similar issue I
think, discussed at <https://issues.guix.gnu.org/53258>.

Not sure whether it’s a new problem here.  Konrad, Maxim, Josselin,
WDYT? :-)

Thanks,
Ludo’.

Message #14 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> ditigal.xyz>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Josselin Poiret <dev <at> jpoiret.xyz>, 73304 <at> debbugs.gnu.org,
 Konrad Hinsen <konrad.hinsen <at> fastmail.net>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Tue, 01 Oct 2024 21:39:23 +0200

Ludovic Courtès <ludo <at> gnu.org> writes:

> Commit d5e0180805f52ef38a03ff9d6c11dd6174e3526b fixed a similar issue I
> think, discussed at <https://issues.guix.gnu.org/53258>.
>
> Not sure whether it’s a new problem here.  Konrad, Maxim, Josselin,
> WDYT? :-)

Hello Ludovic,

if I understand correctly this commit was merged to master with
the quite-recent core-updates merge? The experiment I did was
after this.

Moreover, I tried with newest master now to make sure,
and still the same behavior.

I will try to investigate this myself after I get some time (which may
be quite a while) if no one else will get to it until that time.

Regards,
Rutherther.

PS: sorry for the e-mail change, I recently set up self-hosted one, so
let's see how well it works with debbugs / your gnu.org address. :)

Message #17 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: Ludovic Courtès <ludo <at> gnu.org>, Rutherther
 <rutherther <at> protonmail.com>
Cc: 73304 <at> debbugs.gnu.org, Josselin Poiret <dev <at> jpoiret.xyz>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Wed, 02 Oct 2024 08:20:51 +0200

Ludovic Courtès <ludo <at> gnu.org> writes:

> Rutherther <rutherther <at> protonmail.com> skribis:
>
>> When I add both python and a python package (seems like any) to a
>> relocatable guix pack, the resulting python in the merged profile seems
>> to be broken. Specifically its `sys.path` contains missing paths.

> Commit d5e0180805f52ef38a03ff9d6c11dd6174e3526b fixed a similar issue I
> think, discussed at <https://issues.guix.gnu.org/53258>.
>
> Not sure whether it’s a new problem here.  Konrad, Maxim, Josselin,
> WDYT? :-)

This problem clearly looks like it's caused by our sitecustomize.py.
One indicator is "When I add both python and a python package": if there
is no additional package, only python by itself, our sitecustomize
doesn't do anything.

However, I don't see how the recent patch (see
https://issues.guix.gnu.org/68241) could be responsible for this new
problem. It should have happened before as well.

Here is what sitecustomize.py does:

 1. Collect the "site-packages" subdirectories of everything on
    $GUIX_PYTHONPATH.
 2. Normalize them via sys.normpath()
 3. Add them to sys.path
 4. Re-arrange the elements in sys.path such that the add-on
    packages come before Python's standard library.

The recent patch fixes a bug in the last step, which wasn't executed at
all (and generated an error message). So the difference between the
current and the previous behavior is only the order of entries in
sys.path.

The current problem is that even the standard library's paths are wrong.
I suspect that this is due to sys.normpath not implementing the correct
POSIX semantics for dot-dot.

Cheers,
  Konrad.

Message #20 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: Ludovic Courtès <ludo <at> gnu.org>, Rutherther
 <rutherther <at> protonmail.com>
Cc: 73304 <at> debbugs.gnu.org, Josselin Poiret <dev <at> jpoiret.xyz>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Wed, 02 Oct 2024 13:59:56 +0200

Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:

> This problem clearly looks like it's caused by our sitecustomize.py.
> One indicator is "When I add both python and a python package": if there
> is no additional package, only python by itself, our sitecustomize
> doesn't do anything.

I have changed my mind. In the sys.path outputs shown, there are no
paths from add-on packages. It's just the Python standard library.
Maybe our sitecustomize.py is not run at all, but if it is, it didn't do
anything to sys.path. There must be a bug somewhere else.

Cheers,
  Konrad.

Message #23 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: Ludovic Courtès <ludo <at> gnu.org>, Rutherther
 <rutherther <at> protonmail.com>
Cc: 73304 <at> debbugs.gnu.org, Josselin Poiret <dev <at> jpoiret.xyz>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Sun, 06 Oct 2024 10:29:22 +0200

[Message part 1 (text/plain, inline)]

Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:

> I have changed my mind. In the sys.path outputs shown, there are no
> paths from add-on packages. It's just the Python standard library.
> Maybe our sitecustomize.py is not run at all, but if it is, it didn't do
> anything to sys.path. There must be a bug somewhere else.

Our sitecustomize.py is indeed not run at all, so this definitely is a
different problem.

Evidence: Run Rutherther's example, adding the -v option.  The long
output is attached, both for "./profile/bin/python3 -v" and "$(realpath
./profile/bin/python3) -v". Search for "site-packages" to find the
interesting parts. If you don't use realpath, large parts of the
initialization are not done.

There are lots of ../../ in the path shown in these log files. If Python
resolves them lexically, as the normpath function does, that would
probably explain most of these issues.

Cheers,
  Konrad.

[without-realpath.log (application/octet-stream, attachment)]

[with-realpath.log (application/octet-stream, attachment)]

Message #26 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: Rutherther <rutherther <at> protonmail.com>, 73304 <at> debbugs.gnu.org,
 Josselin Poiret <dev <at> jpoiret.xyz>, Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Mon, 14 Oct 2024 13:11:36 +0200

Hello,

Konrad Hinsen <konrad.hinsen <at> fastmail.net> skribis:

> Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:
>
>> I have changed my mind. In the sys.path outputs shown, there are no
>> paths from add-on packages. It's just the Python standard library.
>> Maybe our sitecustomize.py is not run at all, but if it is, it didn't do
>> anything to sys.path. There must be a bug somewhere else.
>
> Our sitecustomize.py is indeed not run at all, so this definitely is a
> different problem.
>
> Evidence: Run Rutherther's example, adding the -v option.  The long
> output is attached, both for "./profile/bin/python3 -v" and "$(realpath
> ./profile/bin/python3) -v". Search for "site-packages" to find the
> interesting parts. If you don't use realpath, large parts of the
> initialization are not done.
>
> There are lots of ../../ in the path shown in these log files. If Python
> resolves them lexically, as the normpath function does, that would
> probably explain most of these issues.

Ouch.  Any idea where to look for solutions here?

Ludo’.

Message #29 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> protonmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>,
 Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: 73304 <at> debbugs.gnu.org, Josselin Poiret <dev <at> jpoiret.xyz>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Fri, 31 Jan 2025 10:10:51 +0000

> Hello,
>
> Konrad Hinsen <konrad.hinsen <at> fastmail.net> skribis:
>
>> Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:
>>
>>> I have changed my mind. In the sys.path outputs shown, there are no
>>> paths from add-on packages. It's just the Python standard library.
>>> Maybe our sitecustomize.py is not run at all, but if it is, it didn't do
>>> anything to sys.path. There must be a bug somewhere else.
>>
>> Our sitecustomize.py is indeed not run at all, so this definitely is a
>> different problem.
>>
>> Evidence: Run Rutherther's example, adding the -v option.  The long
>> output is attached, both for "./profile/bin/python3 -v" and "$(realpath
>> ./profile/bin/python3) -v". Search for "site-packages" to find the
>> interesting parts. If you don't use realpath, large parts of the
>> initialization are not done.
>>
>> There are lots of ../../ in the path shown in these log files. If Python
>> resolves them lexically, as the normpath function does, that would
>> probably explain most of these issues.

I am not sure I understand this right. I am looking at the output, and
it has `/tmp/b/profile/bin/../../hash-python-3.10.7R/{rest}`, omitting
the gnu/store even here. Why is it significant how these paths are
evaluated, if there is the gnu/store part missing already?

Regards,
Rutherther

Message #32 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Rutherther <rutherther <at> protonmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>,
 Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: 73304 <at> debbugs.gnu.org, Josselin Poiret <dev <at> jpoiret.xyz>,
 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Fri, 31 Jan 2025 11:59:45 +0000

[Message part 1 (text/plain, inline)]

"Rutherther" <rutherther <at> protonmail.com> writes:

>> Hello,
>>
>> Konrad Hinsen <konrad.hinsen <at> fastmail.net> skribis:
>>
>>> Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:
>>>
>>>> I have changed my mind. In the sys.path outputs shown, there are no
>>>> paths from add-on packages. It's just the Python standard library.
>>>> Maybe our sitecustomize.py is not run at all, but if it is, it didn't do
>>>> anything to sys.path. There must be a bug somewhere else.
>>>
>>> Our sitecustomize.py is indeed not run at all, so this definitely is a
>>> different problem.
>>>
>>> Evidence: Run Rutherther's example, adding the -v option.  The long
>>> output is attached, both for "./profile/bin/python3 -v" and "$(realpath
>>> ./profile/bin/python3) -v". Search for "site-packages" to find the
>>> interesting parts. If you don't use realpath, large parts of the
>>> initialization are not done.
>>>
>>> There are lots of ../../ in the path shown in these log files. If Python
>>> resolves them lexically, as the normpath function does, that would
>>> probably explain most of these issues.
>
> I am not sure I understand this right. I am looking at the output, and
> it has `/tmp/b/profile/bin/../../hash-python-3.10.7R/{rest}`, omitting
> the gnu/store even here. Why is it significant how these paths are
> evaluated, if there is the gnu/store part missing already?
>
> Regards,
> Rutherther

Okay, I spent a bit of time on this, and I think I get it now.
This seems to boil down to something in Python wrongly handling
the argv0, where it looks at the path, being /tmp/a/profile/bin/python3,
substitutes the symlink /tmp/a/profile/bin/python3/../../hash-python-3.10.7/bin/python3, and then
probably calls something like normpath on it, and it becomes
/tmp/a/hash-python-3.10.7. Whereas the correct approach is to first get
the actual path /tmp/a/profile/bin/python3 is at, being
/tmp/a/gnu/store/hash-python-3.10.7R/bin/python3, and unwrap the symlink
from there. This happens even if python3 is called from PATH, I suppose
Python then looks through PATH and again finds the python3 file outside
of the store.

As a workaround,

[run-in-namespace.patch (text/x-patch, attachment)]

Message #35 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Rutherther <rutherther <at> protonmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 73304 <at> debbugs.gnu.org,
 Konrad Hinsen <konrad.hinsen <at> fastmail.net>, Josselin Poiret <dev <at> jpoiret.xyz>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Fri, 31 Jan 2025 22:52:02 +0900

Hi,

Rutherther <rutherther <at> protonmail.com> writes:

[...]

> Okay, I spent a bit of time on this, and I think I get it now.
> This seems to boil down to something in Python wrongly handling
> the argv0, where it looks at the path, being /tmp/a/profile/bin/python3,
> substitutes the symlink /tmp/a/profile/bin/python3/../../hash-python-3.10.7/bin/python3, and then
> probably calls something like normpath on it, and it becomes
> /tmp/a/hash-python-3.10.7. Whereas the correct approach is to first get
> the actual path /tmp/a/profile/bin/python3 is at, being
> /tmp/a/gnu/store/hash-python-3.10.7R/bin/python3, and unwrap the symlink
> from there. This happens even if python3 is called from PATH, I suppose
> Python then looks through PATH and again finds the python3 file outside
> of the store.
>
> As a workaround,

The workaround looks smart (too smart for my tired brain tonight), but
to be honest I'd rather have Python handle the relative paths correctly,
or work around it in our sitecustomize.py script that runs early when
Python runs.  I doubt submitting a patch fixing this in Python will be
accepted, for backward compatibility reasons (I think Josselin had done
so already).  Looking back for it, I've seen reports of the issue going
back to 2008 [0], and a more recent one in 2021 [1]:

[0]  https://github.com/python/cpython/issues/46542
[1]  https://github.com/python/cpython/issues/88482

It seems using realpath wouldperhaps do the right thing instead of using
normpath?  It resolves symlinks, appears to conform to POSIX.  See some
sample output from [2].

[2]  https://stackoverflow.com/a/40311142/2896799

-- 
Thanks,
Maxim

Message #38 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>, Rutherther
 <rutherther <at> protonmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 73304 <at> debbugs.gnu.org,
 Josselin Poiret <dev <at> jpoiret.xyz>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Fri, 31 Jan 2025 15:12:40 +0100

Hi,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> The workaround looks smart (too smart for my tired brain tonight), but
> to be honest I'd rather have Python handle the relative paths correctly,
> or work around it in our sitecustomize.py script that runs early when

That's indeed preferable, but according to my analysis not possible, as
there are situations of faulty startup behavior in which out
sitecustomize.py is not run at all.

What we can envisage is a combination of
 1. patching Python
 2. using realpath rather than normpath in our sitecustomize.py

Cheers,
  Konrad.

Message #41 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: Rutherther <rutherther <at> protonmail.com>,
 Ludovic Courtès <ludo <at> gnu.org>,
 73304 <at> debbugs.gnu.org, Josselin Poiret <dev <at> jpoiret.xyz>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Sat, 01 Feb 2025 01:04:09 +0900

Hi Konrad,

Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:

> Hi,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>
>> The workaround looks smart (too smart for my tired brain tonight), but
>> to be honest I'd rather have Python handle the relative paths correctly,
>> or work around it in our sitecustomize.py script that runs early when
>
> That's indeed preferable, but according to my analysis not possible, as
> there are situations of faulty startup behavior in which out
> sitecustomize.py is not run at all.

I think I had read this from you before but I hadn't grasped how that's
possible/when it occurs.  Do yo have a link, or refresher?

> What we can envisage is a combination of
>  1. patching Python
>  2. using realpath rather than normpath in our sitecustomize.py

I hope we could get by with just 2.

-- 
Thanks,
Maxim

Message #44 received at 73304 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: Rutherther <rutherther <at> protonmail.com>,
 Ludovic Courtès <ludo <at> gnu.org>,
 73304 <at> debbugs.gnu.org, Josselin Poiret <dev <at> jpoiret.xyz>
Subject: Re: bug#73304: Python in relocatable guix pack leads to wrong sys.path
Date: Sat, 01 Feb 2025 01:09:44 +0900

Hi,

[...]

>> That's indeed preferable, but according to my analysis not possible, as
>> there are situations of faulty startup behavior in which out
>> sitecustomize.py is not run at all.
>
> I think I had read this from you before but I hadn't grasped how that's
> possible/when it occurs.  Do yo have a link, or refresher?
>
>> What we can envisage is a combination of
>>  1. patching Python
>>  2. using realpath rather than normpath in our sitecustomize.py
>
> I hope we could get by with just 2.

I've just reread the original message in this thread/issue, and it has a
reproducer demonstrating the problem.  If Python itself breaks the
symlinks paths before it even gets to run our sitecustomize.py, then
yes, we'd need a fix on the Python side.

I guess we should revisit this issue upstream (perhaps some are still
opened), show our broken use case, and discuss a solution with them?

-- 
Thanks,
Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.