Subject: Re: bug#35969: 26.2, Excorporate

From: "tenspd137 ." <[email-masked]>
Date: Fri, 14 Jun 2019 17:14:26 -0600

Sorry - I was working fast:

*I created a function that changed the line
in  url-http-create-request so that the request looks like:*

In a file I did:

(defun thefile() "/EWS/Exchange.asmx")

(copied from url-http library)
(defun url-http-create-request (&optional ref-url)
.....
   ;; This was done with a call to `format'.  Concatenating parts has
    ;; the advantage of keeping the parts of each header together and
    ;; allows us to elide null lines directly, at the cost of making
    ;; the layout less clear.
    (setq request
          (concat
             ;; The request
             (or url-http-method "GET") " "
             (url-http--encode-string
              (if using-proxy (thefile) real-fname))
<------------Changed this to "hardcode" the request as proof of
concept
             " HTTP/" url-http-version "\r\n"
             ;; Version of MIME we speak
             "MIME-Version: 1.0\r\n"
             ;; (maybe) Try to keep the connection open
             "Connection: " (if (or using-proxy .....
.......
.......

Loaded the file and then ran (url-retriev-synchronously "https://......")

Thanks!

-C

On Fri, Jun 14, 2019 at 5:07 PM tenspd137 . <[email-masked]> wrote:
>
> I think I figured it out.  I created a function that changed the line
> in  url-http-create-request so that the request looks like:
> GET /EWS/Exchange.asmx HTTP/1.1 <-----  *no server / host*
> MIME-Version: 1.0
> Connection: close
> Extension: Security/Digest Security/SSL
> Host: outlook.office365.com
> Accept-encoding: gzip
> Accept: */*
> User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
> Cookie: OIDC=1; ClientId=8998C5691CD143E784857A0D01537963
> Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOlMxa3kzbGk3bmUwNzMxJSU=
>
> instead of
>
> GET https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1
> MIME-Version: 1.0
> Connection: close
> Extension: Security/Digest Security/SSL
> Host: outlook.office365.com
> Accept-encoding: gzip
> Accept: */*
> User-Agent: URL/Emacs Emacs/26.2 (X11; x86_64-pc-linux-gnu)
> Cookie: OIDC=1; ClientId=8998C5691CD143E784857A0D01537963
> Authorization: Basic ZGF2aWQuYy5kYXlAaHAuY29tOlMxa3kzbGk3bmUwNzMxJSU=
>
> and the results buffer gives: *http....*-#####
> HTTP/1.1 200 OK
> Cache-Control: private
> Content-Length: 1213
> Content-Type: text/html; charset=UTF-8
> Content-Encoding: gzip
> Vary: Accept-Encoding
> Server: Microsoft-IIS/10.0
> request-id: bcd31568-29fd-44e4-935f-35b54d697f33
> X-CalculatedFETarget: CY4PR18CU003.internal.outlook.com
> X-BackEndHttpStatus: 200
> Set-Cookie: exchangecookie=6c731ced364846f2a979bfaa84496f1e;
> expires=Sun, 14-Jun-2020 23:02:57 GMT; path=/; secure; HttpOnly
> X-FEProxyInfo: CY4PR18CA0059.NAMPRD18.PROD.OUTLOOK.COM
> X-CalculatedBETarget: CS1PR8401MB1223.NAMPRD84.PROD.OUTLOOK.COM
> X-BackEndHttpStatus: 200
> X-RUM-Validated: 1
> X-AspNet-Version: 4.0.30319
> X-BeSku: Gen9
> X-DiagInfo: CS1PR8401MB1223
> X-BEServer: CS1PR8401MB1223
> X-FEServer: CY4PR18CA0059
> X-Powered-By: ASP.NET
> X-FEServer: SN4PR0501CA0003
> Date: Fri, 14 Jun 2019 23:02:56 GMT
> Connection: close
>
> <HTML><HEAD><link rel="alternate" type="text/xml"
> href="https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Exchange.asmx?disco"/><STYLE
> type="text/css">#content{ FONT-SIZE: 0.7em; PADDING-BOTTOM: 2em;
> MARGIN-LEFT: 30px}BODY{MARGIN-TOP: 0px; MARGIN-LEFT: 0px; COLOR:
> #000000; FONT-FAMILY: Verdana; BACKGROUND-COLOR: white}P{MARGIN-TOP:
> 0px; MARGIN-BOTTOM: 12px; COLOR: #000000; FONT-FAMILY:
> Verdana}PRE{BORDER-RIGHT: #f0f0e0 1px solid; PADDING-RIGHT: 5px;
> BORDER-TOP: #f0f0e0 1px solid; MARGIN-TOP: -5px; PADDING-LEFT: 5px;
> FONT-SIZE: 1.2em; PADDING-BOTTOM: 5px; BORDER-LEFT: #f0f0e0 1px solid;
> PADDING-TOP: 5px; BORDER-BOTTOM: #f0f0e0 1px solid; FONT-FAMILY:
> Courier New; BACKGROUND-COLOR: #e5e5cc}.heading1{MARGIN-TOP: 0px;
> PADDING-LEFT: 15px; FONT-WEIGHT: normal; FONT-SIZE: 26px;
> MARGIN-BOTTOM: 0px; PADDING-BOTTOM: 3px; MARGIN-LEFT: -30px; WIDTH:
> 100%; COLOR: #ffffff; PADDING-TOP: 10px; FONT-FAMILY: Tahoma;
> BACKGROUND-COLOR: #003366}.intro{MARGIN-LEFT:
> -15px}</STYLE><TITLE>Service</TITLE></HEAD><BODY><DIV id="content"><P
> class="heading1">Service</P><BR/><P class="intro">You have created a
> service.<P class='intro'>To test this service, you will need to create
> a client and use it to call the service. You can do this using the
> svcutil.exe tool from the command line with the following syntax:</P>
> <BR/><PRE>svcutil.exe <A
> HREF="https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Services.wsdl">https://cs1pr8401mb1223.namprd84.prod.outlook.com:444/EWS/Services.wsdl</A></PRE></P><P
> class="intro"/>This will generate a configuration file and a code file
> that contains the client class. Add the two files to your client
> application and use the generated client class to call the Service.
> For example:<BR/><P class='intro'><B>C#</B></P><PRE><font
> color="blue">class </font><font color="teal">Test
> </font>{
> <font color="blue">    static void </font>Main()
>     {
>         <font color="teal">HelloClient</font> client = <font
> color="blue">new </font><font color="teal">HelloClient</font>();
>
> <font color="green">        // Use the 'client' variable to call
> operations on the service.
>
> </font><font color="green">        // Always close the client.
> </font>        client.Close();
>     }
> }
> </PRE><BR/><P class='intro'><B>Visual Basic</B></P><PRE><font
> color="blue">Class </font><font color="teal">Test
> </font><font color="blue">    Shared Sub </font>Main()
> <font color="blue">        Dim </font>client As <font
> color="teal">HelloClient</font> = <font color="blue">New </font><font
> color="teal">HelloClient</font>()
> <font color="green">        ' Use the 'client' variable to call
> operations on the service.
>
> </font><font color="green">        ' Always close the client.
> </font>        client.Close()
> <font color="blue">    End Sub
> </font><font color="blue">End Class</font></PRE></DIV></BODY></HTML>
>
> which matches the file saved with wget.
>
> That has to be worth something....
>
> Thanks!
>
> -C
>
> On Fri, Jun 14, 2019 at 4:07 PM tenspd137 . <[email-masked]> wrote:
> >
> > Just some more things I noticed from the wget log vs. the emacs buffers:
> >
> > emacs: Accept-encoding:gzip, wget:Accept-encoding: identity
> >
> > emacs: GET  https://outlook.office365.com/EWS/Exchange.asmx HTTP/1.1,
> > wget: GET /EWS/Exchange.asmx HTTP/1.1
> >
> > Don't know if any of that is helpful, but there it is.
> >
> > Thanks!
> >
> > -C
> >
> > On Fri, Jun 14, 2019 at 3:48 PM tenspd137 . <[email-masked]> wrote:
> > >
> > > Sorry - I am not in control of my proxy.  So, before I give up on this
> > > for today, I was going over the wget logs from trying the same thing.
> > > One thing I noticed is:
> > >
> > > Connection: Keep-Alive
> > > Proxy-Connection: Keep-Alive
> > >
> > > I see stuff in the emacs buffers mentioning Connection, and it says
> > > closed (unless marking the connection as busy is the same as keeping
> > > it open), but I don't see anything mentioning the proxy
> > > connection....?  Maybe another place to look?
> > >
> > > Just some ideas...
> > >
> > > Thanks!
> > >
> > > -C
> > >
> > > On Fri, Jun 14, 2019 at 3:02 PM Thomas Fitzsimmons <[email-masked]> wrote:
> > > >
> > > > Hi,
> > > >
> > > > Thanks for following up with these further test results.
> > > >
> > > > "tenspd137 ." <[email-masked]> writes:
> > > >
> > > > > I was able to try stepping through an Emacs/proxy/Exchange test in an
> > > > > emacs -Q session.  After setting the proxy and configuring the
> > > > > debugger to step through url-http and url-http-async-sentilnel, the
> > > > > only thing I noticed is that it appears url-http-async-sentinel is not
> > > > > being called.
> > > >
> > > > OK, that's probably expected.  I listed it for completeness (all the
> > > > call sites of url-https-proxy-connect), but I guess it would only be
> > > > called under url-retrieve, not under url-retrieve-synchronously.
> > > >
> > > > > I also put the url-https-proxy-connect override you gave me earlier
> > > > > into a file, loaded it and set the debugger to run through that as
> > > > > well as set up proxies.  A broken down list of steps:
> > > > >
> > > > > 1.  Load file containing proxy and altered url-https-proxyconnect, set
> > > > > debugger to run through it when hit.
> > > > > 2.  set up url-http and url-http-async-sentinel to be picked up by debugger
> > > > > 3.  eval (url-retrieve-synchronously
> > > > > "https://outlook.office365.com/EWS/Exchange.asmx"), step through
> > > > > url-http
> > > > > 4.  Input username and password when asked
> > > > > 5.  Continue stepping until end
> > > > >
> > > > > url-http-async-sentinel is never called. " *http* ... -####" has text
> > > > > indicating failure. url-https-proxy-connect is indeed called.
> > > >
> > > > OK, it's good to know that url-http ultimately calls
> > > > url-https-proxy-connect.  Unfortunately, despite this, proxying does not
> > > > work.
> > > >
> > > > > I don't know how to look at the actual <process>, if I try to C-x C-e
> > > > > or M-: "connection", it goes into the debugger.  The url's, etc look
> > > > > good as far as I can tell.  Not sure what else I can do.  If there are
> > > > > certain pieces of url-http you want me to look at, just let me know,
> > > > > but not really knowing what has to happen under the hood, I am not
> > > > > going to be able to do much else.
> > > >
> > > > I think I'll probably have to set up my own test environment.  Are you
> > > > in control of the proxy and its configuration?  If so, can you provide
> > > > rough configuration instructions (proxy software, version, relevant
> > > > config settings)?  If not, that's OK, I can try setting up generic proxy
> > > > software.
> > > >
> > > > Thanks,
> > > > Thomas