From debbugs-submit-bounces@debbugs.gnu.org Wed Feb 08 23:54:55 2012 Received: (at submit) by debbugs.gnu.org; 9 Feb 2012 04:54:55 +0000 Received: from localhost ([127.0.0.1]:33121 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvM21-0000MH-J0 for submit@debbugs.gnu.org; Wed, 08 Feb 2012 23:54:55 -0500 Received: from eggs.gnu.org ([140.186.70.92]:60926) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvM1x-0000M3-E0 for submit@debbugs.gnu.org; Wed, 08 Feb 2012 23:54:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RvM0l-0003gK-8y for submit@debbugs.gnu.org; Wed, 08 Feb 2012 23:53:38 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([140.186.70.17]:37717) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RvM0l-0003gA-2O for submit@debbugs.gnu.org; Wed, 08 Feb 2012 23:53:35 -0500 Received: from eggs.gnu.org ([140.186.70.92]:51314) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RvM0i-0005yU-PM for bug-gnu-emacs@gnu.org; Wed, 08 Feb 2012 23:53:34 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RvM0e-0003db-Qz for bug-gnu-emacs@gnu.org; Wed, 08 Feb 2012 23:53:32 -0500 Received: from mail237c25.carrierzone.com ([64.29.147.233]:57231) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RvM0e-0003bz-0K for bug-gnu-emacs@gnu.org; Wed, 08 Feb 2012 23:53:28 -0500 X-POP-User: cjwhite.grierwhite.com Received: from unknown (pool-108-49-4-207.bstnma.east.verizon.net [108.49.4.207]) by mail237c25.carrierzone.com (8.13.6/8.13.1) with ESMTP id q194rLRo008420 for ; Thu, 9 Feb 2012 04:53:22 GMT Date: Wed, 8 Feb 2012 23:53:21 -0500 From: "Christopher J. White" To: bug-gnu-emacs@gnu.org Subject: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Message-ID: <20120208235321.00003609@unknown> X-Mailer: Claws Mail 3.8.0cvs6 (GTK+ 2.16.6; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-CSC: 0 X-CHA: v=1.1 cv=qdcj7FnDz3iMLgEWBqV3+jpNlV01afZ0RwhqZvBJZRE= c=1 sm=1 a=euseRXvtOwMA:10 a=3pyGGIT-qAEA:10 a=IkcTkHD0fZMA:10 a=iW5S9OHVd49uhICS+jddKg==:17 a=G0BRwk1oAAAA:8 a=lfE3qB7g5qHWJGPjs7kA:9 a=LzhmfEsTCxz2ayXrZB4A:7 a=QEXdDO2ut3YA:10 a=8-SRZUljcj1cYEcQ:21 a=-0uOsexy1oTl1_WT:21 a=iW5S9OHVd49uhICS+jddKg==:117 X-CTCH-Spam: Unknown X-CTCH-RefID: str=0001.0A020208.4F335143.0099,ss=1,re=0.000,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.2 (----) I=E2=80=99m using url-http to retrieve an XML response. In noticed somewha= t by chance that certain page sizes were cutting off the last few characters of the response, which in turn caused the XML to be poorly formed. I've noticed the issue occurs when the payload response has to be sent in two TCP packets and the last packet has 8 or fewer bytes. I observe this by looking at the HTTP stream in Wireshark. I spent time tracing the response by adding some debug using url-http-debug. The first chunk of data comes from the first TCP packet and includes the HTTP header and all the XML data minus the last 8 bytes. url-http-wait-for-headers-change-function is called to process this chunk of data. After processing this chunk, the connection is closed and the buffer of XML data returned minux the 8 bytes. Here's the http trace, along with my added debug messages (inlcuding dumping of the full data): ---------------------------------------- 22:06.781000 http -> Contacting host: api.toodledo.com:80 22:06.827000 http -> Marking connection as busy: api.toodledo.com:80 #> 22:06.828000 http -> Request is:=20 POST /2/tasks/get.php HTTP/1.1=20 MIME-Version: 1.0=20 Connection: keep-alive=20 Extension: Security/Digest Security/SSL=20 Host: api.toodledo.com=20 Accept-charset: nil=20 Accept: */*=20 User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit)=20 Content-Type: application/x-www-form-urlencoded=20 Content-length: 212=20 =20 f=3Dxml&key=3D9f2a92de083e8d63c2bd14771661139f&unix=3D1&fields=3Dstatus%2cr= epeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2c= folder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=3D%2d1&modafter=3D0= =20 22:06.828000 retrieval -> Spinning in url-retrieve-synchronously: nil (#>) 22:06.889000 http -> Calling after change function `url-http-wait-for-heade= rs-change-function' for `#>' 22:06.889000 http -> data (point-max 1) (len 1380): "HTTP/1.1 200 OK=20 Date: Thu, 09 Feb 2012 04:22:06 GMT=20 Server: Apache/2.2.3 (Red Hat)=20 Content-Length: 1189=20 Keep-Alive: timeout=3D2, max=3D100=20 Connection: Keep-Alive=20 Content-Type: application/xml=20 =20 = 9105869Test 11328482938<= completed>132792480000000<= duedate>1328097600013281372000029108309Test 2 -foobar132= 87613220000000000029149333org-toodledo1328011949000020<= duedate>0000= 02 url-http-wait-for-headers-change-function ( *http api.= toodledo.com:80*<80>) 22:06.890000 http -> Saw end of headers... ( *http api.toodledo.com:80*<80>= ) @ 197 22:06.890000 http -> url-http-clean-headers, end-of-headers now at 189 22:06.890000 http -> url-http-parse-response called in ( *http api.toodledo= .com:80*<80>) 22:06.890000 http -> Got a content-length 1189, being smart about document = end. 22:06.890000 http -> Calling initial content-length for extra data at end o= f headers 22:06.890000 http -> url-http-content-length-after-change-function 189 1381= 1192 22:06.891000 http -> End of document found! 22:06.891000 http -> Marking connection as free: api.toodledo.com:80 #> 22:06.891000 http -> url-http-parse-headers called in ( *http api.toodledo.= com:80*<80>) 22:06.891000 http -> url-http-parse-response called in ( *http api.toodledo= .com:80*<80>) 22:06.891000 http -> Parsed HTTP headers: class=3D2 status=3D200 22:06.891000 http -> Finished parsing HTTP headers: t 22:06.891000 http -> Marking connection as free: api.toodledo.com:80 #> 22:06.891000 http -> Activating callback in buffer ( *http api.toodledo.com= :80*<80>) ---------------------------------------- Notice that the XML in the data finishes with "...". =20 A key debug message I added is "End of document found!", and this is placed in the function url-http-content-length-after-change-function right after the comment "Found the end of the document! Wheee!": (defun url-http-content-length-after-change-function (st nd length) <....> (if (> (- nd url-http-end-of-headers) url-http-content-length) (progn ;; Found the end of the document! Wheee! (url-http-debug "End of document found!") (url-display-percentage nil nil) (url-lazy-message "Reading... done.") (if (url-http-parse-headers) (url-http-activate-callback))))) If I slightly increase or decrease the amount of data returned by the XML (by adding or deleting characters from title), I can get it to work properly or not. =20 For example, by adding 5 chars, I get: ---------------------------------------- 39:20.144000 http -> Finding free connection: api.toodledo.com:80 39:20.144000 http -> Contacting host: api.toodledo.com:80 39:20.187000 http -> Marking connection as busy: api.toodledo.com:80 #> 39:20.188000 http -> Request is:=20 POST /2/tasks/get.php HTTP/1.1=20 MIME-Version: 1.0=20 Connection: keep-alive=20 Extension: Security/Digest Security/SSL=20 Host: api.toodledo.com=20 Accept-charset: nil=20 Accept: */*=20 User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit)=20 Content-Type: application/x-www-form-urlencoded=20 Content-length: 212=20 =20 f=3Dxml&key=3Dxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&unix=3D1&fields=3Dstatus%2cr= epeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2c= folder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=3D%2d1&modafter=3D0= =20 39:20.188000 retrieval -> Spinning in url-retrieve-synchronously: nil (#>) 39:20.242000 http -> Calling after change function `url-http-wait-for-heade= rs-change-function' for `#>' 39:20.243000 http -> data (point-max 1) (len 1380): "HTTP/1.1 200 OK=20 Date: Thu, 09 Feb 2012 04:39:20 GMT=20 Server: Apache/2.2.3 (Red Hat)=20 Content-Length: 1194=20 Keep-Alive: timeout=3D2, max=3D100=20 Connection: Keep-Alive=20 Content-Type: application/xml=20 =20 = 9105869Test 11328482938<= completed>132792480000000<= duedate>1328097600013281372000029108309Test 2 -foobar123451328762353000= 00= 00000029149333org-toodledo13280119= 49000= 020000002 url-http-wait-for-headers-change-function ( *http api.= toodledo.com:80*<86>) 39:20.243000 http -> Saw end of headers... ( *http api.toodledo.com:80*<86>= ) @ 197 39:20.243000 http -> url-http-clean-headers, end-of-headers now at 189 39:20.243000 http -> url-http-parse-response called in ( *http api.toodledo= .com:80*<86>) 39:20.243000 http -> Got a content-length 1194, being smart about document = end. 39:20.243000 http -> Calling initial content-length for extra data at end o= f headers 39:20.243000 http -> url-http-content-length-after-change-function 189 1381= 1192 39:20.244000 http -> Spinning waiting for headers... 39:20.244000 retrieval -> Spinning in url-retrieve-synchronously: nil (#>) 39:20.247000 http -> Calling after change function `url-http-content-length= -after-change-function' for `#>' 39:20.247000 http -> data (point-max 1373) (len 11): "sk>" 39:20.247000 http -> url-http-content-length-after-change-function 1373 138= 4 11 39:20.247000 http -> End of document found! 39:20.247000 http -> Marking connection as free: api.toodledo.com:80 #> 39:20.247000 http -> url-http-parse-headers called in ( *http api.toodledo.= com:80*<86>) 39:20.247000 http -> url-http-parse-response called in ( *http api.toodledo= .com:80*<86>) 39:20.247000 http -> Parsed HTTP headers: class=3D2 status=3D200 39:20.248000 http -> Finished parsing HTTP headers: t 39:20.248000 http -> Marking connection as free: api.toodledo.com:80 #> 39:20.248000 http -> Activating callback in buffer ( *http api.toodledo.com= :80*<86>) ---------------------------------------- In this case, url-http waits for the remaining data from the second TCP packet, which comes in slightly later and is pieced together and forms a good XML document. If I reduce the size by an appropriate number of bytes so that the XML data fits entirely in the first packet it also works.=20 I think the problem relates to url-http-clean-headers. I have 8-lines of headers, and this function removes the trailing \r from each of those lines. I think this is not accounted for when using the content-length to determine if all the data has been received. After dropping these 8 trailing \r charaters, the buffer positions all change. =20 I'm not quite able to figure out how the content-length is used to detect the end of processing, so I thought I'd submit this bug and see if any other folks can point me in the right direction. ...cj In GNU Emacs 23.3.1 (i386-mingw-nt6.1.7601) of 2011-03-10 on 3249CTO Windowing system distributor `Microsoft Corp.', version 6.1.7601 configured using `configure --with-gcc (4.5) --no-opt --cflags -Ic:/imagesu= pport/include' Important settings: value of $LC_ALL: nil value of $LC_COLLATE: nil value of $LC_CTYPE: nil value of $LC_MESSAGES: nil value of $LC_MONETARY: nil value of $LC_NUMERIC: nil value of $LC_TIME: nil value of $LANG: C.UTF-8 value of $XMODIFIERS: nil locale-coding-system: cp1252 default enable-multibyte-characters: t Major mode: Fundamental Minor modes in effect: show-paren-mode: t display-time-mode: t tooltip-mode: t mouse-wheel-mode: t global-font-lock-mode: t auto-encryption-mode: t auto-compression-mode: t column-number-mode: t line-number-mode: t transient-mark-mode: t Recent input: - - - - - - - - - - - - - - - - =20 C-x=20 C-/ C-/ C-/ C-/ SPC C-/=20 C-/ C-a C-x [ C-x ] C-x [ C-/ C-/ C-/ C-x=20 [ C-SPC C-x ] C-w C-/ C-/ C-/ C-/ C-/ C-/ C-/ C-/ C-a=20 C-o s =20 C-SPC C-x ] =20 =20 =20 =20 =20 C-w =20 C-x 2 =20 =20 C-s C-w C-w C-w C-w C-w C-w C-w C-x=20 o C-s C-s C-a C-a C-l C-n C-n C-n C-n C-n=20 C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n =20 C-x o C-p C-n C-n C-n C-n C-n=20 C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-n C-l=20 C-n C-n C-l C-n C-n C-n C-n C-n C-l C-n C-n C-n C-n=20 C-l C-x o C-n C-n C-n =20 =20 =20 C-o s =20 C-x ] =20 =20 SPC C-x =20 C-x o C-o s =20 C-x ] C-r t a s k s C-r=20 C-r C-r C-s C-a =20 =20 x r e p o =20 r t SPC e m SPC SPC Recent messages: Reading [application/xml=20 ]... 1k of 1k (100%) Contacting host: api.toodledo.com:80 Reading [application/xml=20 ]... 718 bytes of 717 bytes (100%) Sync complete, 794 changes: recv 3 mod 791 del=20 Mark set Mark saved where search started Auto-saving...done Making completion list... Load-path shadows: ~cwhite/lib/lisp/org-toodledo/w3mexcerpt hides ~cwhite/lib/lisp/w3mexcerpt ~cwhite/lib/lisp/org-toodledo/http-post-simple hides ~cwhite/lib/lisp/http-= post-simple ~/lib/lisp/org-mode/lisp/org hides c:/Users/cwhite/emacs-23.3/lisp/org/org ~/lib/lisp/org-mode/lisp/org-xoxo hides c:/Users/cwhite/emacs-23.3/lisp/org= /org-xoxo ~/lib/lisp/org-mode/lisp/org-wl hides c:/Users/cwhite/emacs-23.3/lisp/org/o= rg-wl ~/lib/lisp/org-mode/lisp/org-w3m hides c:/Users/cwhite/emacs-23.3/lisp/org/= org-w3m ~/lib/lisp/org-mode/lisp/org-vm hides c:/Users/cwhite/emacs-23.3/lisp/org/o= rg-vm ~/lib/lisp/org-mode/lisp/org-timer hides c:/Users/cwhite/emacs-23.3/lisp/or= g/org-timer ~/lib/lisp/org-mode/lisp/org-table hides c:/Users/cwhite/emacs-23.3/lisp/or= g/org-table ~/lib/lisp/org-mode/lisp/org-src hides c:/Users/cwhite/emacs-23.3/lisp/org/= org-src ~/lib/lisp/org-mode/lisp/org-rmail hides c:/Users/cwhite/emacs-23.3/lisp/or= g/org-rmail ~/lib/lisp/org-mode/lisp/org-remember hides c:/Users/cwhite/emacs-23.3/lisp= /org/org-remember ~/lib/lisp/org-mode/lisp/org-publish hides c:/Users/cwhite/emacs-23.3/lisp/= org/org-publish ~/lib/lisp/org-mode/lisp/org-protocol hides c:/Users/cwhite/emacs-23.3/lisp= /org/org-protocol ~/lib/lisp/org-mode/lisp/org-plot hides c:/Users/cwhite/emacs-23.3/lisp/org= /org-plot ~/lib/lisp/org-mode/lisp/org-mouse hides c:/Users/cwhite/emacs-23.3/lisp/or= g/org-mouse ~/lib/lisp/org-mode/lisp/org-mobile hides c:/Users/cwhite/emacs-23.3/lisp/o= rg/org-mobile ~/lib/lisp/org-mode/lisp/org-mhe hides c:/Users/cwhite/emacs-23.3/lisp/org/= org-mhe ~/lib/lisp/org-mode/lisp/org-mew hides c:/Users/cwhite/emacs-23.3/lisp/org/= org-mew ~/lib/lisp/org-mode/lisp/org-macs hides c:/Users/cwhite/emacs-23.3/lisp/org= /org-macs ~/lib/lisp/org-mode/lisp/org-mac-message hides c:/Users/cwhite/emacs-23.3/l= isp/org/org-mac-message ~/lib/lisp/org-mode/lisp/org-list hides c:/Users/cwhite/emacs-23.3/lisp/org= /org-list ~/lib/lisp/org-mode/lisp/org-latex hides c:/Users/cwhite/emacs-23.3/lisp/or= g/org-latex ~/lib/lisp/org-mode/lisp/org-jsinfo hides c:/Users/cwhite/emacs-23.3/lisp/o= rg/org-jsinfo ~/lib/lisp/org-mode/lisp/org-irc hides c:/Users/cwhite/emacs-23.3/lisp/org/= org-irc ~/lib/lisp/org-mode/lisp/org-install hides c:/Users/cwhite/emacs-23.3/lisp/= org/org-install ~/lib/lisp/org-mode/lisp/org-inlinetask hides c:/Users/cwhite/emacs-23.3/li= sp/org/org-inlinetask ~/lib/lisp/org-mode/lisp/org-info hides c:/Users/cwhite/emacs-23.3/lisp/org= /org-info ~/lib/lisp/org-mode/lisp/org-indent hides c:/Users/cwhite/emacs-23.3/lisp/o= rg/org-indent ~/lib/lisp/org-mode/lisp/org-id hides c:/Users/cwhite/emacs-23.3/lisp/org/o= rg-id ~/lib/lisp/org-mode/lisp/org-icalendar hides c:/Users/cwhite/emacs-23.3/lis= p/org/org-icalendar ~/lib/lisp/org-mode/lisp/org-html hides c:/Users/cwhite/emacs-23.3/lisp/org= /org-html ~/lib/lisp/org-mode/lisp/org-habit hides c:/Users/cwhite/emacs-23.3/lisp/or= g/org-habit ~/lib/lisp/org-mode/lisp/org-gnus hides c:/Users/cwhite/emacs-23.3/lisp/org= /org-gnus ~/lib/lisp/org-mode/lisp/org-freemind hides c:/Users/cwhite/emacs-23.3/lisp= /org/org-freemind ~/lib/lisp/org-mode/lisp/org-footnote hides c:/Users/cwhite/emacs-23.3/lisp= /org/org-footnote ~/lib/lisp/org-mode/lisp/org-feed hides c:/Users/cwhite/emacs-23.3/lisp/org= /org-feed ~/lib/lisp/org-mode/lisp/org-faces hides c:/Users/cwhite/emacs-23.3/lisp/or= g/org-faces ~/lib/lisp/org-mode/lisp/org-exp hides c:/Users/cwhite/emacs-23.3/lisp/org/= org-exp ~/lib/lisp/org-mode/lisp/org-exp-blocks hides c:/Users/cwhite/emacs-23.3/li= sp/org/org-exp-blocks ~/lib/lisp/org-mode/lisp/org-docbook hides c:/Users/cwhite/emacs-23.3/lisp/= org/org-docbook ~/lib/lisp/org-mode/lisp/org-datetree hides c:/Users/cwhite/emacs-23.3/lisp= /org/org-datetree ~/lib/lisp/org-mode/lisp/org-crypt hides c:/Users/cwhite/emacs-23.3/lisp/or= g/org-crypt ~/lib/lisp/org-mode/lisp/org-compat hides c:/Users/cwhite/emacs-23.3/lisp/o= rg/org-compat ~/lib/lisp/org-mode/lisp/org-colview hides c:/Users/cwhite/emacs-23.3/lisp/= org/org-colview ~/lib/lisp/org-mode/lisp/org-clock hides c:/Users/cwhite/emacs-23.3/lisp/or= g/org-clock ~/lib/lisp/org-mode/lisp/org-bibtex hides c:/Users/cwhite/emacs-23.3/lisp/o= rg/org-bibtex ~/lib/lisp/org-mode/lisp/org-bbdb hides c:/Users/cwhite/emacs-23.3/lisp/org= /org-bbdb ~/lib/lisp/org-mode/lisp/org-attach hides c:/Users/cwhite/emacs-23.3/lisp/o= rg/org-attach ~/lib/lisp/org-mode/lisp/org-ascii hides c:/Users/cwhite/emacs-23.3/lisp/or= g/org-ascii ~/lib/lisp/org-mode/lisp/org-archive hides c:/Users/cwhite/emacs-23.3/lisp/= org/org-archive ~/lib/lisp/org-mode/lisp/org-agenda hides c:/Users/cwhite/emacs-23.3/lisp/o= rg/org-agenda Features: (shadow sort mail-extr message ecomplete rfc822 mml mml-sec password-cache mm-decode mm-bodies mm-encode mailabbrev nnheader gnus-util netrc gmm-utils mailheader canlock sha1 hex-util hashcash emacsbug sgml-mode cl-specs edebug find-func vc-git nxml-uchnm rng-xsd xsd-regexp rng-cmpct rng-nxml rng-valid rng-loc rng-uri rng-parse nxml-parse rng-match rng-dt rng-util rng-pttrn nxml-ns nxml-mode nxml-outln nxml-rap nxml-util nxml-glyph nxml-enc xmltok browse-url cperl-mode url-cache org-colview diary-lib diary-loaddefs cal-iso org-table bookmark pp image-file mule-util grep sh-script executable tabify org-ascii help-mode view org-wl org-w3m org-vm org-rmail org-mhe org-mew org-irc org-jsinfo org-infojs org-html format-spec org-exp ob-exp org-exp-blocks org-agenda org-info org-gnus org-docview org-bibtex bibtex org-bbdb multi-isearch newcomment org-toodledo http-post-simple url-http tls url-auth mail-parse rfc2231 rfc2047 rfc2045 qp ietf-drums url-gw url url-proxy url-privacy url-expand url-methods url-history url-cookie url-util url-parse url-vars mm-util mail-prsvr mailcap json xml w3mexcerpt remember org-remember org-datetree org ob-emacs-lisp ob-tangle ob-ref ob-lob ob-table org-footnote org-src ob-comint ob-keys ob ob-eval org-pcomplete pcomplete org-list org-faces org-compat org-entities org-macs time-date noutline outline easy-mmode cal-menu calendar cal-loaddefs cygwin-mount ange-ftp uniquify filecache server byte-opt warnings advice help-fns advice-preload paren time etags python-mode info-look css-mode scr rsync twiki edmacro kmacro info doku pib php-mode speedbar sb-image ezimage dframe assoc highline pop3 mail-utils cl cl-19 cc-mode cc-fonts cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs regexp-opt cus-edit easymenu cus-start cus-load wid-edit exec compile comint ring bytecomp byte-compile tooltip ediff-hook vc-hooks lisp-float-type mwheel dos-w32 disp-table ls-lisp w32-win w32-vars tool-bar dnd fontset image fringe lisp-mode register page menu-bar rfn-eshadow timer select scroll-bar mldrag mouse jit-lock font-lock syntax facemenu font-core frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev loaddefs button minibuffer faces cus-face files text-properties overlay md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote make-network-process multi-tty emacs) From debbugs-submit-bounces@debbugs.gnu.org Thu Feb 09 04:17:03 2012 Received: (at 10768) by debbugs.gnu.org; 9 Feb 2012 09:17:03 +0000 Received: from localhost ([127.0.0.1]:33227 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvQ7g-0008FZ-L4 for submit@debbugs.gnu.org; Thu, 09 Feb 2012 04:17:01 -0500 Received: from mail-out.m-online.net ([212.18.0.10]:43219) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvQ7d-0008FN-8E for 10768@debbugs.gnu.org; Thu, 09 Feb 2012 04:16:59 -0500 Received: from frontend1.mail.m-online.net (frontend1.mail.intern.m-online.net [192.168.8.180]) by mail-out.m-online.net (Postfix) with ESMTP id EC55918000C9; Thu, 9 Feb 2012 10:15:48 +0100 (CET) X-Auth-Info: mN0WbObgKddscO1T36E8V9fu1SznhKYRtKd3Pz09m1Y= Received: from igel.home (ppp-88-217-114-208.dynamic.mnet-online.de [88.217.114.208]) by mail.mnet-online.de (Postfix) with ESMTPA id DA1F21C00075; Thu, 9 Feb 2012 10:15:48 +0100 (CET) Received: by igel.home (Postfix, from userid 501) id EBD36CA299; Thu, 9 Feb 2012 10:15:47 +0100 (CET) From: Andreas Schwab To: "Christopher J. White" Subject: Re: bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used References: <20120208235321.00003609@unknown> X-Yow: Four thousand different MAGNATES, MOGULS & NABOBS are romping in my gothic solarium!! Date: Thu, 09 Feb 2012 10:15:47 +0100 In-Reply-To: <20120208235321.00003609@unknown> (Christopher J. White's message of "Wed, 8 Feb 2012 23:53:21 -0500") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.93 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 10768 Cc: 10768@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) "Christopher J. White" writes: > "HTTP/1.1 200 OK=20 > Date: Thu, 09 Feb 2012 04:22:06 GMT=20 > Server: Apache/2.2.3 (Red Hat)=20 > Content-Length: 1189=20 ^^^^ > Keep-Alive: timeout=3D2, max=3D100=20 > Connection: Keep-Alive=20 > Content-Type: application/xml=20 >=20=20 > 9105869Test 11328482938132792480000000132809760001328137200002<= /note>9108309Test 2 -foobar1= 328761322000000000= 0029149333org-toodledo1328011949<= /modified>000020000002) id 1RvTSp-0006Ol-TG for submit@debbugs.gnu.org; Thu, 09 Feb 2012 07:51:06 -0500 Received: from mail13c25.carrierzone.com ([64.29.147.23]:57474) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvTSj-0006OF-1d for 10768@debbugs.gnu.org; Thu, 09 Feb 2012 07:51:01 -0500 X-POP-User: cjwhite.grierwhite.com Received: from unknown (pool-108-49-4-207.bstnma.east.verizon.net [108.49.4.207]) by mail13c25.carrierzone.com (8.13.6/8.13.1) with ESMTP id q19CniV2027159; Thu, 9 Feb 2012 12:49:47 GMT Date: Thu, 9 Feb 2012 07:49:33 -0500 From: "Christopher J. White" To: Andreas Schwab Subject: Re: bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Message-ID: <20120209074933.000047ef@unknown> In-Reply-To: References: <20120208235321.00003609@unknown> X-Mailer: Claws Mail 3.8.0cvs6 (GTK+ 2.16.6; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-CSC: 0 X-CHA: v=1.1 cv=N1q7tLlpVkgwgCgcIMmsDvmmpjdUzSJ6hE3/gn3DElE= c=1 sm=1 a=hdTLrLAGt2gA:10 a=3pyGGIT-qAEA:10 a=kj9zAlcOel0A:10 a=iW5S9OHVd49uhICS+jddKg==:17 a=tBb2bbeoAAAA:8 a=JJJ-DeXCAAAA:8 a=G0BRwk1oAAAA:8 a=TLsayHnw2igtNOCkYogA:9 a=NEYPxOaO2VozE6VtqKUA:7 a=CjuIK1q_8ugA:10 a=xQdB8fRqRzQA:10 a=YuKU6ANggZ8A:10 a=afs47w9HCcgA:10 a=jVKp2qe0AqUA:10 a=NWVoK91CQyQA:10 a=4dI1X88jbcPzcsmA:21 a=Yz3JGOi1g-it91eD:21 a=iW5S9OHVd49uhICS+jddKg==:117 X-CTCH-Spam: Unknown X-CTCH-RefID: str=0001.0A020201.4F33C0ED.0007,ss=1,re=0.000,fgs=0 X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 10768 Cc: 10768@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On Thu, 09 Feb 2012 10:15:47 +0100 Andreas Schwab wrote: > "Christopher J. White" writes: > > Content-Length: 1189 > ^^^^ > > > total='3'>9105869Test > > 113284829381327924800000001328097600013281372000029108309Test > > 2 > > -foobar13287613220000000000029149333org-toodledo1328011949000020000002 > 1189 bytes of data follow. > > That looks like a bug in the server. I considered that and looked hard for it to be the server. In the data output the double quote chars add an additional slash "\" that is not present in the actual data. The content-length and the actual number of raw bytes matches precisely when decoded from wireshark. I reran the test and collected the 3 sets of data, the recombined HTTM response as built by wireshark (TCP Follow Stream), the ascii dump of the wireshark packets (2 involved), and the URL-Debug. The key point is that the data in the second packet is never processed by url-http, only the bytes in the first packet. HTTP response as seen in Wireshark: ---------------------------------------- f=xml&key=444ed11d926131624b2ce2ed81f58ad4&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0 HTTP/1.1 200 OK Date: Thu, 09 Feb 2012 12:34:49 GMT Server: Apache/2.2.3 (Red Hat) Content-Length: 1189 Keep-Alive: timeout=2, max=100 Connection: Keep-Alive Content-Type: application/xml 9105869Test 113284829381327924800000001328097600013281372000029108309Test 2 -foobar13287908820000000000029149333org-toodledo1328011949000020000002 ---------------------------------------- There is no newline at the end, the last character is the ">" after tasks. The xml data is all a single line, there are no CR/LF characters any where in it, so if any show up in this message they should be removed if you're trying to count it out. Here's the wireshark decode for the 2 packets involved: ---------------------------------------- No. Time Source Destination Protocol Length Info 312 49.870066 72.4.112.215 11.1.1.89 TCP 1434 [TCP segment of a reassembled PDU] Frame 312: 1434 bytes on wire (11472 bits), 1434 bytes captured (11472 bits) Arrival Time: Feb 9, 2012 07:34:49.261829000 Eastern Standard Time Epoch Time: 1328790889.261829000 seconds [Time delta from previous captured frame: 0.008369000 seconds] [Time delta from previous displayed frame: 0.008369000 seconds] [Time since reference or first frame: 49.870066000 seconds] Frame Number: 312 Frame Length: 1434 bytes (11472 bits) Capture Length: 1434 bytes (11472 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ip:tcp] [Coloring Rule Name: HTTP] [Coloring Rule String: http || tcp.port == 80] Ethernet II, Src: WestellT_44:0a:ea (00:0f:db:44:0a:ea), Dst: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00) Destination: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00) Address: IntelCor_5e:fd:00 (00:21:6b:5e:fd:00) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Source: WestellT_44:0a:ea (00:0f:db:44:0a:ea) Address: WestellT_44:0a:ea (00:0f:db:44:0a:ea) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Type: IP (0x0800) Internet Protocol Version 4, Src: 72.4.112.215 (72.4.112.215), Dst: 11.1.1.89 (11.1.1.89) Version: 4 Header length: 20 bytes Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport)) 0000 00.. = Differentiated Services Codepoint: Default (0x00) .... ..00 = Explicit Congestion Notification: Not-ECT (Not ECN-Capable Transport) (0x00) Total Length: 1420 Identification: 0xe548 (58696) Flags: 0x02 (Don't Fragment) 0... .... = Reserved bit: Not set .1.. .... = Don't fragment: Set ..0. .... = More fragments: Not set Fragment offset: 0 Time to live: 55 Protocol: TCP (6) Header checksum: 0x93ee [correct] [Good: True] [Bad: False] Source: 72.4.112.215 (72.4.112.215) Destination: 11.1.1.89 (11.1.1.89) Transmission Control Protocol, Src Port: http (80), Dst Port: 60823 (60823), Seq: 1, Ack: 527, Len: 1380 Source port: http (80) Destination port: 60823 (60823) [Stream index: 10] Sequence number: 1 (relative sequence number) [Next sequence number: 1381 (relative sequence number)] Acknowledgement number: 527 (relative ack number) Header length: 20 bytes Flags: 0x10 (ACK) 000. .... .... = Reserved: Not set ...0 .... .... = Nonce: Not set .... 0... .... = Congestion Window Reduced (CWR): Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgement: Set .... .... 0... = Push: Not set .... .... .0.. = Reset: Not set .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set Window size value: 54 [Calculated window size: 6912] [Window size scaling factor: 128] Checksum: 0x836f [validation disabled] [Good Checksum: False] [Bad Checksum: False] [SEQ/ACK analysis] [Bytes in flight: 1380] TCP segment data (1380 bytes) 0000 00 21 6b 5e fd 00 00 0f db 44 0a ea 08 00 45 00 .!k^.....D....E. 0010 05 8c e5 48 40 00 37 06 93 ee 48 04 70 d7 0b 01 ...H@.7...H.p... 0020 01 59 00 50 ed 97 30 cd 5f f0 eb ef 69 24 50 10 .Y.P..0._...i$P. 0030 00 36 83 6f 00 00 48 54 54 50 2f 31 2e 31 20 32 .6.o..HTTP/1.1 2 0040 30 30 20 4f 4b 0d 0a 44 61 74 65 3a 20 54 68 75 00 OK..Date: Thu 0050 2c 20 30 39 20 46 65 62 20 32 30 31 32 20 31 32 , 09 Feb 2012 12 0060 3a 33 34 3a 34 39 20 47 4d 54 0d 0a 53 65 72 76 :34:49 GMT..Serv 0070 65 72 3a 20 41 70 61 63 68 65 2f 32 2e 32 2e 33 er: Apache/2.2.3 0080 20 28 52 65 64 20 48 61 74 29 0d 0a 43 6f 6e 74 (Red Hat)..Cont 0090 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 31 31 38 39 ent-Length: 1189 00a0 0d 0a 4b 65 65 70 2d 41 6c 69 76 65 3a 20 74 69 ..Keep-Alive: ti 00b0 6d 65 6f 75 74 3d 32 2c 20 6d 61 78 3d 31 30 30 meout=2, max=100 00c0 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 4b 65 ..Connection: Ke 00d0 65 70 2d 41 6c 69 76 65 0d 0a 43 6f 6e 74 65 6e ep-Alive..Conten 00e0 74 2d 54 79 70 65 3a 20 61 70 70 6c 69 63 61 74 t-Type: applicat 00f0 69 6f 6e 2f 78 6d 6c 0d 0a 0d 0a 3c 3f 78 6d 6c ion/xml.... 0140 3c 69 64 3e 39 31 30 35 38 36 39 3c 2f 69 64 3e 9105869 0150 3c 74 69 74 6c 65 3e 54 65 73 74 20 31 3c 2f 74 Test 1</t 0160 69 74 6c 65 3e 3c 6d 6f 64 69 66 69 65 64 3e 31 itle><modified>1 0170 33 32 38 34 38 32 39 33 38 3c 2f 6d 6f 64 69 66 328482938</modif 0180 69 65 64 3e 3c 63 6f 6d 70 6c 65 74 65 64 3e 31 ied><completed>1 0190 33 32 37 39 32 34 38 30 30 3c 2f 63 6f 6d 70 6c 327924800</compl 01a0 65 74 65 64 3e 3c 66 6f 6c 64 65 72 3e 30 3c 2f eted><folder>0</ 01b0 66 6f 6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74 3e folder><context> 01c0 30 3c 2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61 6c 0</context><goal 01d0 3e 30 3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72 69 >0</goal><priori 01e0 74 79 3e 30 3c 2f 70 72 69 6f 72 69 74 79 3e 3c ty>0</priority>< 01f0 72 65 70 65 61 74 3e 3c 2f 72 65 70 65 61 74 3e repeat></repeat> 0200 3c 72 65 70 65 61 74 66 72 6f 6d 3e 30 3c 2f 72 <repeatfrom>0</r 0210 65 70 65 61 74 66 72 6f 6d 3e 3c 64 75 65 64 61 epeatfrom><dueda 0220 74 65 3e 31 33 32 38 30 39 37 36 30 30 3c 2f 64 te>1328097600</d 0230 75 65 64 61 74 65 3e 3c 73 74 61 72 74 64 61 74 uedate><startdat 0240 65 3e 30 3c 2f 73 74 61 72 74 64 61 74 65 3e 3c e>0</startdate>< 0250 64 75 65 74 69 6d 65 3e 31 33 32 38 31 33 37 32 duetime>13281372 0260 30 30 3c 2f 64 75 65 74 69 6d 65 3e 3c 73 74 61 00</duetime><sta 0270 72 74 74 69 6d 65 3e 30 3c 2f 73 74 61 72 74 74 rttime>0</startt 0280 69 6d 65 3e 3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c ime><length>0</l 0290 65 6e 67 74 68 3e 3c 73 74 61 74 75 73 3e 32 3c ength><status>2< 02a0 2f 73 74 61 74 75 73 3e 3c 6e 6f 74 65 3e 3c 2f /status><note></ 02b0 6e 6f 74 65 3e 3c 2f 74 61 73 6b 3e 3c 74 61 73 note></task><tas 02c0 6b 3e 3c 69 64 3e 39 31 30 38 33 30 39 3c 2f 69 k><id>9108309</i 02d0 64 3e 3c 74 69 74 6c 65 3e 54 65 73 74 20 32 20 d><title>Test 2 02e0 2d 66 6f 6f 62 61 72 3c 2f 74 69 74 6c 65 3e 3c -foobar< 02f0 6d 6f 64 69 66 69 65 64 3e 31 33 32 38 37 39 30 modified>1328790 0300 38 38 32 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63 88200< 0330 2f 66 6f 6c 64 65 72 3e 3c 63 6f 6e 74 65 78 74 /folder>000 0370 3c 72 65 70 65 61 74 3e 3c 2f 72 65 70 65 61 74 00< 03b0 73 74 61 72 74 64 61 74 65 3e 30 3c 2f 73 74 61 startdate>0 03d0 30 3c 2f 64 75 65 74 69 6d 65 3e 3c 73 74 61 72 00029149333org-tood 0450 6c 65 64 6f 3c 2f 74 69 74 6c 65 3e 3c 6d 6f 64 ledo1328011949 0470 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63 6f 6d 70 000< 04b0 2f 63 6f 6e 74 65 78 74 3e 3c 67 6f 61 6c 3e 30 /context>0 04c0 3c 2f 67 6f 61 6c 3e 3c 70 72 69 6f 72 69 74 79 200000 0560 3c 6c 65 6e 67 74 68 3e 30 3c 2f 6c 65 6e 67 74 02 9105869 Test 1 1328482938 1327924800 0 0 0 0 0 1328097600 0 1328137200 0 0 2 9108309 Test 2 -foobar 1328790882 0 0 0 0 0 0 0 0 0 0 0 2 9149333 org-toodledo 1328011949 0 0 0 0 2 0 0 0 0 0 0 2 Frame (60 bytes): 0000 00 21 6b 5e fd 00 00 0f db 44 0a ea 08 00 45 00 .!k^.....D....E. 0010 00 2e e5 49 40 00 37 06 99 4b 48 04 70 d7 0b 01 ...I@.7..KH.p... 0020 01 59 00 50 ed 97 30 cd 65 54 eb ef 69 24 50 18 .Y.P..0.eT..i$P. 0030 00 36 b6 32 00 00 74 61 73 6b 73 3e .6.2..tasks> Reassembled TCP (1386 bytes): 0000 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK. 0010 0a 44 61 74 65 3a 20 54 68 75 2c 20 30 39 20 46 .Date: Thu, 09 F 0020 65 62 20 32 30 31 32 20 31 32 3a 33 34 3a 34 39 eb 2012 12:34:49 0030 20 47 4d 54 0d 0a 53 65 72 76 65 72 3a 20 41 70 GMT..Server: Ap 0040 61 63 68 65 2f 32 2e 32 2e 33 20 28 52 65 64 20 ache/2.2.3 (Red 0050 48 61 74 29 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 Hat)..Content-Le 0060 6e 67 74 68 3a 20 31 31 38 39 0d 0a 4b 65 65 70 ngth: 1189..Keep 0070 2d 41 6c 69 76 65 3a 20 74 69 6d 65 6f 75 74 3d -Alive: timeout= 0080 32 2c 20 6d 61 78 3d 31 30 30 0d 0a 43 6f 6e 6e 2, max=100..Conn 0090 65 63 74 69 6f 6e 3a 20 4b 65 65 70 2d 41 6c 69 ection: Keep-Ali 00a0 76 65 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 ve..Content-Type 00b0 3a 20 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 6d : application/xm 00c0 6c 0d 0a 0d 0a 3c 3f 78 6d 6c 20 76 65 72 73 69 l....91 0110 30 35 38 36 39 3c 2f 69 64 3e 3c 74 69 74 6c 65 05869Test 1< 0130 6d 6f 64 69 66 69 65 64 3e 31 33 32 38 34 38 32 modified>1328482 0140 39 33 38 3c 2f 6d 6f 64 69 66 69 65 64 3e 3c 63 9381327924 0160 38 30 30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e 3c 800< 0170 66 6f 6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65 72 folder>00000132 01f0 38 30 39 37 36 30 30 3c 2f 64 75 65 64 61 74 65 80976000132813720000 0260 3c 73 74 61 74 75 73 3e 32 3c 2f 73 74 61 74 75 2< 0280 2f 74 61 73 6b 3e 3c 74 61 73 6b 3e 3c 69 64 3e /task> 0290 39 31 30 38 33 30 39 3c 2f 69 64 3e 3c 74 69 74 9108309Test 2 -fooba 02b0 72 3c 2f 74 69 74 6c 65 3e 3c 6d 6f 64 69 66 69 r13287908820 02f0 3c 66 6f 6c 64 65 72 3e 30 3c 2f 66 6f 6c 64 65 0000< 0330 2f 70 72 69 6f 72 69 74 79 3e 3c 72 65 70 65 61 /priority>00< 0370 2f 64 75 65 64 61 74 65 3e 3c 73 74 61 72 74 64 /duedate>00 03b0 30 3c 2f 73 74 61 72 74 74 69 6d 65 3e 3c 6c 65 00< 03d0 73 74 61 74 75 73 3e 32 3c 2f 73 74 61 74 75 73 status>29 0400 31 34 39 33 33 33 3c 2f 69 64 3e 3c 74 69 74 6c 149333org-toodledo 0430 31 33 32 38 30 31 31 39 34 39 3c 2f 6d 6f 64 69 1328011949 0450 30 3c 2f 63 6f 6d 70 6c 65 74 65 64 3e 3c 66 6f 00< 0470 63 6f 6e 74 65 78 74 3e 30 3c 2f 63 6f 6e 74 65 context>002< 04b0 2f 72 65 70 65 61 74 3e 3c 72 65 70 65 61 74 66 /repeat>0000002 ---------------------------------------- And the URL-DEBUG: ---------------------------------------- 34:49.154000 http -> Finding free connection: api.toodledo.com:80 34:49.154000 http -> Contacting host: api.toodledo.com:80 34:49.200000 http -> Marking connection as busy: api.toodledo.com:80 #> 34:49.201000 http -> Request is: POST /2/tasks/get.php HTTP/1.1 MIME-Version: 1.0 Connection: keep-alive Extension: Security/Digest Security/SSL Host: api.toodledo.com Accept-charset: nil Accept: */* User-Agent: URL/Emacs (i386-mingw-nt6.1.7601; Windows-NT; 32bit) Content-Type: application/x-www-form-urlencoded Content-length: 212 f=xml&key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&unix=1&fields=status%2crepeat%2crepeatfrom%2ccontext%2cduedate%2cduetime%2cstartdate%2cstarttime%2cfolder%2cgoal%2cpriority%2cnote%2clength%2cparent&comp=%2d1&modafter=0 34:49.201000 retrieval -> Spinning in url-retrieve-synchronously: nil (#>) 34:49.264000 http -> Calling after change function `url-http-wait-for-headers-change-function' for `#>' 34:49.264000 http -> data (point-max 1) (len 1380): "HTTP/1.1 200 OK Date: Thu, 09 Feb 2012 12:34:49 GMT Server: Apache/2.2.3 (Red Hat) Content-Length: 1189 Keep-Alive: timeout=2, max=100 Connection: Keep-Alive Content-Type: application/xml 9105869Test 113284829381327924800000001328097600013281372000029108309Test 2 -foobar13287908820000000000029149333org-toodledo1328011949000020000002 url-http-wait-for-headers-change-function ( *http api.toodledo.com:80*<92>) 34:49.264000 http -> Saw end of headers... ( *http api.toodledo.com:80*<92>) @ 197 34:49.264000 http -> url-http-clean-headers, end-of-headers now at 189 34:49.264000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<92>) 34:49.264000 http -> Got a content-length 1189, being smart about document end. 34:49.264000 http -> Calling initial content-length for extra data at end of headers 34:49.264000 http -> url-http-content-length-after-change-function 189 1381 1192 34:49.265000 http -> End of document found! 34:49.265000 http -> Marking connection as free: api.toodledo.com:80 #> 34:49.265000 http -> url-http-parse-headers called in ( *http api.toodledo.com:80*<92>) 34:49.265000 http -> url-http-parse-response called in ( *http api.toodledo.com:80*<92>) 34:49.266000 http -> Parsed HTTP headers: class=2 status=200 34:49.266000 http -> Finished parsing HTTP headers: t 34:49.266000 http -> Marking connection as free: api.toodledo.com:80 #> 34:49.266000 http -> Activating callback in buffer ( *http api.toodledo.com:80*<92>) ---------------------------------------- ...cj From debbugs-submit-bounces@debbugs.gnu.org Thu Feb 09 08:08:18 2012 Received: (at 10768) by debbugs.gnu.org; 9 Feb 2012 13:08:18 +0000 Received: from localhost ([127.0.0.1]:33431 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvTjW-0006oD-B4 for submit@debbugs.gnu.org; Thu, 09 Feb 2012 08:08:18 -0500 Received: from mail-out.m-online.net ([212.18.0.10]:42680) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvTjU-0006o6-KH for 10768@debbugs.gnu.org; Thu, 09 Feb 2012 08:08:18 -0500 Received: from frontend1.mail.m-online.net (frontend1.mail.intern.m-online.net [192.168.8.180]) by mail-out.m-online.net (Postfix) with ESMTP id E55D518001C8; Thu, 9 Feb 2012 14:07:06 +0100 (CET) X-Auth-Info: uMOBsXBlpcWmq90IJX5J2OfbnM09glPTqjphK5ux1Ec= Received: from igel.home (ppp-88-217-114-208.dynamic.mnet-online.de [88.217.114.208]) by mail.mnet-online.de (Postfix) with ESMTPA id 3FCF01C000A2; Thu, 9 Feb 2012 14:07:07 +0100 (CET) Received: by igel.home (Postfix, from userid 501) id C61AFCA299; Thu, 9 Feb 2012 14:07:06 +0100 (CET) From: Andreas Schwab To: "Christopher J. White" Subject: Re: bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used References: <20120208235321.00003609@unknown> <20120209074933.000047ef@unknown> X-Yow: An Italian is COMBING his hair in suburban DES MOINES! Date: Thu, 09 Feb 2012 14:07:06 +0100 In-Reply-To: <20120209074933.000047ef@unknown> (Christopher J. White's message of "Thu, 9 Feb 2012 07:49:33 -0500") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.93 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 10768 Cc: 10768@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) "Christopher J. White" writes: > I considered that and looked hard for it to be the server. In the data > output the double quote chars add an additional slash "\" that is not > present in the actual data. You are right, I missed that. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." From debbugs-submit-bounces@debbugs.gnu.org Thu Feb 09 10:44:56 2012 Received: (at 10768) by debbugs.gnu.org; 9 Feb 2012 15:44:56 +0000 Received: from localhost ([127.0.0.1]:33979 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvWB2-000263-Rg for submit@debbugs.gnu.org; Thu, 09 Feb 2012 10:44:56 -0500 Received: from mail-out.m-online.net ([212.18.0.9]:51821) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvWAu-00025h-4z for 10768@debbugs.gnu.org; Thu, 09 Feb 2012 10:44:48 -0500 Received: from frontend1.mail.m-online.net (unknown [192.168.8.180]) by mail-out.m-online.net (Postfix) with ESMTP id 891A81C1DA12; Thu, 9 Feb 2012 16:43:28 +0100 (CET) X-Auth-Info: wWUr0HW35poWZJZkJZRQbDeGAwUtUdBkz+Vsws8LJ+E= Received: from igel.home (ppp-88-217-114-208.dynamic.mnet-online.de [88.217.114.208]) by mail.mnet-online.de (Postfix) with ESMTPA id 455681C001F7; Thu, 9 Feb 2012 16:43:28 +0100 (CET) Received: by igel.home (Postfix, from userid 501) id 98A23CA299; Thu, 9 Feb 2012 16:43:27 +0100 (CET) From: Andreas Schwab To: "Christopher J. White" Subject: Re: bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used References: <20120208235321.00003609@unknown> X-Yow: Yow! Are we in the perfect mood? Date: Thu, 09 Feb 2012 16:43:27 +0100 In-Reply-To: <20120208235321.00003609@unknown> (Christopher J. White's message of "Wed, 8 Feb 2012 23:53:21 -0500") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.93 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 10768 Cc: 10768@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) Does this help? diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el index b43ed76..140824f 100644 --- a/lisp/url/url-http.el +++ b/lisp/url/url-http.el @@ -352,11 +352,14 @@ request.") ;; Parsing routines (defun url-http-clean-headers () "Remove trailing \r from header lines. -This allows us to use `mail-fetch-field', etc." +This allows us to use `mail-fetch-field', etc. +Return the number of characters removed." (declare (special url-http-end-of-headers)) - (goto-char (point-min)) - (while (re-search-forward "\r$" url-http-end-of-headers t) - (replace-match ""))) + (let ((end (marker-position url-http-end-of-headers))) + (goto-char (point-min)) + (while (re-search-forward "\r$" url-http-end-of-headers t) + (replace-match "")) + (- end url-http-end-of-headers))) (defun url-http-handle-authentication (proxy) (declare (special status success url-http-method url-http-data @@ -1051,7 +1054,7 @@ the end of the document." (setq url-http-end-of-headers (set-marker (make-marker) (point)) end-of-headers t) - (url-http-clean-headers))) + (setq nd (- nd (url-http-clean-headers)))) (if (not end-of-headers) ;; Haven't seen the end of the headers yet, need to wait Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." From debbugs-submit-bounces@debbugs.gnu.org Thu Feb 09 13:47:57 2012 Received: (at 10768) by debbugs.gnu.org; 9 Feb 2012 18:47:57 +0000 Received: from localhost ([127.0.0.1]:34023 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvZ2C-0006GW-Nw for submit@debbugs.gnu.org; Thu, 09 Feb 2012 13:47:57 -0500 Received: from mail15c25.carrierzone.com ([64.29.147.25]:35578) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvZ28-0006GM-KJ for 10768@debbugs.gnu.org; Thu, 09 Feb 2012 13:47:55 -0500 X-POP-User: ajgrier.grierwhite.com Received: from unknown (pool-108-49-4-207.bstnma.east.verizon.net [108.49.4.207]) by mail15c25.carrierzone.com (8.13.6/8.13.1) with ESMTP id q19IkcYJ031813; Thu, 9 Feb 2012 18:46:42 GMT Date: Thu, 9 Feb 2012 13:46:37 -0500 From: "Christopher J. White" To: Andreas Schwab Subject: Re: bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Message-ID: <20120209134637.00005ee1@unknown> In-Reply-To: References: <20120208235321.00003609@unknown> X-Mailer: Claws Mail 3.8.0cvs6 (GTK+ 2.16.6; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-CSC: 0 X-CHA: v=1.1 cv=Iru08Ra2f8yZNR1/ykVqFZRxNF25rzdVns/bQLcDnmA= c=1 sm=1 a=hdTLrLAGt2gA:10 a=3pyGGIT-qAEA:10 a=kj9zAlcOel0A:10 a=iW5S9OHVd49uhICS+jddKg==:17 a=tBb2bbeoAAAA:8 a=cZ9j34NmEipaR1qePO0A:9 a=APF60CQRSNhWCpaWhUoA:7 a=CjuIK1q_8ugA:10 a=YuKU6ANggZ8A:10 a=u-AaD2zGYqUVSgAb:21 a=VQqgcTnmhXS0qg0t:21 a=iW5S9OHVd49uhICS+jddKg==:117 X-CTCH-Spam: Unknown X-CTCH-RefID: str=0001.0A02020A.4F341493.006E,ss=1,re=0.000,fgs=0 X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 10768 Cc: 10768@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) Quick work! First past looks good, except you're missing one additional paren after the last diff line: - (url-http-clean-headers))) + (setq nd (- nd (url-http-clean-headers))))) Let me run this through a number of cases and see how well it holds up for a few days. Thanks ...cj On Thu, 09 Feb 2012 16:43:27 +0100 Andreas Schwab wrote: > Does this help? > > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el > index b43ed76..140824f 100644 > --- a/lisp/url/url-http.el > +++ b/lisp/url/url-http.el > @@ -352,11 +352,14 @@ request.") > ;; Parsing routines > (defun url-http-clean-headers () > "Remove trailing \r from header lines. > -This allows us to use `mail-fetch-field', etc." > +This allows us to use `mail-fetch-field', etc. > +Return the number of characters removed." > (declare (special url-http-end-of-headers)) > - (goto-char (point-min)) > - (while (re-search-forward "\r$" url-http-end-of-headers t) > - (replace-match ""))) > + (let ((end (marker-position url-http-end-of-headers))) > + (goto-char (point-min)) > + (while (re-search-forward "\r$" url-http-end-of-headers t) > + (replace-match "")) > + (- end url-http-end-of-headers))) > > (defun url-http-handle-authentication (proxy) > (declare (special status success url-http-method url-http-data > @@ -1051,7 +1054,7 @@ the end of the document." > (setq url-http-end-of-headers (set-marker (make-marker) > (point)) > end-of-headers t) > - (url-http-clean-headers))) > + (setq nd (- nd (url-http-clean-headers)))) > > (if (not end-of-headers) > ;; Haven't seen the end of the headers yet, need to wait > > Andreas. > From debbugs-submit-bounces@debbugs.gnu.org Thu Feb 09 15:16:03 2012 Received: (at 10768) by debbugs.gnu.org; 9 Feb 2012 20:16:03 +0000 Received: from localhost ([127.0.0.1]:34077 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvaPR-0008KQ-9j for submit@debbugs.gnu.org; Thu, 09 Feb 2012 15:16:02 -0500 Received: from mail237c25.carrierzone.com ([64.29.147.233]:51254) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1RvaPL-0008K9-RB for 10768@debbugs.gnu.org; Thu, 09 Feb 2012 15:15:58 -0500 X-POP-User: ajgrier.grierwhite.com Received: from unknown (pool-108-49-4-207.bstnma.east.verizon.net [108.49.4.207]) by mail237c25.carrierzone.com (8.13.6/8.13.1) with ESMTP id q19KEhFD012084; Thu, 9 Feb 2012 20:14:45 GMT Date: Thu, 9 Feb 2012 15:14:43 -0500 From: "Christopher J. White" To: Andreas Schwab Subject: Re: bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used Message-ID: <20120209151443.000011a6@unknown> In-Reply-To: References: <20120208235321.00003609@unknown> X-Mailer: Claws Mail 3.8.0cvs6 (GTK+ 2.16.6; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-CSC: 0 X-CHA: v=1.1 cv=eOIpK0TUyV1bo+1wo/1fQvFrj8DjMnYhoAngSdkCsuU= c=1 sm=1 a=hdTLrLAGt2gA:10 a=3pyGGIT-qAEA:10 a=kj9zAlcOel0A:10 a=iW5S9OHVd49uhICS+jddKg==:17 a=JJJ-DeXCAAAA:8 a=G0BRwk1oAAAA:8 a=tBb2bbeoAAAA:8 a=zF1JnFZfGnnx8OOGMiAA:9 a=yR2xMyxR96uYDJB_MEoA:7 a=CjuIK1q_8ugA:10 a=uyOpwfsfqbsA:10 a=YuKU6ANggZ8A:10 a=iW5S9OHVd49uhICS+jddKg==:117 X-CTCH-Spam: Unknown X-CTCH-RefID: str=0001.0A020203.4F342936.0003,ss=1,re=0.000,fgs=0 X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 10768 Cc: 10768@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) Hi Andreas, I have created a small set of files with lengths varying from 1200 to 1225, and from 2658 to 2662 and put them up on my personal web server. (e.g. http://www.grierwhite.com/url-http-1220.txt) I chose lengths that just barely stretch from 1 to 2 payload packets, and from 2 to 3 packets. 1200-1207 -- 1 packets 1208-1225 -- 2 packets, 1-18 bytes in the 2nd 2658-2659 -- 2 packets 2660-2662 -- 3 packets, 1-3 bytes in the 3rd Prior to the change, the results are as follows: 1200-1207 -- pass 1208-1217 -- fail 1217-1225 -- pass 2658-2662 -- pass This indicates that the failure occurs when there are up to 10 bytes in the 2nd packet. This correlates with the fact that the header is 10 lines, thus will replace 10 \r characters and shorten the buffer by that amount. Consistent with the results I found with the original server toodledo.com that only has 8 header lines and only has the issue up to 8 bytes in the 2nd packet. It is interesting to note that when there are 3 packets and just a few bytes in the last packet, the failure does *not* occur. I think this may be explained by the fact that url-http-wait-for-headers-change-function may be called a second time, thus all position references are recomputed. Andreas, can you verify this logic? If this is correct, the problem only occurs for responses that just border on 2 packets. After the change, all lengths pass. Here is the lisp code that tests the lengths, feel free to run this as you should be able to reproduce this from any machine that can access the internet. I don't have a trunk version of emacs (24), but looking at the trunk copy of url-http.el, it appears to suffer from the same problem. (defun test-url-http-patch (minlen maxlen) (let ((passed 0) (failed 0)) (do ((len minlen (+ 1 len))) ((> len maxlen) nil) (set-buffer (url-retrieve-synchronously (format "http://www.grierwhite.com/url-http-%d.txt" len))) (goto-char (point-min)) (re-search-forward "^\n") (let ((recvlen (- (point-max) (point)))) (if (= len recvlen) (setq passed (1+ passed)) (setq failed (1+ failed)) (message "FAILED orig-len: %d, received len: %d" len recvlen)))) (message "Passed: %d/%d" passed (+ passed failed)))) ;; Run without the patch (test-url-http-patch 1200 1225) "Passed: 16/26" (test-url-http-patch 2658 2662) "Passed: 5/5" ;; Run with the patch (test-url-http-patch 1200 1225) "Passed: 26/26" (test-url-http-patch 2658 2662) "Passed: 5/5" ...cj On Thu, 09 Feb 2012 16:43:27 +0100 Andreas Schwab wrote: > Does this help? > > diff --git a/lisp/url/url-http.el b/lisp/url/url-http.el > index b43ed76..140824f 100644 > --- a/lisp/url/url-http.el > +++ b/lisp/url/url-http.el > @@ -352,11 +352,14 @@ request.") > ;; Parsing routines > (defun url-http-clean-headers () > "Remove trailing \r from header lines. > -This allows us to use `mail-fetch-field', etc." > +This allows us to use `mail-fetch-field', etc. > +Return the number of characters removed." > (declare (special url-http-end-of-headers)) > - (goto-char (point-min)) > - (while (re-search-forward "\r$" url-http-end-of-headers t) > - (replace-match ""))) > + (let ((end (marker-position url-http-end-of-headers))) > + (goto-char (point-min)) > + (while (re-search-forward "\r$" url-http-end-of-headers t) > + (replace-match "")) > + (- end url-http-end-of-headers))) > > (defun url-http-handle-authentication (proxy) > (declare (special status success url-http-method url-http-data > @@ -1051,7 +1054,7 @@ the end of the document." > (setq url-http-end-of-headers (set-marker (make-marker) > (point)) > end-of-headers t) > - (url-http-clean-headers))) > + (setq nd (- nd (url-http-clean-headers)))) > > (if (not end-of-headers) > ;; Haven't seen the end of the headers yet, need to wait > > Andreas. > From debbugs-submit-bounces@debbugs.gnu.org Fri Feb 10 06:48:50 2012 Received: (at 10768-done) by debbugs.gnu.org; 10 Feb 2012 11:48:50 +0000 Received: from localhost ([127.0.0.1]:34697 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Rvoy9-00078F-JR for submit@debbugs.gnu.org; Fri, 10 Feb 2012 06:48:49 -0500 Received: from mail-out.m-online.net ([212.18.0.9]:40485) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Rvoy1-00077r-Lb for 10768-done@debbugs.gnu.org; Fri, 10 Feb 2012 06:48:46 -0500 Received: from frontend1.mail.m-online.net (unknown [192.168.8.180]) by mail-out.m-online.net (Postfix) with ESMTP id 6B2911C0C0D0; Fri, 10 Feb 2012 12:47:21 +0100 (CET) X-Auth-Info: hffvnyHB/AjG6tQkiHQhHSKwZAxqTXQYXkkntnalQd0= Received: from igel.home (ppp-93-104-130-54.dynamic.mnet-online.de [93.104.130.54]) by mail.mnet-online.de (Postfix) with ESMTPA id 2A2611C00085; Fri, 10 Feb 2012 12:47:21 +0100 (CET) Received: by igel.home (Postfix, from userid 501) id BBB3BCA29F; Fri, 10 Feb 2012 12:47:20 +0100 (CET) From: Andreas Schwab To: "Christopher J. White" Subject: Re: bug#10768: 23.3; url-http misses data when last few bytes are in 2nd packet and content-length is used References: <20120208235321.00003609@unknown> <20120209151443.000011a6@unknown> X-Yow: Used staples are good with SOY SAUCE! Date: Fri, 10 Feb 2012 12:47:20 +0100 In-Reply-To: <20120209151443.000011a6@unknown> (Christopher J. White's message of "Thu, 9 Feb 2012 15:14:43 -0500") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.93 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -1.9 (-) X-Debbugs-Envelope-To: 10768-done Cc: 10768-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) Thanks for testing, installed in trunk. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." From unknown Mon Aug 18 15:41:22 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 09 Mar 2012 12:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator