Board index » delphi » HTTP Page Request Preview Utility

HTTP Page Request Preview Utility


2005-03-22 04:12:23 AM
delphi283
I have automated IE to login and download pages requested from a web site.
As is was very important that I got the process working again quickly, I had
to go the browser-based route, due to some strict and as yet unknown
authentication the web site performs expecting a browser to be used.
Now that I have IE working (actually TEmbeddedWB and TIEDownload), I would
like to convert my page requests to Indy, but I still cannot seem to
determine all of the criteria the web site uses for authentication. I have
copied all of the headers and cookies from IE to an IdHTTP and Cookie
Manager, and even the UserAgent, but the requests still do not work at all
with TIdHTTP.
I don't know everything that is sent by IE or any addtional header that may
need to be set, so I was hoping there was some utility program I could
install that would intercept Internet Explorer when it requested a page and
would show me absolutely everything that was being sent during the request
in the exact format sent so I could again try to use TIdHTTP for my page
requests.
Thanks for any suggestions.
-Johnnie
 
 

Re:HTTP Page Request Preview Utility

"Johnnie Norsworthy" <XXXX@XXXXX.COM>writes
Quote
Now that I have IE working (actually TEmbeddedWB and TIEDownload),
I would like to convert my page requests to Indy, but I still cannot seem
to
determine all of the criteria the web site uses for authentication.
Use a packet sniffer to view everything that the browser receives and
transmits. Then you can mimic that with Indy as needed.
Quote
I have copied all of the headers and cookies from IE to an IdHTTP and
Cookie Manager, and even the UserAgent, but the requests still do not
work at all with TIdHTTP.
Did you set up any actual authentication settings in TIdHTTP?
Quote
I don't know everything that is sent by IE, or any addtional header
that may need to be set
You will if you use a packet sniffer.
Quote
so I was hoping there was some utility program I could install that would
intercept Internet Explorer when it requested a page and would show me
absolutely everything that was being sent during the request in the exact
format sent so I could again try to use TIdHTTP for my page requests.
Yes - a packet sniffer.
Gambit
 

Re:HTTP Page Request Preview Utility

"Remy Lebeau (TeamB)" <XXXX@XXXXX.COM>writes
Quote

"Johnnie Norsworthy" <XXXX@XXXXX.COM>writes
news:423f2aab$XXXX@XXXXX.COM...

>Now that I have IE working (actually TEmbeddedWB and TIEDownload),
>I would like to convert my page requests to Indy, but I still cannot seem
to
>determine all of the criteria the web site uses for authentication.

Use a packet sniffer to view everything that the browser receives and
transmits. Then you can mimic that with Indy as needed.
Any recommendations for a packet sniffer? I could Google, but I value your
(Remy's) recommendation as you seem to be the main person that helps me in
this group. (Thanks!)
Quote

>I have copied all of the headers and cookies from IE to an IdHTTP and
>Cookie Manager, and even the UserAgent, but the requests still do not
>work at all with TIdHTTP.

Did you set up any actual authentication settings in TIdHTTP?
No, I turned off basic authentication when the site started using ASP
session cookies for authentication.
Thank you, as always,
-Johnnie
 

Re:HTTP Page Request Preview Utility

In article <423f3724$XXXX@XXXXX.COM>, XXXX@XXXXX.COM
says...
Quote
"Remy Lebeau (TeamB)" <XXXX@XXXXX.COM>writes
news:423f2c99$XXXX@XXXXX.COM...
>
>"Johnnie Norsworthy" <XXXX@XXXXX.COM>writes
>news:423f2aab$XXXX@XXXXX.COM...
>
>>Now that I have IE working (actually TEmbeddedWB and TIEDownload),
>>I would like to convert my page requests to Indy, but I still cannot seem
>to
>>determine all of the criteria the web site uses for authentication.
>
>Use a packet sniffer to view everything that the browser receives and
>transmits. Then you can mimic that with Indy as needed.

Any recommendations for a packet sniffer? I could Google, but I value your
(Remy's) recommendation as you seem to be the main person that helps me in
this group. (Thanks!)

Ethereal : www.ethereal.com/
Quote
>
>>I have copied all of the headers and cookies from IE to an IdHTTP and
>>Cookie Manager, and even the UserAgent, but the requests still do not
>>work at all with TIdHTTP.
>
>Did you set up any actual authentication settings in TIdHTTP?

No, I turned off basic authentication when the site started using ASP
session cookies for authentication.

Well...there could be your problem. I have never been able to get ASP
sessions working correctly using Indy.
 

Re:HTTP Page Request Preview Utility

"Johnnie Norsworthy" <XXXX@XXXXX.COM>writes
Quote
Any recommendations for a packet sniffer?
Ethereal (www.ethereal.com)
Gambit
 

Re:HTTP Page Request Preview Utility

"eshipman" <mr_delphi_developer@yahoo!!!.com>writes
Quote
Well...there could be your problem. I have never been able
to get ASP sessions working correctly using Indy.
Why not? Please be more specific.
Gambit
 

Re:HTTP Page Request Preview Utility

"Remy Lebeau (TeamB)" <XXXX@XXXXX.COM>writes
Quote

"Johnnie Norsworthy" <XXXX@XXXXX.COM>writes
news:423f3724$XXXX@XXXXX.COM...

>Any recommendations for a packet sniffer?

Ethereal (www.ethereal.com)
Already downloaded and installed. It works great and I have logged all of
the requests from TIEDownload precisely. I also see a bunch more traffic I
am going to have to check into.
I am going to play around with substituting all of the header information I
found from TIEDownload and see if I can get TIDHTTP to work again. I get the
feeling I will.
Then maybe I can stop asking so many questions here.
Thanks for all the invaluable help.
-Johnnie
 

Re:HTTP Page Request Preview Utility

"Remy Lebeau (TeamB)" <XXXX@XXXXX.COM>writes
Quote

"eshipman" <mr_delphi_developer@yahoo!!!.com>writes
news:XXXX@XXXXX.COM...

>Well...there could be your problem. I have never been able
>to get ASP sessions working correctly using Indy.

Why not? Please be more specific.
I got it working today. Perfectly.
-Johnnie
 

Re:HTTP Page Request Preview Utility

"Johnnie Norsworthy" <XXXX@XXXXX.COM>writes
Quote
I got it working today. Perfectly.
That is fine and all, but you never said what the problem was to begin with,
or how you got it working. Other people might benefit from that
information.
Gambit
 

Re:HTTP Page Request Preview Utility

"Remy Lebeau (TeamB)" <XXXX@XXXXX.COM>writes
Quote

"Johnnie Norsworthy" <XXXX@XXXXX.COM>writes
news:423fca27$XXXX@XXXXX.COM...

>I got it working today. Perfectly.

That is fine and all, but you never said what the problem was to begin
with,
or how you got it working. Other people might benefit from that
information.
The problem was that the web site was partly basing their authentication on
some other browser produced header that was specific to IE but was not in
TIdHTTP. By capturing the actual headers and cookies sent by IE and
duplicating them to TIdHTTP it works.
The Ethereal packet sniffer exposed the exact header values for me to
duplicate them.
Thank you again for your help.
-Johnnie
 

Re:HTTP Page Request Preview Utility

In article <42409442$XXXX@XXXXX.COM>, XXXX@XXXXX.COM
says...
Quote
"Remy Lebeau (TeamB)" <XXXX@XXXXX.COM>writes
news:42408e6f$XXXX@XXXXX.COM...
>
>"Johnnie Norsworthy" <XXXX@XXXXX.COM>writes
>news:423fca27$XXXX@XXXXX.COM...
>
>>I got it working today. Perfectly.
>
>That is fine and all, but you never said what the problem was to begin
>with,
>or how you got it working. Other people might benefit from that
>information.

The problem was that the web site was partly basing their authentication on
some other browser produced header that was specific to IE but was not in
TIdHTTP. By capturing the actual headers and cookies sent by IE and
duplicating them to TIdHTTP it works.

The Ethereal packet sniffer exposed the exact header values for me to
duplicate them.

Mind telling us what they were?
 

Re:HTTP Page Request Preview Utility

"eshipman" <mr_delphi_developer@yahoo!!!.com>writes
Quote
Mind telling us what they were?
Sorry, I didn't think the exact changes would interest anyone. I did not
attempt to find the single header value that made a difference, instead I
just changed them all to mimic IE.
In TIdDHTTP, I set:
Request.Accept=*/*
Request.CacheControl=no-cache
Request.Connection=Keep-Alive
Request.ContentType=application/x-www-form-urlencodedAccept-Encoding: gzip,
deflate
Request.UserAgent=Mozilla/4.0 (compatible; MSIE 6.0; Win32)
Request.Host=(web site name)
-Johnnie
 

Re:HTTP Page Request Preview Utility

In article <424099d9$XXXX@XXXXX.COM>, XXXX@XXXXX.COM
says...
Quote
"eshipman" <mr_delphi_developer@yahoo!!!.com>writes
news:XXXX@XXXXX.COM...

>Mind telling us what they were?

Sorry, I didn't think the exact changes would interest anyone. I did not
attempt to find the single header value that made a difference, instead I
just changed them all to mimic IE.

In TIdDHTTP, I set:
Request.Accept=*/*
Request.CacheControl=no-cache
Request.Connection=Keep-Alive
Request.ContentType=application/x-www-form-urlencodedAccept-Encoding: gzip,
deflate
Request.UserAgent=Mozilla/4.0 (compatible; MSIE 6.0; Win32)
Request.Host=(web site name)
My bet is that it was the Request.Host. Most sites check the referrer as
that is most readily avail but some use the request.host because most
developers don't think to set it and it would be easy to overlook.
 

Re:HTTP Page Request Preview Utility

"eshipman" <mr_delphi_developer@yahoo!!!.com>writes
Quote
My bet is that it was the Request.Host. Most sites check the referrer as
that is most readily avail but some use the request.host because most
developers don't think to set it and it would be easy to overlook.
Not in this case. That was one I tried before setting the other headers. It
didn't work.
-Johnnie