Board index » cppbuilder » Logging in with TIdHTTP

Logging in with TIdHTTP


2007-07-01 06:44:34 PM
cppbuilder94
I was using the following to fetch html from a site
MyIdHTTP->Get("www.racingpost.co.uk/horses/card_meeting.sd
which worked fine until the site required a login, after which I received a
'302 Found' exception.
I've tried setting the IdHTTP Request properties
Password=MyPassword;
and
UserID=MyUserID;
but this makes no difference.
On setting the HandleRedirection property to true I find that the text
returned is for the sites login / register web page. Opening an IE window
and manually logging in doesn't help. Is there a way of logging in using
TIdHTTP? If not, is there a way of finding where the web site is looking for
the cookies so I can put these in place programatically.
 
 

Re:Logging in with TIdHTTP

For UserID read Username in the above.
 

Re:Logging in with TIdHTTP

You will need use TIdHttp to POST data to the login form. Then TIdHttp will
store the login cookies and then you can load
the desired page just like you have been
-Tom
 

{smallsort}

Re:Logging in with TIdHTTP

"Tom Byars" < XXXX@XXXXX.COM >wrote in message
Quote
On setting the HandleRedirection property to true I find that the
text returned is for the sites login / register web page. Opening an
IE window and manually logging in doesn't help. Is there a way of
logging in using TIdHTTP? If not, is there a way of finding where
the web site is looking for the cookies so I can put these in place
programatically.
The website expects you to POST an input form containing your
username/password. You can do that very easily using a TStringList
with the TIdHTTP::Post() method, ie:
TStringList *Fields = new TStringList;
try
{
Fields->Values["in_un"] = "username";
Fields->Values["in_pw"] = "password";
IdHTTP1.Post("https://reg.racingpost.co.uk/cde/login_iframe.sd",
Fields);
}
__finally
{
delete Fields;
}
If successful, then your original Get() should work. Something else
to keep in mind - the above URL is a secure page, so make sure that
you have an SSL-enabled IOHandler attached to the TIdHTTP beforehand.
Gambit
 

Re:Logging in with TIdHTTP

"Remy Lebeau (TeamB)" < XXXX@XXXXX.COM >wrote in message
Quote

"Tom Byars" < XXXX@XXXXX.COM >wrote in message
news: XXXX@XXXXX.COM ...

>On setting the HandleRedirection property to true I find that the
>text returned is for the sites login / register web page. Opening an
>IE window and manually logging in doesn't help. Is there a way of
>logging in using TIdHTTP? If not, is there a way of finding where
>the web site is looking for the cookies so I can put these in place
>programatically.

The website expects you to POST an input form containing your
username/password. You can do that very easily using a TStringList
with the TIdHTTP::Post() method, ie:

TStringList *Fields = new TStringList;
try
{
Fields->Values["in_un"] = "username";
Fields->Values["in_pw"] = "password";

IdHTTP1.Post("https://reg.racingpost.co.uk/cde/login_iframe.sd",
Fields);
}
__finally
{
delete Fields;
}

If successful, then your original Get() should work. Something else
to keep in mind - the above URL is a secure page, so make sure that
you have an SSL-enabled IOHandler attached to the TIdHTTP beforehand.


Gambit


Tom & Remy, thanks for the replies.
Remy, I'm using a TIdHTTP component called Betting with a
TIdConnectionInterceptOpenSSL component called InterceptSSL. The .dfm file
shows
object Betting: TIdHTTP
Intercept = InterceptSSL
InterceptEnabled = True
HandleRedirects = True
Request.Accept = 'text/html, */*'
Request.ContentLength = 0
Request.ContentRangeEnd = 0
Request.ContentRangeStart = 0
Request.ProxyPort = 0
Request.UserAgent = 'Mozilla/3.0 (compatible; Indy Library)'
end
object InterceptSSL: TIdConnectionInterceptOpenSSL
SSLOptions.Method = sslvSSLv2
SSLOptions.Mode = sslmUnassigned
SSLOptions.VerifyMode = []
SSLOptions.VerifyDepth = 0
end
I then tried the following code (much as you suggested)
AnsiString S;
TStringList *Login=new TStringList;
TStringStream *Response=new TStringStream(S);
Login->Values["in_un"]="username";
Login->Values["in_pw"]="password";
Betting->Post("https://reg.racingpost.co.uk/cde/login_iframe.sd",Login,Response);
S=Response->DataString;
delete Login;
delete Response;
After executing the above S contains the html for the login page again (so
presumably it hasn't accepted the details) and the subsequent Get doesn't
work (program seems to hang when executing the Get line).
Someone using a TCppWebBrowser component tells me that when you manually
login on the home page the parameters for the BeforeNavigate2 procedure have
the following values
PostData
in_un=username&in_pw=password&process=IN&PARGS=http%253A%252F%252Fwww.racingpost.co.uk%252Fnews%252Fhome.sd
Headers
Content-Type: application/x-www-form-urlencoded
TargetFrameName
hm_RP_Login
Can you give me any hints as to what goes where based on that as I'm totally
lost. Apologies if that's all a bit long winded.
 

Re:Logging in with TIdHTTP

"Tom Byars" < XXXX@XXXXX.COM >wrote in message
Quote
Someone using a TCppWebBrowser component tells me that
when you manually login on the home page the parameters for
the BeforeNavigate2 procedure have the following values
Then you will have to add those additional values to your TStringList:
Login->Values["in_un"] = "username";
Login->Values["in_pw"] = "password";
Login->Values["process"] = "IN";
Login->Values["PARGS"] =
"www.racingpost.co.uk/news/home.sd";
Gambit
 

Re:Logging in with TIdHTTP

"Remy Lebeau (TeamB)" < XXXX@XXXXX.COM >wrote in message
Quote

"Tom Byars" < XXXX@XXXXX.COM >wrote in message
news:468a2b26$ XXXX@XXXXX.COM ...

>Someone using a TCppWebBrowser component tells me that
>when you manually login on the home page the parameters for
>the BeforeNavigate2 procedure have the following values

Then you will have to add those additional values to your TStringList:

Login->Values["in_un"] = "username";
Login->Values["in_pw"] = "password";
Login->Values["process"] = "IN";
Login->Values["PARGS"] =
"www.racingpost.co.uk/news/home.sd";


Gambit


Remy, I tried
AnsiString S;
TStringList *Login=new TStringList;
TStringStream *Response=new TStringStream(S);
Login->Values["in_un"]="username";
Login->Values["in_pw"]="password";
Login->Values["process"]="IN";
Login->Values["PARGS"]="http%253A%252F%252Fwww.racingpost.co.uk%252Fnews%252Fhome.sd";
Betting->Post("www.racingpost.co.uk/news/home.sd",Login,Response);
S=Response->DataString;
delete Login;
delete Response;
but it doesn't work. Incidentally, I had to set
Betting->InterceptEnabled=false as the program hung when the Betting->Put
line was executed. I'm thinking that then makes the InterceptSSL component
redundant which in turn makes me think there are properties in the component
I've not set right (although I don't know what the properties mean and indy
help is no help).
 

Re:Logging in with TIdHTTP

"Tom Byars" < XXXX@XXXXX.COM >wrote in message
Quote

Login->Values["PARGS"]="http%253A%252F%252Fwww.racingpost.co.uk%252Fne
ws%252Fhome.sd";
That is wrong. You are sending a string whose characters have already
been URL-encoded twice (which in itself is a problem as they should
not be encoded twice). In other words, characters that the original
encoder thought was not considered safe by the URI specification (in
this case, the encoded characters were safe and did not need encoding)
were converted to their "%HH" equivilents, where "HH" is the 2-digit
hex value for the character (the ":" character is "%3A", and the "/"
character is "%2F"). Then the encoded string was encoded again,
converting the "%" characters themselves into "%25".
Post() is going to take that double-encoded string and encode it a
third time (which it is supposed to). Thus, the final URL that is
transmitted to the server will look like this:
"http%25253A%25252F%25252Fwww.racingpost.co.uk%25252Fnews%25252Fhome.s
d"
When the server receives that, it is going to decode it back to the
value you originally stored in the TStringList:
"http%253A%252F%252Fwww.racingpost.co.uk%252Fnews%252Fhome.sd"
Which is not a valid URL for the server to act on. The URL that you
store in the TStringList should not be initially encoded at all. Did
you try specifying the original unencoded URL like I showed you
earlier?
Login->Values["PARGS"] =
"www.racingpost.co.uk/news/home.sd";
TIdHTTP will not encode that at all as all of the characters are safe.
Then there is nothing for the server to decode, and it will then have
the original URL available to use.
Quote
Incidentally, I had to set Betting->InterceptEnabled=false as the
program
hung when the Betting->Put line was executed.
The website expects you to post to an HTTPS URL. If you disable the
Intercept, then you won't be able to access any HTTPS resources at
all.
Quote
I'm thinking that then makes the InterceptSSL component redundant
which in turn makes me think there are properties in the component
I've
not set right (although I don't know what the properties mean and
indy
help is no help).
TIdHTTP does not directly implement any SSL handling. You must use an
SSL-enabled Intercept for that.
Gambit
 

Re:Logging in with TIdHTTP

Apologies Remy. I misunderstood you. Thanks for the explanation.
I tried the following
AnsiString S;
TStringList *Login=new TStringList;
TStringStream *Response=new TStringStream(S);
Login->Values["in_un"]="username";
Login->Values["in_pw"]="password";
Login->Values["process"]="IN";
Login->Values["PARGS"]="www.racingpost.co.uk/news/home.sd";
Betting->InterceptEnabled=true;
Betting->Post("https://reg.racingpost.co.uk/cde/login_iframe.sd",Login,Response);
S=Response->DataString;
delete Login;
delete Response;
If I then attempt the Betting->Get then, after a long pause, I get an
exception "EIdOSSLConnectError with message 'Error connecting with SSL'".
If I first set
Betting->InterceptEnabled=false;
and then attempt the Betting->Get, the html returned is for the page that
states "This section now requires you to have Registered and Logged In" etc.
I've just used the default values for the InterceptSSL component. I tried
various values for the InterceptSSL Mode property, but nothing seemed to
change. Can you think of anything I've missed?
 

Re:Logging in with TIdHTTP

"Tom Byars" < XXXX@XXXXX.COM >wrote in message
Quote
If I then attempt the Betting->Get then, after a long pause, I
get an exception "EIdOSSLConnectError with message
'Error connecting with SSL'".
Then you likely don't have the Intercept configured properly.
Gambit
 

Re:Logging in with TIdHTTP

"Remy Lebeau (TeamB)" < XXXX@XXXXX.COM >wrote in message
Quote

"Tom Byars" < XXXX@XXXXX.COM >wrote in message
news:468b7929$ XXXX@XXXXX.COM ...

>If I then attempt the Betting->Get then, after a long pause, I
>get an exception "EIdOSSLConnectError with message
>'Error connecting with SSL'".

Then you likely don't have the Intercept configured properly.


Gambit

That's why I earlier included the .dfm contents (below) hoping you could
maybe tell me what I was missing.
object Betting: TIdHTTP
Intercept = InterceptSSL
InterceptEnabled = True
HandleRedirects = True
Request.Accept = 'text/html, */*'
Request.ContentLength = 0
Request.ContentRangeEnd = 0
Request.ContentRangeStart = 0
Request.ProxyPort = 0
Request.UserAgent = 'Mozilla/3.0 (compatible; Indy Library)'
end
object InterceptSSL: TIdConnectionInterceptOpenSSL
SSLOptions.Method = sslvSSLv2
SSLOptions.Mode = sslmUnassigned
SSLOptions.VerifyMode = []
SSLOptions.VerifyDepth = 0
end
 

Re:Logging in with TIdHTTP

"Tom Byars" < XXXX@XXXXX.COM >wrote in message
Quote
That's why I earlier included the .dfm contents (below) hoping
you could maybe tell me what I was missing.
I can't do that. I'm not an SSL expert. I've never used it myself.
Gambit
 

Re:Logging in with TIdHTTP

"Remy Lebeau (TeamB)" < XXXX@XXXXX.COM >wrote in message
Quote

"Tom Byars" < XXXX@XXXXX.COM >wrote in message
news: XXXX@XXXXX.COM ...

>That's why I earlier included the .dfm contents (below) hoping
>you could maybe tell me what I was missing.

I can't do that. I'm not an SSL expert. I've never used it myself.


Gambit

OK Remy. Thanks for your help anyway. Wouldn't imagine there are many
experts given how sh1t the 'help' is on these things.
 

Re:Logging in with TIdHTTP

Sorry, I've been on vacation since I posted last. I'll take a look at it a
bit later. Of course the code
I post will be C and not Delphi.
-Tom