Monday, October 10, 2005

Accessing Postings via HttpWebRequest/WebClient

After the post - Using CompareHTML to compare postings , few people asked me - how to access postings via WebClient.

Normally when we try to access posting via HttpWebRequest/WebClient, it will return "The remote server returned an error: (500) Internal Server Error."

The reason for the 500 server error is that authoring with MCMS is only supported with IE. And IE will always send the user agent string. So the code in WebAuthor requires this user agent string and does not check for this. As access with a user higher as subscriber is always treated as authoring the problem the above error will occur (thanks Stefan – for giving the correct reason!).

To avoid the above error when accessing a posting with a higher role than subscriber, it is required to specify the “User Agent”.

Via HttpWebRequest
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("url of the posting");
NetworkCredential networkCredential = new
NetworkCredential("username","password","domain");
httpWebRequest.Credentials = networkCredential;
httpWebRequest.UserAgent = "Mozilla/4.0+";
//UserAgent should be specified before getting the response.
HttpWebResponse webResponse = (HttpWebResponse)httpWebRequest.GetResponse();

Via WebClient
WebClient webClient = new WebClient();
NetworkCredential networkCredential = new
NetworkCredential("username","password","domain");
webClient.Credentials = networkCredential;
webClient.Headers.Add("User-Agent","Mozilla/4.0+");
//User-Agent should be specified in header before getting the response.
Stream webStream= webClient.OpenRead("url of the posting");

8 Comments:

Anonymous Anonymous said...

MCMS does not make a difference between request using the GUID and the friendly path.
In addition 500 server errors will only occur to users with more than subscriber rights if the user agent is missing - except if your own template code has a bug that requires this.

The reason for the 500 server error is that authoring with MCMS is only supported with IE. And IE will always send the user agent string. So the code in Webauthor requires this user agent string and does not check for this.

As access with a user higher as subscriber is always treated as authoring the problem described below will happen.

Again: it does not matter if you are using the guid based or the friendly URLs.

3:03 PM  
Blogger Chester said...

Thanks for giving the correct reason. Updated!

12:59 AM  
Anonymous Anonymous said...

hi chester,

i had do to the same but with forms-authentication...so i made a request to the "ManualLoginSubmit.asp"-page (i had to pass some parameters like userid, pwd etc)...this will also work...if you're interested i can send you the example...

björn

5:00 PM  
Blogger Chester said...

Hi björn,

Thanks for thinking of sharing the information. You can post it here.

Cheers,
Chester.

1:56 AM  
Anonymous Anonymous said...

Hi Chester,

I am trying to do the same with forms authentication. Still i am not able to get through the login screen. Please see the code below and let me know if i am doing anything wrong.

string sGuid = Item.Cells[1].Text;
Posting posting = CmsHttpContext.Current.Searches.GetByGuid(sGuid) as Posting;

Label1.Text = Label1.Text + posting.Url;

string HTMLResult = "";
Label1.Text = "http://localhost";
Label1.Text = Label1.Text +

posting.Url;
NetworkCredential MyCredentials = new NetworkCredential("Subscriber", "Acc1234$$", "CSR");
CredentialCache Cache = new CredentialCache();

Cache.Add(new Uri(Label1.Text), "Basic", MyCredentials);
HttpWebRequest RequestHTML = (HttpWebRequest)WebRequest.Create(Label1.Text);
RequestHTML.UserAgent = "Mozilla/4.0+";
RequestHTML.Credentials = Cache;
RequestHTML.AllowAutoRedirect = true;
RequestHTML.MaximumAutomaticRedirections = 100;
HttpWebResponse HTMLResponse = (HttpWebResponse)RequestHTML.GetResponse();
StreamReader HTMLStream = new StreamReader(HTMLResponse.GetResponseStream());
HTMLResult = HTMLStream.ReadToEnd();
LabelHTML.Text = HTMLResult;
HTMLStream.Close();

Thanks,
Santhosh Kumar.P

9:24 AM  
Anonymous Anonymous said...

Hi I still having troubles with "Internal Server Error HTTP 500" when trying to GetResponse().

So far, I login into the site, then I go to the upload page, I post the data and then when I try to read the response I get the server error.

Can anyone help me with this.

webRequest.ContentLength = postData.Length;
webRequest.Timeout = 10000;

using (Stream s = webRequest.GetRequestStream())

postData.WriteTo(s);
postData.Close();
webRequest.UserAgent = "Mozilla/4.0+";

HttpWebResponse myResponse = (HttpWebResponse)webRequest.GetResponse();
myResponse.GetResponseStream();
myResponse.Close();

6:42 AM  
Blogger Soar Iceland said...

Thank you for sharing this, I have been searching for a solution for three hours now so, thanks a lot.

For others then my problem was with this code:
m_req = (HttpWebRequest) HttpWebRequest.Create(UpdateUrl);

m_req.Credentials = new NetworkCredential("user", "password", "domain");

m_req.UserAgent = "Mozilla/4.0+";

m_req.Method = "GET";
m_req.BeginGetResponse(new AsyncCallback(ResponseReceived), null);

And the solution was to add this line as in your code:

m_req.UserAgent = "Mozilla/4.0+";

I was working against IIS 6 and for this to work you will need to set authentication method to basic authentication for the webapp you are connecting to.

Thanks a lot, regards, Sigtryggur.

9:44 AM  
Anonymous Anonymous said...

i have test harness
private static bool ResponseHasTarget(string uri,
string postData,
string target)
{
byte[] buffer = Encoding.ASCII.GetBytes(postData);

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = buffer.Length;
req.Timeout = 5000;
//req.Credentials =CredentialCache.DefaultCredentials;
req.UserAgent = "MSIE 7.0;";
Stream reqst = req.GetRequestStream();
reqst.Write(buffer, 0, buffer.Length);

reqst.Flush();
reqst.Close();

HttpWebResponse res = (HttpWebResponse)req.GetResponse();
Stream resst = res.GetResponseStream();
StreamReader sr = new StreamReader(resst);

string result = sr.ReadToEnd();

sr.Close();
resst.Close();
if (result.IndexOf(target) >= 0)
return true;
else
return false;
}
but error:"the remote server returned an error 500 internal server error";
some body help me, if i run it on local what domain name??
please send mail to tvnhan@gmail.com thansk alot.

11:35 AM  

Post a Comment

<< Home