Monday, October 10, 2005

Accessing Postings via HttpWebRequest/WebClient

After the post - Using CompareHTML to compare postings , few people asked me - how to access postings via WebClient.

Normally when we try to access posting via HttpWebRequest/WebClient, it will return "The remote server returned an error: (500) Internal Server Error."

The reason for the 500 server error is that authoring with MCMS is only supported with IE. And IE will always send the user agent string. So the code in WebAuthor requires this user agent string and does not check for this. As access with a user higher as subscriber is always treated as authoring the problem the above error will occur (thanks Stefan – for giving the correct reason!).

To avoid the above error when accessing a posting with a higher role than subscriber, it is required to specify the “User Agent”.

Via HttpWebRequest
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("url of the posting");
NetworkCredential networkCredential = new
httpWebRequest.Credentials = networkCredential;
httpWebRequest.UserAgent = "Mozilla/4.0+";
//UserAgent should be specified before getting the response.
HttpWebResponse webResponse = (HttpWebResponse)httpWebRequest.GetResponse();

Via WebClient
WebClient webClient = new WebClient();
NetworkCredential networkCredential = new
webClient.Credentials = networkCredential;
//User-Agent should be specified in header before getting the response.
Stream webStream= webClient.OpenRead("url of the posting");


Anonymous Stefan Goßner [MSFT] said...

MCMS does not make a difference between request using the GUID and the friendly path.
In addition 500 server errors will only occur to users with more than subscriber rights if the user agent is missing - except if your own template code has a bug that requires this.

The reason for the 500 server error is that authoring with MCMS is only supported with IE. And IE will always send the user agent string. So the code in Webauthor requires this user agent string and does not check for this.

As access with a user higher as subscriber is always treated as authoring the problem described below will happen.

Again: it does not matter if you are using the guid based or the friendly URLs.

3:03 PM  
Blogger Chester said...

Thanks for giving the correct reason. Updated!

12:59 AM  
Anonymous Björn said...

hi chester,

i had do to the same but with i made a request to the "ManualLoginSubmit.asp"-page (i had to pass some parameters like userid, pwd etc)...this will also work...if you're interested i can send you the example...


5:00 PM  
Blogger Chester said...

Hi björn,

Thanks for thinking of sharing the information. You can post it here.


1:56 AM  
Anonymous Anonymous said...

Hi Chester,

I am trying to do the same with forms authentication. Still i am not able to get through the login screen. Please see the code below and let me know if i am doing anything wrong.

string sGuid = Item.Cells[1].Text;
Posting posting = CmsHttpContext.Current.Searches.GetByGuid(sGuid) as Posting;

Label1.Text = Label1.Text + posting.Url;

string HTMLResult = "";
Label1.Text = "http://localhost";
Label1.Text = Label1.Text +

NetworkCredential MyCredentials = new NetworkCredential("Subscriber", "Acc1234$$", "CSR");
CredentialCache Cache = new CredentialCache();

Cache.Add(new Uri(Label1.Text), "Basic", MyCredentials);
HttpWebRequest RequestHTML = (HttpWebRequest)WebRequest.Create(Label1.Text);
RequestHTML.UserAgent = "Mozilla/4.0+";
RequestHTML.Credentials = Cache;
RequestHTML.AllowAutoRedirect = true;
RequestHTML.MaximumAutomaticRedirections = 100;
HttpWebResponse HTMLResponse = (HttpWebResponse)RequestHTML.GetResponse();
StreamReader HTMLStream = new StreamReader(HTMLResponse.GetResponseStream());
HTMLResult = HTMLStream.ReadToEnd();
LabelHTML.Text = HTMLResult;

Santhosh Kumar.P

9:24 AM  
Anonymous Anonymous said...

Hi I still having troubles with "Internal Server Error HTTP 500" when trying to GetResponse().

So far, I login into the site, then I go to the upload page, I post the data and then when I try to read the response I get the server error.

Can anyone help me with this.

webRequest.ContentLength = postData.Length;
webRequest.Timeout = 10000;

using (Stream s = webRequest.GetRequestStream())

webRequest.UserAgent = "Mozilla/4.0+";

HttpWebResponse myResponse = (HttpWebResponse)webRequest.GetResponse();

6:42 AM  
Blogger Zippo said...

Thank you for sharing this, I have been searching for a solution for three hours now so, thanks a lot.

For others then my problem was with this code:
m_req = (HttpWebRequest) HttpWebRequest.Create(UpdateUrl);

m_req.Credentials = new NetworkCredential("user", "password", "domain");

m_req.UserAgent = "Mozilla/4.0+";

m_req.Method = "GET";
m_req.BeginGetResponse(new AsyncCallback(ResponseReceived), null);

And the solution was to add this line as in your code:

m_req.UserAgent = "Mozilla/4.0+";

I was working against IIS 6 and for this to work you will need to set authentication method to basic authentication for the webapp you are connecting to.

Thanks a lot, regards, Sigtryggur.

9:44 AM  
Anonymous Anonymous said...

i have test harness
private static bool ResponseHasTarget(string uri,
string postData,
string target)
byte[] buffer = Encoding.ASCII.GetBytes(postData);

HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = buffer.Length;
req.Timeout = 5000;
//req.Credentials =CredentialCache.DefaultCredentials;
req.UserAgent = "MSIE 7.0;";
Stream reqst = req.GetRequestStream();
reqst.Write(buffer, 0, buffer.Length);


HttpWebResponse res = (HttpWebResponse)req.GetResponse();
Stream resst = res.GetResponseStream();
StreamReader sr = new StreamReader(resst);

string result = sr.ReadToEnd();

if (result.IndexOf(target) >= 0)
return true;
return false;
but error:"the remote server returned an error 500 internal server error";
some body help me, if i run it on local what domain name??
please send mail to thansk alot.

11:35 AM  
Anonymous Anonymous said...

There were three guys,World of warcraft gold an American and Power Leveling They were all going to be executed.knight noah The executioner wedding dresses said that since wedding dress designers all three were to be ms mesos executed that night, flyff money that they would each get world of warcraft power leveling to choose the flyff money method by which wedding dresses they would die Their warhammer power leveling choices were:cabal alz lethal injection,cheap aion kinah electric chair Knight Gold or world of warcraft gold by hanging.he American was afraid of needles and did'nt Dragonica crone want to be hanged. flyff money The American 2moons dil chose the wedding dresses electric chair.cdkey He sat in the chair and they wow gold pulled the flyff gold switch and nothing happened.

10:22 AM  
Blogger  said...



1:59 PM  
Blogger  said...


人妻奥様熟女熟女アダルト エロ動画マダムパンチラアダルトフェラフェラパイズリ手こき手こき痴女熟女熟女熟女熟女美脚美尻巨乳熟女巨乳

2:00 PM  

Post a Comment

<< Home