Accessing Postings via HttpWebRequest/WebClient

After the post - Using CompareHTML to compare postings , few people asked me - how to access postings via WebClient.

Normally when we try to access posting via HttpWebRequest/WebClient, it will return "The remote server returned an error: (500) Internal Server Error."

The reason for the 500 server error is that authoring with MCMS is only supported with IE. And IE will always send the user agent string. So the code in WebAuthor requires this user agent string and does not check for this. As access with a user higher as subscriber is always treated as authoring the problem the above error will occur (thanks Stefan – for giving the correct reason!).

To avoid the above error when accessing a posting with a higher role than subscriber, it is required to specify the “User Agent”.

Via HttpWebRequest
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("url of the posting");
NetworkCredential networkCredential = new
NetworkCredential("username","password","domain");
httpWebRequest.Credentials = networkCredential;
httpWebRequest.UserAgent = "Mozilla/4.0+";
//UserAgent should be specified before getting the response.
HttpWebResponse webResponse = (HttpWebResponse)httpWebRequest.GetResponse();

Via WebClient
WebClient webClient = new WebClient();
NetworkCredential networkCredential = new
NetworkCredential("username","password","domain");
webClient.Credentials = networkCredential;
webClient.Headers.Add("User-Agent","Mozilla/4.0+");
//User-Agent should be specified in header before getting the response.
Stream webStream= webClient.OpenRead("url of the posting");