httpWebRequest through Proxy

6 New Member

Hi there.

I have written a screen scraping application (both web based and windows forms) in vb.net. When testing on a public broadband link it works fine. However it fails at work due to our proxy server.

To authenticate in the asp version, I just added:

Expand|Select|Wrap|Line Numbers

<defaultProxy useDefaultCredentials="true">

to my web.config - this works fine.

However, I need to integrate this with an existing windows form app (vb6) so need to get the windows form version working.

I am struggling to achieve the same settings (as per web.config) directly within my code. This is what I am trying to use so far:

Expand|Select|Wrap|Line Numbers

 Dim mywebRequest As HttpWebRequest = TryCast(WebRequest.Create("http://www.thedomain.com/login.aspx"), HttpWebRequest)
 
mywebRequest.Proxy = New WebProxy("http://myproxyserver")

mywebRequest.Proxy.Credentials = CredentialCache.DefaultCredentials

mywebRequest.Credentials = CredentialCache.DefaultCredentials

As the page I am trying to access requires authentication, I first capture the viewstate of the page. Once I have this I can then simulate a post of my login credentials and use a streamreader to retrieve the contents of the page.

Click here for a C# tutorial.

The thing I dont understand is that the first part of the code appears to work (and passes through the proxy). It is only when i try and close the contents of the response:

Expand|Select|Wrap|Line Numbers

mywebRequest.GetResponse().Close()

that I get the following error:

The remote server returned an error: (407) Proxy Authentication Required.

Could someone advise me as to whether I am defining the proxy correctly?

Thanks.

Ben

Jun 20 '07 #1

Subscribe Reply

9634

retroviz

New Member

Just to let everyone know I got this working in the end. Made a schoolboy error as I had reinitiated the web request but did not add the proxy information for that instance.

If anyone would like a copy of the complete code (to authenticate then screen scrape) then sent me a message

Jun 20 '07 #2

Frinavale

9,735

Recognized Expert Moderator Expert

Just to let everyone know I got this working in the end. Made a schoolboy error as I had reinitiated the web request but did not add the proxy information for that instance.

If anyone would like a copy of the complete code (to authenticate then screen scrape) then sent me a message

I'm glad you got it working!
Could you please share the solution so others can learn as well.

Thanks a lot!

-Frinny

Jun 20 '07 #3

retroviz

New Member

Well I am glad to help. For some tutorials on httpwebrequest and persisting viewstate for sites that require authentication I would seriously recommend:

http://odetocode.com/Articles/162.aspx (c# version of my code)

and

http://aspnet.4guysfromrolla.com/art...spx#postadlink (great article covering the basics of the httpwebrequest class by Scott Mitchell.

I do have a C# version of the following code if you require (as originally created in this then needed to write in vb.net). This also works as a windows form application or a asp.net app.

Create a new class called prices.vb. Insert the following code (comments explain it pretty well):

Expand|Select|Wrap|Line Numbers

 Imports System

Imports System.Net

Imports System.IO

Imports System.Text

Imports System.Web

Imports System.Text.RegularExpressions

Imports Microsoft.VisualBasic

Namespace PageGetter

    Public Class prices

        Public targetURL As String

        Private Function ExtractViewState(ByVal s As String) As String

            Dim viewStateNameDelimiter As String = "__VIEWSTATE"

            Dim valueDelimiter As String = "value="""
 
            Dim viewStateNamePosition As Integer = s.IndexOf(viewStateNameDelimiter)

            Dim viewStateValuePosition As Integer = s.IndexOf(valueDelimiter, viewStateNamePosition)
 
            Dim viewStateStartPosition As Integer = viewStateValuePosition + valueDelimiter.Length

            Dim viewStateEndPosition As Integer = s.IndexOf("""", viewStateStartPosition)
 
            Return HttpUtility.UrlEncodeUnicode(s.Substring(viewStateStartPosition, viewStateEndPosition - viewStateStartPosition))

        End Function
 
        Public Function GetPrices(ByVal targetUrl As String) As String

*********************************************************************************

'only need this if you are behind a proxy            

Dim pxy As New WebProxy("http://yourproxyaddress:0000")

            pxy.Credentials = CredentialCache.DefaultCredentials

*********************************************************************************

            ' first, request the login form to get the viewstate value

            Dim mywebRequest As HttpWebRequest = TryCast(WebRequest.Create("http://somedomain.com/TheLogOnCheck.aspx"), HttpWebRequest)
 
            mywebRequest.Proxy = pxy
 
            'Set the timeout to 1 second (or 1,000 milliseconds)

            mywebRequest.Timeout = 1000
 
            Try
 
                Dim responseReader As New StreamReader(mywebRequest.GetResponse().GetResponseStream())

                Dim responseData As String = responseReader.ReadToEnd()
 
                responseReader.Close()
 
                ' extract the viewstate value and build out POST data

                Dim viewState As String = ExtractViewState(responseData)

                Dim postData As String = [String].Format("__VIEWSTATE={0}&TUserName={1}&TPassword={2}&_ctl0%3AContent%3AbtnLogon=Logon&__PREVIOUSPAGE=cys_as-zp6tmeXXlc07FggKJUKD96k3RyL8XYHQ-U3I1&__EVENTVALIDATION=%2FwEWAwLV3qjMAQKL2pbeCALf%2B9ffB29vWwhgfdAvHzzk%2F%2BqB%2BKkddRGi", viewState, "yourname@domain.co.uk", "password")
 
                ' have a cookie container ready to receive the forms auth cookie

                Dim cookies As New CookieContainer()
 
                ' now post to the login form

                mywebRequest = TryCast(WebRequest.Create("http://somedomain.com/TheLogOnCheck.aspx"), HttpWebRequest)
 
                mywebRequest.Proxy = pxy 'only if behind proxy (see above)
 
                mywebRequest.Method = "POST"

                mywebRequest.ContentType = "application/x-www-form-urlencoded"

                mywebRequest.CookieContainer = cookies
 
                ' write the form values into the request message

                Dim requestWriter As New StreamWriter(mywebRequest.GetRequestStream())

                requestWriter.Write(postData)

                requestWriter.Close()
 
                ' we don't need the contents of the response, just the cookie it issues

                mywebRequest.GetResponse().Close()
 
                ' now we can send out cookie along with a request for the protected page

                mywebRequest = TryCast(WebRequest.Create(targetUrl), HttpWebRequest)

                mywebRequest.CookieContainer = cookies

                responseReader = New StreamReader(mywebRequest.GetResponse().GetResponseStream())
 
                ' and read the response

                responseData = responseReader.ReadToEnd()

                responseReader.Close()
 
                'Here we set up our Regular expression to snatch what's between the tags we want in our html source

                Dim regex As New Regex("<!-- main table -->((.|" & Chr(10) & ")*?)<!-- / main table -->", RegexOptions.IgnoreCase)
 
                'Here we apply our regular expression to our string using the 

                'Match object. 

                Dim oM As Match = regex.Match(responseData)
 
                Return oM.Value
 
            Catch wex As WebException

                'Something went wrong in the HTTP request!  See if it was a timeout problem

                If wex.Status = WebExceptionStatus.Timeout Then

                    Return ("<font color=red>The httpWebRequest has timed out. Please contact the helpdesk</font>")

                Else

                    Return ("<font color=red>FAILED TO CONNECT<br />Status: " & wex.Status & " Message: " & wex.Message & "</font>")

                End If

            End Try
 
        End Function

    End Class

End Namespace

In my windows form I then used the following code to extract (scrape) the page contents (puts the retrieved html into a webbrowser control):

Expand|Select|Wrap|Line Numbers

     Private Sub myForm_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

        Dim myPrice As New prices

        Dim temp As String = myPrice.GetPrices("http://www.somedomain.org/thepageyouwant.aspx")

        WebBrowser1.DocumentText = temp

    End Sub

If you have an asp.net application you can do exactly the same but on your page_load event:

Expand|Select|Wrap|Line Numbers

 Dim futuresprice As New prices

        Dim temp As String = futuresprice.GetPrices("http://www.somedomain.org/thepageyouwant.aspx")

Reponse.Write(temp)

Just some points worth noting. The viewstate value that you build your post data with will not necessarily be the same as mine. What you need to do is download a tool called Fiddler (http://www.fiddler2.com/fiddler2). Open it up and log into your target site. Have a look in the session inspector of the page that is used to authenticate the log in (mine was called logoncheck.aspx ) and you can see the complete viewstate value. You can then ammend the code to suit the site your requirements. Also I cannot guarantee that his will work for everyone's proxy servers.
Finally credit must be given to Scott Mitchell and Scott Allen as it is their tutorials I used to produce this application.

Hope this was of help.
Ben Foster
<email removed>

Jun 21 '07 #4

Frinavale

9,735

Recognized Expert Moderator Expert

Wow!
Thank you for providing your solution.

-Frinny

Jun 21 '07 #5

Similar topics

12615

Downloading WebSites using HttpWebRequest

by: thomas peter | last post by:

I am building a precache engine... one that request over 100 pages on an remote server to cache them remotely... can i use the HttpWebRequest and WebResponse classes for this? or must i use the MSHTML objects to really load the HTML and request all of the images on site? string lcUrl = http://www.cnn.com; // *** Establish the request

C# / C Sharp

14085

errors running HttpWebRequest.GetResponse - 401 unauthorized

by: Steve Richter | last post by:

I have a page that uses simple HTTP GET to do an ISBN lookup via Amazon.com. The page works when I run it from //localhost. But I have moved it to my godaddy.com shared hoster site, and I get errors on the HttpWebRequest.GetResponse statement. The remote server returned an error: (401) Unauthorized also, when I use the network...

ASP.NET

12278

HttpWebRequest and proxy issues

by: Imran Aziz | last post by:

Hello All, I am using HttpWebRequest to fetch webpages in my ASP.net C# application. The request works fine without the proxy, but on using the code from within a network that uses proxy the request does not work. I tried to use the MS code to get around it, but having problems using it. The first thing is that the this conversion ...

ASP.NET

2349

HttpWebRequest POST result is not the same as POST via ServerXMLHT

by: Dave Brown | last post by:

I am attempting to post to a url (https://FakeURL/logon.asp) using the HttpWebRequest class. The response for a succesful post will contain the html for the logon user's default page. We've accomplished this in the past utilizing the ServerXMLHTTP object. When I try an equivalent? post utilizing the HttpWebRequest class, the response...

ASP.NET

3939

HttpWebRequest/.ServerXMLHTTP.4.0 - different results

by: Dave Brown | last post by:

ASP.NET

3320

httpwebrequest problems

by: Tyler | last post by:

I am using httpwebrequest to do a screen scrape. This works great on my development box, but does not on the production box. Here is the code. Dim webRequest As HttpWebRequest = CType(webRequest.Create(LOGIN_URL), HttpWebRequest) webRequest.Proxy = System.Net.GlobalProxySelection.GetEmptyWebProxy webRequest.KeepAlive = False

ASP.NET

5672

HttpWebRequest Changes from .NET 1.1 to 2.0: Proxy Authentication

by: =?Utf-8?B?U2ltb25EZXY=?= | last post by:

Hi I have a utility class, called MailHandler, that I wrote to read and operate on emails on an Exchange server using WebDAV. The WebDAV SQL statements are sent using an HttpWebRequest object. This worked fine in a .NET 1.1 project. I have copied the same Mailhandler class into a .NET 2.0 project. This, too, worked fine when I...

.NET Framework

4000

HTTPWebRequest, Proxy, Credentials?

by: moo | last post by:

Is there a simple way to get my logon credentials to make my web request work through our proxy server? I tried CredentialCache.DefaultCredentials, but I get nothing back. I can get it to work if I just create a new NetworkCredential object specifying user, password and domain. But, I would rather get that info automatically if possible,...

ASP.NET

3472

.Net 2.0 issue with HttpWebRequest/Proxy breaking WebService

by: =?Utf-8?B?TGFycnlLdXBlcm1hbg==?= | last post by:

Our WebDev team seems to have found a problem that exposes a bug in .NET 2.0. This problem can be shown when trying to access a WebService using SSL and through a proxy server after using the HttpWebRequest object. Under normal circumstances I am able to use the webservice without any problems. But after using an HttpWebRequest object to...

ASP.NET

8145

HttpWebRequest over Https Via Proxy Fails using NTLM

by: =?Utf-8?B?TGVuc3Rlcg==?= | last post by:

A C# (.NET 2) application which uses the System.Net.HttpWebRequest object to request a resource over HTTPS is failing following the installation of a new proxy server on our internal network with 407 Proxy Authentication Required. The same request through the old proxy succeeds. The same request to an HTTP address through the new proxy...

ASP.NET

7461

Changing the language in Windows 10

by: Hystou | last post by:

Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language...

Windows Server

7971

Maximizing Business Potential: The Nexus of Website Design and Digital Marketing

by: jinu1996 | last post by:

In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that...

Online Marketing

7491

The easy way to turn off automatic updates for Windows 10/11

by: Hystou | last post by:

Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...

Windows Server

5381

Access Europe - Using VBA to create a class based on a table - Wed 1 May

by: isladogs | last post by:

The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...

Microsoft Access / VBA

5101

Couldn’t get equations in html when convert word .docx file to html file in C#.

by: conductexam | last post by:

I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert...

C# / C Sharp

3509

Trying to create a lan-to-lan vpn between two differents networks

by: TSSRALBI | last post by:

Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...

Networking - Hardware / Configuration

1956

transfer the data from one system to another through ip address

by: 6302768590 | last post by:

Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system

C# / C Sharp

1068

How to add payments to a PHP MySQL app.

by: muto222 | last post by:

How can i add a mobile payment intergratation into php mysql website.

PHP

776

Comprehensive Guide to Website Development in Toronto: Expert Insights from BSMN Consultancy

by: bsmnconsultancy | last post by:

In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

General