AJAX :: Webscrape Website Using Httpwebrequest
Jul 19, 2010
I've searched everywhere for this but I'm unable to solve it by myself. I'm try to webscrape a website using HttpWebRequest. I'm able to login successfuly and to browse through the different pages to obtain all cookies and viewstates as needed. Fiddler supports my claim as I compared all the information. There is one page which I'm unable to load - This page has a certain AJAX loader which prevents me from getting to the page I need. Apperntly page
(1) makes some sort of calculations and when finished transfers to page
(2) using AJAX. If I try to scrape page (2) directly I get an empty page, without the calculations which I need to scrape.
When I send a request to page (1) it just returns the error posted in the subject: 179|error|500|The page is performing an async postback but the ScriptManager.SupportsPartialRendering property is set to false. Ensure that the property is set to true during an async postback.
View 1 Replies
Similar Messages:
Aug 30, 2010
I have a php website with a login page containing username and password fields with the same exact html names. I would like to have a page in my ASP.NET page with a username and a password textbox field that sends the corresponding credentials to the php website using HttpWebRequest. How would I do that and how could I check the response to see if the authentication has been successful?
View 1 Replies
May 16, 2013
I am using HttpWebRequest to post XML data to SSL URL.
I am using .NET 4.0 and tried many things but nothing worked for me.
1. I have tried adding Cert using in request but it also fails.X509Certificate Cert
try {
ServicePointManager.ServerCertificateValidationCallback += new System.Net.Security.RemoteCertificateValidationCallback(ValidateServerCertificate);
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
X509Certificate Cert = null
[CODE]...
View 1 Replies
Dec 22, 2010
I'm trying to do some simple stuff, I've already looked at the examples through the web and I'm not sure of what I'm doing wrong It's a unit test that i'm doing to test some functionality that later will be performed by some different devices Basically I'm creating a webrequest to my site, which returns a set of cookies, which we later on need. Then I want to create a new webrequest, using the returned cookies from the first response, but when i'm reading that info, the cookies are empty
var request = (HttpWebRequest)WebRequest.Create("http://localhost/bla");
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "GET";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(originalResponse.Cookies); // originalResponse.Cookies has several cookies needed by "bla"
var response = request.GetResponse();
In another place... (inside "bla")
HttpContext.Current.Request.Cookies // this is empty
View 2 Replies
Dec 10, 2010
I am attempting to create a single sign on experience between an asp.net site and a wordpress site using a simple form POST method. I have built a simple php page that uses the native wordpress functions wp_insert_user and wp_signon to create user account in the mysql db and sign them in. In my asp.net 'create new user' page code behind, I'm using the post method of an HttpWebRequest to send the required information to the php page.
It almost works! The new wordpress user is created in the mysql database, but they are not logged in. How can I get wordpress to log them in?Here is my HttpWebRequest
'get the values
Dim fn As String = TxtFirstName.Text
Dim ln As String = TxtLastName.Text[code].....
View 1 Replies
Jun 11, 2010
I'm implementing PayPal PDT (Payment Data Transfer) for payment confirmations in ASP.NET.'m going to receive a url post from PayPal with query string parameters. Then I need to send back the form at the bottom of the page.I'd like to implement the form at the bottom as a non-visual HttpWebRequest.
string urlPayPalEnvironment = "https://www.paypal.com/cgi-bin/webscr";
// Prepare web request.
HttpWebRequest request = WebRequest.Create(urlPayPalEnvironment) as HttpWebRequest;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
[code]...
View 1 Replies
Jul 9, 2010
I perform a httpwebrequest without setting the useragent property.The page where I send my request to response with [Code]....
View 3 Replies
Apr 4, 2011
I am testing an aspx page I wish to use to post to but I cannot seem to send the certificate over with the data. Here is my client code:
System.Text.ASCIIEncoding encoding = new System.Text.ASCIIEncoding();
string postData = "age=23&state=47";
byte[] data = encoding.GetBytes(postData);
X509Certificate x509certificate = X509Certificate.CreateFromCertFile(@"C:Documents and Settings
cimbaloMy DocumentsDownloadsSSL Certificates estCert.cer");
string key = x509certificate.GetPublicKeyString();
string certData = Encoding.ASCII.GetString(x509certificate.Export(X509ContentType.Cert));
string pks12 = Encoding.ASCII.GetString(x509certificate.Export(X509ContentType.Pkcs12));
HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create("http://localhost:3493/Index.aspx");
myRequest.Credentials = new NetworkCredential("xxxx", "yyyy");
myRequest.Method = "POST";
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.ContentLength = data.Length;
myRequest.ClientCertificates.Add(x509certificate);
Stream newStream = myRequest.GetRequestStream();
newStream.Write(data, 0, data.Length);
newStream.Close();
System.IO.StreamReader st = new StreamReader(((HttpWebResponse)myRequest.GetResponse()).GetResponseStream());
Console.Write(st.ReadLine());
and here is the code from my aspx page:
protected void Page_Load(object sender, EventArgs e)
{
if (this.CheckCredentials())
{
string data = this.Context.Request.Form.ToString();
}
}
private bool CheckCredentials()
{
string userName = User.Identity.Name;
HttpClientCertificate certificate = Request.ClientCertificate;
if (certificate.IsPresent)
{
string w = certificate.Get("SUBJECT 0");
return true;
}
return false;
}
There is never a certificate present.
I found this earlier post X509Certificate not getting transmitted to the server
But did not understand the answer. I created a certificate using the MakeCert.exe utility but I do not really understand why it does not work.
View 1 Replies
May 13, 2010
I'm developing a website that will connect to a credit card processing gateway webservice. For security purposes this webservice accepts requests only from IP addresses that were previously informed to them.
Since I'm developing locally, my IP changes almost every day. Is there a way for me to change the IP address of a HttpWebRequest so that I can test the Webservice calls locally?
This webservice is accessed through a https address and the methods must be sent via POST.
View 2 Replies
Sep 14, 2010
I am trying to connect to a web service that uses Kerberos Authentication to authorize the user, but all I get is a 401 unauthorized everytime I try to make the request. Below is the code that I am using.
public XPathNavigator GSASearch(string url, string searchString)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url + searchString);
request.CookieContainer = new CookieContainer();
request.Credentials = CredentialCache.DefaultCredentials;
request.ContentType = "text/xml";
request.Method = "POST";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream receiveStream = response.GetResponseStream();
XPathDocument doc = new XPathDocument(receiveStream);
return doc.CreateNavigator();
}
EDIT: I feel I should explain a bit more what I am attempting to do. I have been tasked with providing a new interface for my company's Google Search Appliance. I am using an ASP.NET page, which does some things like choose a Collection depending on where a user is located, etc. and then sends the appropriate search string the the GSA. This was all working well until they decided to turn authentication on, and now I can't get any results (I either get a 401 unauthorized, or a message stating that 'Data at the root level is invalid'). If I take the search string and provide it directly to the GSA, it authenticates fine, and displays the results, I just can't seem to get it through the HttpWebRequest.
EDIT 2: I did a little more looking (ran the request through Fiddler) and it looks like the request is only attempting Negotiate and not Kerberos. I set the credentials to use Kerberos explicitly as below, but it didn't help...
public XPathNavigator GSASearch(string url, string searchString)
{
CredentialCache credCache = new CredentialCache();
credCache.Add(new Uri(url), "Kerberos", CredentialCache.DefaultNetworkCredentials);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url + searchString);
request.CookieContainer = new CookieContainer();
request.PreAuthenticate = true;
request.Credentials = credCache;
request.ContentType = "text/xml";
request.Method = "POST";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream receiveStream = response.GetResponseStream();
//StreamReader readStream = new StreamReader(receiveStream);
XPathDocument doc = new XPathDocument(receiveStream);
return doc.CreateNavigator();
}
EDIT 3: Ok, looking closer again, the CredentialCache.DefaultCredentials doesn't appear to have my network credentials in it...
View 1 Replies
Nov 17, 2010
I wrote following code to update data to website. Code run; but I am not able to see my data if uploaded. Here we have facility to see what data is getting uploaded but I am not able to see my data.
// Above URL is not real as I do not want to disclose real URL as of Now
Uri targetUrl = new Uri("http://www.x86map.com/post-embed/ewspost");
HttpWebRequest request = null;
StringBuilder sb = new StringBuilder();
[Code]....
View 2 Replies
Aug 30, 2010
I am trying to automate a few things in wordpress blog, using HttpWebrequest.i have tried to get the login page "http://mywebsite.net/wp-admin"and then try to post with login data on page "http://www.mywebsite.net/wp-login.php"data = "log=admin&pwd=mypassword&wp-submit=Log+In&redirect_to=http%3A%2F%2Fmywebsite.net%2Fwp-admin%2F&testcookie=1"i am not able to login, wat i have discovered is that when using browser the cookies are sent to the server, but using HttpWebrequest the cookies are not sent on post, i am configured a cookiecontainer for the httpwebrequest and works fine other wise,.. and also on "post" the request host also changes to "www.mywebsite.net" and on "get" request it is "mywebsite.net"
View 2 Replies
Dec 1, 2010
I am trying to login to the website below using HttpWebRequest. I am passing in the username and password using the Credentials property but keep getting back the Login page of the website. Can anyone explain what I am doing wrong.
[URL]
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(Url);
request.ContentType = "application/x-www-form-urlencoded";
request.Credentials = new NetworkCredential(Username, Password);
request.Method = "POST";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Accept-Language: en-us,en;q=0.5");
request.Headers.Add("Accept-Encoding: gzip,deflate");
request.Headers.Add("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7");
request.KeepAlive = true;
request.Headers.Add("Keep-Alive: 300");
request.Referer = Url;
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705;)";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
string tmp = reader.ReadToEnd();
}
View 3 Replies
Oct 29, 2010
I have a problem posting xml Data using HttpWebRequest..
The server returns Internal error..
I want to simulate a simple Form with Method=Post. The form have three fields. Two for credentials and the third one for Posting XML. Here is the sample HTML which i want to simulate using HttpWebRequest..
<form action="https://myurl.com/samplepage.cfm" method="post" >
integration_id<input type="text" name="integration_id" value="id" /><br />
integration_pwd<input type="text" name="integration_pwd" value="pwd" /><br />
<textarea name="hrxml" style="height: 404px; width: 593px">
<?xml version="1.0" encoding="UTF-8"?>
<BackgroundCheck account="test" userId="test" password="test">
<BackgroundSearchPackage>
<ReferenceId>
<IdValue>12345678</IdValue>
</ReferenceId>
<PersonalData>
<PersonName>
<GivenName>TEST</GivenName>
<FamilyName primary="undefined">TEST</FamilyName>
</PersonName>
<DemographicDetail>
<GovernmentID countryCode="US" issuingAuthority="SSN">00000000000000</GovernmentID>
<DateOfBirth>1901-01-01</DateOfBirth>
</DemographicDetail>
</PersonalData>
</BackgroundSearchPackage>
</BackgroundCheck>
View 1 Replies
Jan 18, 2010
I am calling a RESTful web service in the back-end of some ASP.NET pages.
I am using ASP.NET asynchronous pages, so under the hood I am using the methods:
HttpWebRequest BeginGetResponse() and
HttpWebRequest EndGetResponse()
The response string in my case is always a JSON string. I use the following code to read the entire string:
using (StreamReader sr = new StreamReader(myHttpWebResponse.GetResponseStream()))
{
myObject.JSONData = sr.ReadToEnd();
}
Is this method OK in terms of scalability? I have seen other code samples that instead retrieve the response data in blocks using Read(). My primary goal is scalability, so this back-end call can be made across many concurrent page hits.
View 1 Replies
Jun 12, 2010
My Silverlight4 app is hosted in ASP.NET MVC 2 web application. I do web request through HttpWebRequest class but it gives back a result previously cached. How to disable this caching behavior? There are some links which talks about HttpWebRequest in .NET but Silverlight HttpWebrequest is different. add unique dummy query string on every web request, but I'd prefer more elegant solution. I also tried the following, but it didn't work:
_myHttpWebRequest.BeginGetRequestStream(new AsyncCallback(BeginRequest), new Guid());
In fact, by setting browser history settings it is possible to disable caching. See the following link:
[URL] But asking a user to change browser settings is not an option for me.
View 2 Replies
Jan 26, 2010
I am trying to simulate a POST to a form on an external server that does not require any authentication, and capture a sting containing the resulting page. This is what the form looks like:
<FORM METHOD="POST" ACTION="/controller" NAME="GIN">
<INPUT type="hidden" name="JSPName" value="GIN">
Field1:
<INPUT type="text" name="Field1" size="30"
maxlength="60" class="txtNormal" value="">
</FORM>
This is what my code looks like:
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "Field1=VALUE1&JSPName=GIN";
byte[] data = encoding.GetBytes(postData);
// Prepare web request...
HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create("https://XXX/controller");
myRequest.Method = "POST";
myRequest.ContentType = "text/html";
myRequest.ContentLength = data.Length;
Stream newStream = myRequest.GetRequestStream();
// Send the data.
newStream.Write(data, 0, data.Length);
StreamReader reader = new StreamReader(newStream);
string text = reader.ReadToEnd();
MessageBox.Show(text);
newStream.Close();
Currently, the code returns "Stream was not readable".
View 2 Replies
Feb 8, 2011
I'm trying to debug a specific issue with my ASP.NET application. The client runs the following code:
void uploadFile( string serverUrl, string filePath )
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.
Create( serverUrl );
CredentialCache cache = new CredentialCache();
cache.Add( new Uri( serverUrl ), "Basic", new NetworkCredential( "User", "pass" ) );
request.Credentials = cache;
request.Method = "POST";
request.ContentType = "application/octet-stream";
request.Timeout = 60000;
request.KeepAlive = true;
using( BinaryReader reader = new BinaryReader(
File.OpenRead( filePath ) ) ) {
request.ContentLength = reader.BaseStream.Length;
using( Stream stream = request.GetRequestStream() ) {
byte[] buffer = new byte[1024];
while( true ) {
int bytesRead = reader.Read( buffer, 0, buffer.Length );
if( bytesRead == 0 ) {
break;
}
stream.Write( buffer, 0, bytesRead );
}
}
}
HttpWebResponse result = (HttpWebResponse)request.GetResponse();
//handle result - not relevant
}
and Write() throws an exception with "Unable to write data to the transport connection: An established connection was aborted by the software in your host machine." text. I used System.Net tracing and found that something goes wrong when I send the request with Content-Length set.
Specifically if I omit everything that is inside using statement in the code above the server promptly replies with WWW-Authenticate and then the client reposts the request with WWW-Authenticate and everything goes fine except the file in not uploaded and the request fails much later.
I'd like to do the following: send an request without data, wait for WWW-Authenticate, then repeat it with WWW-Authenticate and data. So I tried to modify the code above: first set all the parameters, then call GetResponse(), then do sending, but when I try to set ContentLength property an exception is thrown with "This property cannot be set after writing has started" text. So HttpWebRequest seems to be non-reusable.
How do I reuse it for resending the request without closing the connection?
View 2 Replies
Mar 31, 2011
I would like to perform HttpWebrequest every 10 min in my website. Right now it is performing onbutton click.
View 1 Replies
Oct 27, 2010
I have a some problem while Posting xml data Using HttpWebRequest. Actually I want to Post 3 Form variables
2 of them are for credential and the third one is for XML data to an api, The api will authenticate and will process the xml data, and will return success if no error found.
Here is what there documentation says.
The data will be passed to the gateway via an HTTPS FORM post and a string value of "success" will be returned upon successful receipt of the data. Three total FORM variables will be posted, two of which will contain credentials, and the third will contain the HR-XML data. A string value of "error" will be returned if the posting fails for any reason.
Form Fields integration_field1 = 1234, integration_field2=2345pwd, hrxml=form field containing the HR-XML order (string)
Here is what I have Coded..
[Code].....
View 1 Replies
Apr 10, 2010
I'm trying to ping Google when my web site's sitemap is updated but I need to know which status code does Google or any other service returns. My code is below:
HttpWebRequest rqst = (HttpWebRequest)WebRequest.Create("http://search.yahooapis.com/ping?sitemap=http%3a%2f%2fhasangursoy.com.tr%2fsitemap.xml");
rqst.Method = "POST";
rqst.ContentType = "text/xml";
rqst.ContentLength = 0;
rqst.Timeout = 3000;
rqst.GetResponse();
View 2 Replies
Feb 23, 2010
i am able to post the data to another page of different application without redirecting by using httpwebrequst.
but now my issue is how to reterive the posted data from the httpwebrequest and strore into database in the pageload.
View 7 Replies
Feb 1, 2010
In my web application (ASP.NET) I have a block of code that uses HttpWebRequest to make a call to a REST service and continue execution. Right now it's taking longer than I would like to complete the full web request. The thing is that what the REST service returns isn't useful. Ideally I would like to send an Async web request to the REST service and then NOT wait for a response. The problem is that I've tried it out using request.BeginGetResponse(New AsyncCallback(AddressOf myFunc), Nothing)
To start an async request and instead of NOT waiting (which I would assume would be the default behavior of an async request) it continuously executes the callback function before executing the next line of code after BeginGetResponse.I'm suspecting that ASP.NET may convert it to a sync request when it's within a web application. I'm led to believe this because there's a IAsyncResult result object that is passed into the callback function and when I examine its CompletedSynchronously property it's always set to true.if it's possible to do an async HttpWebRequest (with no wait) from within an ASP.NET web application or is it always converted to a synchronous request?
View 2 Replies
May 13, 2010
In my ASP.NET app I use HttpWebRequest for fetching external resources which I'd like to be cached. Consider the following code:
var req = WebRequest.Create("http://google.com/");
req.CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.CacheIfAvailable);
var resp = req.GetResponse();
Console.WriteLine(resp.IsFromCache);
var answ = (new StreamReader(resp.GetResponseStream())).ReadToEnd();
Console.WriteLine(answ.Length);
HttpWebRequest uses IE cache, so when I run it as normal user (in tiny cmd test app), data is cached to %userprofile%Local SettingsTemporary Internet Files and next responses are read from cache.
I thought that when such code is run inside ASP.NET app, data will be cached to ...ASPNETLocal SettingsTemporary Internet Files but it is not and cache is never used.
What I am doing wrong? How to force HttpWebRequest to use cache in ASP.NET environment?
View 3 Replies
Dec 21, 2010
I have created a server product that is connecting to some social network servers and sending to data which status update etc. The server already authenticated to necessary social networks servers by users who is using this solution.
Actually, I have no problem at this time, but I think I will.
My server will be open a thousands of concurrent request to neccessary servers via Http with C# HttpWebRequest instance. I already know that It's possible to change concurrent request limits with below propery.
ServicePointManager.DefaultConnectionLimit AFAIK, this limit is max 100 evet you set more than 100. So, I will be faced with bottleneck problem with HttpWebRequest even change the DefaultConnectionLimit property of ServicePointManager.
View 1 Replies
