I'm trying to php/curl scrape data from an .NET site (those with __VIEWSTATE, __EVENTVALIDATION). I monitor headers and post vars using Tamper Data so I'm pretty sure I haven't missed anything. My approach is to micmic the post back when the user click on one of the links and parse the response. But the response I'm getting is a page redirect to "Unable to validate data".
View 1 RepliesI am getting the error "Unable to validate Data" on my ASP.NET 3.5 website. Replication Steps:Open the Default.aspx web page (it just has a few asp.net controls) and leave it idle for approx 1hr.Click on a button (server side standard asp.net button) on the page and we get the error.After going through a couple of blogs, I've got the following suggestionsSome suggested the use of specifying the machine key. But is that really required considering that my website is hosted on a single application server (no web farm scenario). Not sure if I should be trying this.
View 2 RepliesWe have a site that was scraping a site to gather all models available of a product. The 3rd party site recently changed the website so it now uses ajax for users to select the manufacturer and then once they select that it loads a dropdown with products using ajax.
I currently was using httpwebrequest for all requests (see below).
Code:
Public Function fnRequest(ByVal sPOSTData As String, Optional ByVal bAutoRedirect As Boolean = False) As String
Dim uriSite As Uri
Dim sReturn As String
Dim srReader As StreamReader
Dim sTemp As String
[Code] ....
Now in fiddler the post appears to be done with ajax. I tried to send the post data the normal way but it didn't like that. Any example of how to do this? To get an idea go to [URL] ... and see Mount finder and select Projector.
Every night I have a program that runs that creates a natural gas report. In this report is something called "expert analysis." The is just text which is a commentary of the current Natural Gas market and because it's a volatile market there is new commentary every day. So also every day, an employee in our office runs a public webpage which displays this text, copies and pastes it into a webpage that is part of our system and it gets saved to a file which gets pulled into our nightly report.
The page that is in our system is as ASP page and it uses FileUp to create the file. We want to break our dependancy on FileUp (it's a SoftArtisans product, an excellent vendor, but we are trying to save some money). Right now we don't have FileUp because our trial expired. So every day I run the public webpage and copy the text and paste it to my expert.txt file all manually. Another problem is that the expert analysis is updated after our office staff has all gone home,
Here is my question - can I write a program in .NET that runs a page, parses the resulting HTML and extracts out the expert analysis all automatically? I won't need FileUp anymore becaus I can use the .NET upload control. And I won't have to log on every night to manually copy and paste the expert analysis. Is that what web/screen scraping is? What happens then the public page's format changes because they do maintenance on their site? Is it just like if I were to run [URL] and try to read the Moderator column of the ASP.NET forum? I'd be okay for a while (would I?) but then if the layout changed I wouldn't find the Moderator anymore?
I am new to web programming. I am developing a web program using asp.net(vb) that scrapes data of a certain website. I am using System.Net.HttpWebRequest and System.Net.HttpWebResponse to read the HTTP codes.My problem is I can not retrieve the codes of certain frame/container where the data that I needed is located in the website.I understand that iframe has its own URL or link aside from the main URL of the website
View 1 RepliesI'm getting "Unable to validate data" in a webfarm scenario but not all the time. I'm getting it about on AVERAGE 1x or 2x per hour but fairly random. All of the causes of this error that I know of are list below:achinekey in the machine.config (or web.config) not set the same in a web farm scenario. Mine are set the same in the machine.config.A page is submitted after a server recycle and the machinekey is set to auto. I'm using a static key which should
View 1 Replieswhen i click on any control on page following error occur.SOURCE: System.Web MESSAGE: Unable to validate data....
View 1 RepliesI am getting " Unable to validate data" error alert from my web application.hich is developed with asp.net/c#but i didnt reproduced that error from my end. How can i fixed that error?Error Desciption : User Agent: Nokia7230/5.0 (06.90) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 AppleWebKit/420+ (KHTML, like Gecko) Safari/420+User IP: 27.60.200.101
View 4 RepliesI've got the problem with a simple e-mail form, it uses the simple.aspx and ASPEMAILLib.dll from the persits aspemail installation.The application is hosted on IIS 7, the application pool is a .NET 1.1. I followed the guide on http://blogs.iis.net/bills/archive/2008/06/02/installing-asp-net-1-1-with-iis7-on-vista-and-windows-2008.aspx
View 3 RepliesI have several websites which get approximately 3000 pageviews in total per day, and I get this viewstate error roughly 5-10 times per day, caught in global.asax:
System.Web.HttpException: Unable to validate data. at System.Web.Configuration.MachineKeySection.GetDecodedData(Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Int32& dataLength) at System.Web.UI.ObjectStateFormatter.Deserialize(String inputString
I have written the following code for login:
Sessn["IsLogin"] = false;
System.Configuration.Configuration config = System.Web.Configuration.WebConfigurationManager.OpenWebConfiguration("~");
if (txtPassword.Text.Trim() == string.Empty)
[code]...
Server Error in '/'Application. Unable to validate data. Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.Web.HttpException: Unable to validate data.Source Error
View 1 RepliesI'm trying to validate that a GridView logated in one of the steps of the Wizard Control is populated by the user before the Next button is clicked. What I mean is that I need to block Next (or Prev) button until the user populates a gridview, once the gridview has at least one element then enable the (Next/Prev) buttons. How can I enable/disable Next/Prev buttons within a specific step?.
View 1 RepliesOn my website, I only want to allow users that are registerd real estate agents to be able to create an account.Is there a way to have the user registration process something like:1. User puts their "License"on my website Registration page and have them click a "next" button2. There would be a way to have the license validated on the following site: http://www2.dre.ca.gov/publicasp/pplinfo.asp3. If it is a valid license, the user is presented with the next step in the CreateUserWizard form.
View 2 RepliesI am working on a project that uses data scraping technique to retrieve some url links. I encounter this problem when i pass in the url of a [previous page button] link frm the html code and pass it in to httpWebRequest, the httpWebResponse that i get back is different form the actually content. i have been try to sovle this problem for days and no result, as anyone encounter similar problem and manage to sovle it? below is my sample code: [previous page button] [URL] note: i have change the domain name to a dummy address which is localhost
[Code]....
i develop a site that is working fine on my system. but when i publish this the data is not saving into the database. my database in in app_data folder. what is the problem.
View 2 RepliesMy host moved my website to a new server. The DNS configuration is correct but the website is still down and I'm getting this message
"unable to open website at www.aboutmecfs.org. Internal Server error"
I need to allow the user to enter website url either of the following:
http://<<URL>> or https://<<URL>>. Otherwise i need to add the missing portion in the URL and to display it.
How do we do that?
I need to scrape a remote html page looking for images and links. I need to find an image that is "most likely" the product image on the page and links that are "near" that image. I currently do this with a javascript bookmarklet so that I am able to get the rendered x/y coordinates of images and links to help me determine if those are the ones that I want. What I want is the ability to get this information by just using a url and not the bookmarklet. The issues it that by using the url and trying something like httpwebrequest and getting the html on the server, I will not have location values since it wasn't rendered in a browser. I need the location of images and links to help me determine the images and links that I want.So how can I get html from a remote site on the server AND use the rendered location
View 3 RepliesI'm having difficulties scraping dynamically generated table in ASPX. Trying to scrape the gas prices from a site like this GasPrices. I can extract all the information in the gas price table (address, time submitted etc.), except for the actual gas price.
Is there a way I could scrape the gas prices? i.e. somehow get a text representation of it. I'm not very familiar with ASP/ASPX - but what's being generated now is not showing up in the final HTML. I'm using Python to do the scraping, but that's irrelevant unless there's a specific library...
trying to parse an excel file. its structure is very complex. The possible way i know are.
Use Office introp libraries
Use OLEDB provider and read excel file in a dataset.
But the issue is of its complexity like some columns,cells or rows blank etc.
What are the best possible ways to do this ?
I've created a web application in asp.net so far. where i've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but i haven't accessed the secure page yet. Every time the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.
View 1 RepliesI'm trying to write a small application to collect(Scrape) one piece of data from a web site. I would like to be able to simply run the app and it will open the page, find the one piece of data and display it. So far so good...my problem is that the web site is a secure site, meaning I have to provide a user name and password. I've searched all over the web, found many discussions but have yet to find anything that provides specifics on how to accomplish this. I understand a little bit about tokens etc, but I'm really looking for a detailed description of how to do this. Please feel free to direct me to a different forum if I'm in the wrong place.
View 3 RepliesI am building a site that need to scrape information from a partner site. Now my scraping code works great with other sites but not this one. It is a regular .html site. My thoughts is that it might be generated some how with php (site is build with php).
If it matters here is my code I use. The htmlDocument is htmlAgilityPack but that has nothing to do with it. Result is null on the site I try.
[Code]....
this is from the w3 validator, might have something with this? The site checked is this
[URL]
I am unable to validate this document because on line 422 it contained one or more bytes that I cannot interpret as utf-8 (in other words, the bytes found are not valid values in the specified Character Encoding). check both the content of the file and the character encoding indication.
The error was: utf8 "xA9" does not map to Unicode""
I've created a web application in asp.net so far. where I've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but I haven't accessed the secure page yet. Every time
the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.