Web Forms :: Screen-scraping Get Null From A Site?
Jan 18, 2010
I am building a site that need to scrape information from a partner site. Now my scraping code works great with other sites but not this one. It is a regular .html site. My thoughts is that it might be generated some how with php (site is build with php).
If it matters here is my code I use. The htmlDocument is htmlAgilityPack but that has nothing to do with it. Result is null on the site I try.
[Code]....
this is from the w3 validator, might have something with this? The site checked is this
[URL]
I am unable to validate this document because on line 422 it contained one or more bytes that I cannot interpret as utf-8 (in other words, the bytes found are not valid values in the specified Character Encoding). check both the content of the file and the character encoding indication.
The error was: utf8 "xA9" does not map to Unicode""
I've created a web application in asp.net so far. where I've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but I haven't accessed the secure page yet. Every time the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.
I've created a web application in asp.net so far. where i've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but i haven't accessed the secure page yet. Every time the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.
I'm trying to write a small application to collect(Scrape) one piece of data from a web site. I would like to be able to simply run the app and it will open the page, find the one piece of data and display it. So far so good...my problem is that the web site is a secure site, meaning I have to provide a user name and password. I've searched all over the web, found many discussions but have yet to find anything that provides specifics on how to accomplish this. I understand a little bit about tokens etc, but I'm really looking for a detailed description of how to do this. Please feel free to direct me to a different forum if I'm in the wrong place.
Every night I have a program that runs that creates a natural gas report. In this report is something called "expert analysis." The is just text which is a commentary of the current Natural Gas market and because it's a volatile market there is new commentary every day. So also every day, an employee in our office runs a public webpage which displays this text, copies and pastes it into a webpage that is part of our system and it gets saved to a file which gets pulled into our nightly report.
The page that is in our system is as ASP page and it uses FileUp to create the file. We want to break our dependancy on FileUp (it's a SoftArtisans product, an excellent vendor, but we are trying to save some money). Right now we don't have FileUp because our trial expired. So every day I run the public webpage and copy the text and paste it to my expert.txt file all manually. Another problem is that the expert analysis is updated after our office staff has all gone home,
Here is my question - can I write a program in .NET that runs a page, parses the resulting HTML and extracts out the expert analysis all automatically? I won't need FileUp anymore becaus I can use the .NET upload control. And I won't have to log on every night to manually copy and paste the expert analysis. Is that what web/screen scraping is? What happens then the public page's format changes because they do maintenance on their site? Is it just like if I were to run [URL] and try to read the Moderator column of the ASP.NET forum? I'd be okay for a while (would I?) but then if the layout changed I wouldn't find the Moderator anymore?
I need to screen scrape a web page and change its style to match the look and feel of the site where it will be displayed in. Is this possible? I'll be using asp.net to do the screen scraping.
I am new to web programming. I am developing a web program using asp.net(vb) that scrapes data of a certain website. I am using System.Net.HttpWebRequest and System.Net.HttpWebResponse to read the HTTP codes.My problem is I can not retrieve the codes of certain frame/container where the data that I needed is located in the website.I understand that iframe has its own URL or link aside from the main URL of the website
How can I prevent my asp.net 3.5 website from being screen scraped by my competitor? Ideally, I want to ensure that no webbots or screenscrapers can extract data from my website. Is there a way to detect that there is a webbot or screen scraper running ?
I am building a site that need to scrape information from a partner site. Now my scraping code works great with other sites but not this one. It is a regular .html site. My thoughts is that it might be generated some how with php (site is build with php). If it matters here is my code I use. The htmlDocument is htmlAgilityPack but that has nothing to do with it. Result is null on the site I try.
tring result; var objRequest = System.Net.HttpWebRequest.Create(strUrl); var objResponse = objRequest.GetResponse(); sing (var sr = new StreamReader(objResponse.GetResponseStream())) [code]...
I am working on a project that uses data scraping technique to retrieve some url links. I encounter this problem when i pass in the url of a [previous page button] link frm the html code and pass it in to httpWebRequest, the httpWebResponse that i get back is different form the actually content. i have been try to sovle this problem for days and no result, as anyone encounter similar problem and manage to sovle it? below is my sample code: [previous page button] [URL] note: i have change the domain name to a dummy address which is localhost
I am using the below code where I put this tag on a .aspx page:
"<a name="tips"></a>"
Now if I from another page run the code in the hyperlink to navigate to this page2 where I have the above tag. The page will scroll down to this tag automatically where this tag will be at the TOP of the screen.
My question now is if it is instead possible to scroll to the position where you have this tag at the BOTTOM of the screen instead ?
I seem to be having some challenges with the data I am retriveing from a Webpage using the Webclient class. The code works fine, however I observe that the regular expression is not picking up the negative or positive sign in the Daily_Movement data. For example, a daily movement can be -0.31 or +0.31 but the code is not picking the sign in front of the decimal values.Here is my code
[Code]....
I think where the problem lies is the part of the code Regex r1 = new Regex("<span class="quoteData">.*</span>"); It picks up the values between the tag quite well, but not the signs in front of it. [Code]....
I have question regarding printer option page which pop up when i click on print button on crystalreportviewer and when i click okay on that pop up page it displays print preview of the report.
Is there any way to skip that pop up window(Printer Option) and just display (PDF)print preview screen instead?
(I am not exporting report to pdf, just want to display printview screen and when the user click on print button it will print out the page)
I have tried following code but i am getting error:
"Object reference not set to an instance of an object."
I have upgraded web application from VS.2008 to vs 2010. I was using previously ReportViewer 2008 control in a page and replaced with 2010 ReportViewer Control.
I have installed the ReportViewer Redistributable also in my local pc as well as in the server. Also as per the new requirement I have placed the Scriptmanager in the web page where the reportviewer control located.
While running the page, the reportviewer run the report and is not showing the data on the screen. It shows the blank screen. But If I export the data to execl or world it is showing the data. Also the page numbers in the reportviewer showing the total pages available in the navigation bar. I am sure the report is running and bringing the data to the screen but not showing it.
The below is the exact scenario, One form will be there and when user click on "Edit" hyper link other window will open to get more details. In that new window also I have a grid or a table which has other hyperlink which lead to open other window and in that window I will enter few details and save it....
I need to scrape a remote html page looking for images and links. I need to find an image that is "most likely" the product image on the page and links that are "near" that image. I currently do this with a javascript bookmarklet so that I am able to get the rendered x/y coordinates of images and links to help me determine if those are the ones that I want. What I want is the ability to get this information by just using a url and not the bookmarklet. The issues it that by using the url and trying something like httpwebrequest and getting the html on the server, I will not have location values since it wasn't rendered in a browser. I need the location of images and links to help me determine the images and links that I want.So how can I get html from a remote site on the server AND use the rendered location
I'm having difficulties scraping dynamically generated table in ASPX. Trying to scrape the gas prices from a site like this GasPrices. I can extract all the information in the gas price table (address, time submitted etc.), except for the actual gas price.
Is there a way I could scrape the gas prices? i.e. somehow get a text representation of it. I'm not very familiar with ASP/ASPX - but what's being generated now is not showing up in the final HTML. I'm using Python to do the scraping, but that's irrelevant unless there's a specific library...
We have a site that was scraping a site to gather all models available of a product. The 3rd party site recently changed the website so it now uses ajax for users to select the manufacturer and then once they select that it loads a dropdown with products using ajax.
I currently was using httpwebrequest for all requests (see below).
Code: Public Function fnRequest(ByVal sPOSTData As String, Optional ByVal bAutoRedirect As Boolean = False) As String Dim uriSite As Uri Dim sReturn As String Dim srReader As StreamReader Dim sTemp As String
[Code] ....
Now in fiddler the post appears to be done with ajax. I tried to send the post data the normal way but it didn't like that. Any example of how to do this? To get an idea go to [URL] ... and see Mount finder and select Projector.
I'm trying to php/curl scrape data from an .NET site (those with __VIEWSTATE, __EVENTVALIDATION). I monitor headers and post vars using Tamper Data so I'm pretty sure I haven't missed anything. My approach is to micmic the post back when the user click on one of the links and parse the response. But the response I'm getting is a page redirect to "Unable to validate data".
I have created my first MVC application. I am using forms authenticataion. I have created a custom action filter MyAppAuthorizationFilter which inherits from AuthorizeFilter. Inside this filter I'm authenticating the user credentials by checking them against a database table (by overriding OnAuthorize method). If the user is not authenticated, I am redirecting the users to a login page (which is view of LoginController). I am using this filter agaginst my main controller so that all actions in this filter will execute only if the user is authenticated.
I do not have any problem when I use the asp.net development server (shipped with Visual Studio). But when I deploy the app to our IIS7 server, I can see the login screen when I access the default url, but after entering the credentials (both correct and incorrect) and posting back the Login page, it stays on the Login page (clears the pwd, but not the username). Its never getting redirected to the home screen or invalid login screen.
IIS 7 server has .net framework 3.5SP1, but no MVC. So I deployed the System.Web.Mvc dll in the bin folder.
Here's my Code (I've omitted the views and models as they are not needed here):
[MyAppAuthorizationFilter] public class MainController : Controller { // // GET: / [code]......
Is it possible to take a screen capture of a particular area of a web-page from the web-page's own ASP.NET code?
I need to display 6 images that are stacked on top of one another using CSS, and I'd like to offer the user the ability to right-click save-as if possible by stacking the screen capture on top top of the original image stack.
I have setup my business object to have a create_date and edit_date members both datetime datatypes. My company want to display the create_date and edit_date fields in a gridview for each transaction. The problem i have is that after insterting a record it will have a valid create_date but no edit_date and when displayed in the gridview it defaults to datetime.minvalue (My default). How on earth do i show an empty field in my gridview for a null datetime field in the database?
I am using similar architecture to the Imar Spaanjaars example of a tiered solution. With a few small tweeks it has worked well for me for ages. I am passing a List<Database> to my object datasource which connects to my gridview.
I have a problem with a SessionVariable. I will explain what is happening:
1. From the default.aspx I send Session["LoginByAdmin"] = "Dummy"; and then redirect to this page where I have the code below. So the Page_Load sees that this Session["LoginByAdmin"] != null and enter this page. This works fine !
2. In the Page_Load, I later fill a ListBox1 with foldernames.
3. With Button1_Click1, I will now delete the choosen Folder in the ListBox1 wich works fine the first time and the ListBox updates the new existing folders in the ListBox. (I have wrapped this inside an updatepanel)
The problems comes now when I try to delete a choosen folder in the ListBox1 the second time in a row. When I select a Folder in the ListBox1 and now Press Button1, I will be redirected to "Default.aspx". This meens that this code is running: (That meens that Session["LoginByAdmin"] == null and I have not set this to null anywhere. This is my big question how this variable can be = null here?
if (Session["LoginByAdmin"] == null) Response.Redirect("Default.aspx"); break;
