In Screen Scraping, How To Retrieve Source Code Of Iframe Or A Windows Container In Website
Nov 11, 2010
I am new to web programming. I am developing a web program using asp.net(vb) that scrapes data of a certain website. I am using System.Net.HttpWebRequest and System.Net.HttpWebResponse to read the HTTP codes.My problem is I can not retrieve the codes of certain frame/container where the data that I needed is located in the website.I understand that iframe has its own URL or link aside from the main URL of the website
View 1 Replies
Similar Messages:
Apr 8, 2010
Every night I have a program that runs that creates a natural gas report. In this report is something called "expert analysis." The is just text which is a commentary of the current Natural Gas market and because it's a volatile market there is new commentary every day. So also every day, an employee in our office runs a public webpage which displays this text, copies and pastes it into a webpage that is part of our system and it gets saved to a file which gets pulled into our nightly report.
The page that is in our system is as ASP page and it uses FileUp to create the file. We want to break our dependancy on FileUp (it's a SoftArtisans product, an excellent vendor, but we are trying to save some money). Right now we don't have FileUp because our trial expired. So every day I run the public webpage and copy the text and paste it to my expert.txt file all manually. Another problem is that the expert analysis is updated after our office staff has all gone home,
Here is my question - can I write a program in .NET that runs a page, parses the resulting HTML and extracts out the expert analysis all automatically? I won't need FileUp anymore becaus I can use the .NET upload control. And I won't have to log on every night to manually copy and paste the expert analysis. Is that what web/screen scraping is? What happens then the public page's format changes because they do maintenance on their site? Is it just like if I were to run [URL] and try to read the Moderator column of the ASP.NET forum? I'd be okay for a while (would I?) but then if the layout changed I wouldn't find the Moderator anymore?
View 14 Replies
Mar 18, 2010
I've created a web application in asp.net so far. where i've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but i haven't accessed the secure page yet. Every time the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.
View 1 Replies
Jan 7, 2010
I'm trying to write a small application to collect(Scrape) one piece of data from a web site. I would like to be able to simply run the app and it will open the page, find the one piece of data and display it. So far so good...my problem is that the web site is a secure site, meaning I have to provide a user name and password. I've searched all over the web, found many discussions but have yet to find anything that provides specifics on how to accomplish this. I understand a little bit about tokens etc, but I'm really looking for a detailed description of how to do this. Please feel free to direct me to a different forum if I'm in the wrong place.
View 3 Replies
Jan 18, 2010
I am building a site that need to scrape information from a partner site. Now my scraping code works great with other sites but not this one. It is a regular .html site. My thoughts is that it might be generated some how with php (site is build with php).
If it matters here is my code I use. The htmlDocument is htmlAgilityPack but that has nothing to do with it. Result is null on the site I try.
[Code]....
this is from the w3 validator, might have something with this? The site checked is this
[URL]
I am unable to validate this document because on line 422 it contained one or more bytes that I cannot interpret as utf-8 (in other words, the bytes found are not valid values in the specified Character Encoding). check both the content of the file and the character encoding indication.
The error was: utf8 "xA9" does not map to Unicode""
View 4 Replies
Mar 18, 2010
I've created a web application in asp.net so far. where I've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but I haven't accessed the secure page yet. Every time
the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.
View 7 Replies
Aug 31, 2010
I need to screen scrape a web page and change its style to match the look and feel of the site where it will be displayed in. Is this possible? I'll be using asp.net to do the screen scraping.
View 1 Replies
Jan 7, 2011
I need to get the source of aspx page that is stored on my company's server, I don't have access to the server (Or any server - so server language not considered), I'm using HTA localy on my PC.I wrote this code: (JavaScript)var WshShell = new ActiveXObject("WScript.Shell")
WshShell.Run("telnet -fh:/telnetlog.txt xxx.xxx.xxx.com 80"); //The host address
setTimeout("WshShell.SendKeys('GET /subfolder/page.aspx HTTP/1.0~~')",1000);
View 2 Replies
Feb 24, 2010
I am working on a project that uses data scraping technique to retrieve some url links. I encounter this problem when i pass in the url of a [previous page button] link frm the html code and pass it in to httpWebRequest, the httpWebResponse that i get back is different form the actually content. i have been try to sovle this problem for days and no result, as anyone encounter similar problem and manage to sovle it? below is my sample code: [previous page button] [URL] note: i have change the domain name to a dummy address which is localhost
[Code]....
View 1 Replies
Jan 14, 2011
I am looking for a way to display an iframe code part in my website. Also, I am looking for a component to make it easy to copy/paste this iframe code part.
View 3 Replies
Nov 19, 2011
We have a site that was scraping a site to gather all models available of a product. The 3rd party site recently changed the website so it now uses ajax for users to select the manufacturer and then once they select that it loads a dropdown with products using ajax.
I currently was using httpwebrequest for all requests (see below).
Code:
Public Function fnRequest(ByVal sPOSTData As String, Optional ByVal bAutoRedirect As Boolean = False) As String
Dim uriSite As Uri
Dim sReturn As String
Dim srReader As StreamReader
Dim sTemp As String
[Code] ....
Now in fiddler the post appears to be done with ajax. I tried to send the post data the normal way but it didn't like that. Any example of how to do this? To get an idea go to [URL] ... and see Mount finder and select Projector.
View 5 Replies
Jun 3, 2010
Is it possible to retrieve/get the html source code of a certain website and use it in a ASP.NET web application to display them? (without using the copy-paste, of course)
I want my ASP.NET web application to use them as part of a code sample feature I currently working on.
I'm also considering working it on PHP.
View 3 Replies
Jul 6, 2010
I'm trying to php/curl scrape data from an .NET site (those with __VIEWSTATE, __EVENTVALIDATION). I monitor headers and post vars using Tamper Data so I'm pretty sure I haven't missed anything. My approach is to micmic the post back when the user click on one of the links and parse the response. But the response I'm getting is a page redirect to "Unable to validate data".
View 1 Replies
Aug 16, 2010
I have successfully been able to parse HTML from a website:
TestParser parser = new TestParser [URL]
List<Pairing> pairings = parser.ParsePairings();
DBSave.SavePairings(pairings, userID);
----------
This works. Now I have to submit a form via C# code.... DONE. It works. And my C# code saves the "new" HTML to a string variable called "submitParsedHTML". My question is: how to I change out the URL, to go through the string variable instead? I am thinking about doing this: When the user clicks the button to "Post" to the form and the app grabs the HTML. The app logs into my FTP account and saves the HTML as a seperate (temporary) file to be used on the internet [URL] Then the app can then parse the HTML as if it was a normal HTML document and not have to go through formatting the text to an HTML document. What do you all think?
View 1 Replies
Sep 3, 2010
I only want to change case (Mixed to small) of URL in HREF attribute, nowhere else. This is just for convention purpose.eg changing
Its <a href="http://www.abc.com/S pecial-E dition.aspx">Special Ediation</a>.
View <a href="http://www.abc.com/O ther-N ews.aspx">Other News</a>.
to
Its <a href="http://www.abc.com/s pecial-e dition.aspx">Special Ediation</a>.
View <a href="http://www.abc.com/o ther-n ews.aspx">Other News</a>.
View 5 Replies
Nov 30, 2010
i am trying to load a dynamic iframe but it cannot visualize the page.
this is my html code:
[Code]....
and this is my codebehind:
[Code]....
i am using a content update panel on the page but the div of the iframe is out of the content panel.
View 1 Replies
Oct 19, 2010
I am developing an application in which I have a page and at the bottom of the page there is an Iframe and inside of it is a Facebook login button and a asp.net login control.
When I scroll to the bottom of the page where the iframe is and then I put my mouse cursor in the username textbox of the loging control and having loged in before; when I type the first letter of my username under the text box apperas a kind of 'autosuggest' box with my user name in it...
That 'autosuggest' type of box (which is standard of a textbox) which should show right under the text box actually shows up 6 inches below it because I scrolled the page. In fact once it is displayed, that 'autosuggest' box stays fix in its position and I can scroll the page up and down with my mouse roller and the text box goes up and down with the page but the 'autosuggest' box stays in its place without moving...
Also the facebook login button when clicked generates a login dialog box inside a browser popup. But when I have it inside he iframe and the page is scrolled to the bottom when the iframe and the facebook login button are then the popup actually apears off screen, way, way down and low on my computer screen so much so that I cant even see it. I know it is there because of the taskbar..
View 1 Replies
Sep 27, 2010
if i click memu control it should call particalur screen in iframe please provide solution for it
View 2 Replies
Sep 10, 2010
I have a website that occasionally needs to have a handful of the tables in its database updated. The updates come from another system that exports to comma delimited text files. I can then either FTP the text files to the web server, send them in through an admin upload page, or manually log in to Remote Desktop to download the text files. I have all my C# code written to parse the files, check the database contents, and decide what to do.
Should I code the sync logic to be part of a file upload page, protected in the admin section of the site or should I create a Windows Service that constantly looks for files to process in a particular directory that I can drop files in through FTP?
I have used Windows Services in the past and they have worked great, but if I ever have to make a change to the code it can take longer than it would if I just had to modify an ASPX.
Are their security benefits one way or another?
Performance benefits?
ASPX page wins the "ease of maintenance" category.
View 1 Replies
Apr 8, 2010
I want to change source of iframe dynamically. i.e by request.query the url will be assigned.
View 1 Replies
Nov 1, 2010
how to make my source code to display on one line instead of multiple in source view. The display drives me batty when I'm trying to find something and I would prefer to display across the page instead of multiple lines down the page.
View 2 Replies
Sep 6, 2010
asp.net open source Lead management system with source code.
View 9 Replies
May 4, 2010
I am using FBA(Form Based Authen) and Windows Authentication for my web application. FBA is working fine with username and pwd. I want to use the same username, pwd for windows authentication without retering credentials in default login screen. How to avoid the default login screen with custom authentication..
View 2 Replies
Jun 20, 2010
how to integrate a asp.net application in windows user Login screen?
I have a asp.net 2.0 application. Users can able to run this application from windows login screen before they login.
View 1 Replies
May 12, 2010
I created image from byte array
System.Drawing.Image newImage;
using (MemoryStream ms = new MemoryStream(imageBytes, 0, imageBytes.Length))
{
ms.Write(imageBytes, 0, imageBytes.Length);
newImage = System.Drawing.Image.FromStream(ms, true);
}
and now I need to have this image as a source for asp:Image (System.Web.UI.WebControls.Image). Is this possible as I know that conversion is impossible?
View 1 Replies
