Scraping .aspx Content Using Python?
Apr 29, 2010
I'm having difficulties scraping dynamically generated table in ASPX. Trying to scrape the gas prices from a site like this GasPrices. I can extract all the information in the gas price table (address, time submitted etc.), except for the actual gas price.
Is there a way I could scrape the gas prices? i.e. somehow get a text representation of it. I'm not very familiar with ASP/ASPX - but what's being generated now is not showing up in the final HTML. I'm using Python to do the scraping, but that's irrelevant unless there's a specific library...
View 1 Replies
Similar Messages:
Sep 6, 2010
I need to scrape a remote html page looking for images and links. I need to find an image that is "most likely" the product image on the page and links that are "near" that image. I currently do this with a javascript bookmarklet so that I am able to get the rendered x/y coordinates of images and links to help me determine if those are the ones that I want. What I want is the ability to get this information by just using a url and not the bookmarklet. The issues it that by using the url and trying something like httpwebrequest and getting the html on the server, I will not have location values since it wasn't rendered in a browser. I need the location of images and links to help me determine the images and links that I want.So how can I get html from a remote site on the server AND use the rendered location
View 3 Replies
Apr 8, 2010
Every night I have a program that runs that creates a natural gas report. In this report is something called "expert analysis." The is just text which is a commentary of the current Natural Gas market and because it's a volatile market there is new commentary every day. So also every day, an employee in our office runs a public webpage which displays this text, copies and pastes it into a webpage that is part of our system and it gets saved to a file which gets pulled into our nightly report.
The page that is in our system is as ASP page and it uses FileUp to create the file. We want to break our dependancy on FileUp (it's a SoftArtisans product, an excellent vendor, but we are trying to save some money). Right now we don't have FileUp because our trial expired. So every day I run the public webpage and copy the text and paste it to my expert.txt file all manually. Another problem is that the expert analysis is updated after our office staff has all gone home,
Here is my question - can I write a program in .NET that runs a page, parses the resulting HTML and extracts out the expert analysis all automatically? I won't need FileUp anymore becaus I can use the .NET upload control. And I won't have to log on every night to manually copy and paste the expert analysis. Is that what web/screen scraping is? What happens then the public page's format changes because they do maintenance on their site? Is it just like if I were to run [URL] and try to read the Moderator column of the ASP.NET forum? I'd be okay for a while (would I?) but then if the layout changed I wouldn't find the Moderator anymore?
View 14 Replies
Mar 22, 2010
I'm using this Ajax script [URL] to load content from an aspx page on another server than the page calling the content. So far I've learned that this is a no go. The problem seems to be that when using an absolute link to content the script fails as apposed to using a relative link.
I've searched the web for about 10 hours now, and I still haven't found what I'm looking for.
View 2 Replies
Jan 21, 2010
Is it possible to run Python & Django on IIS? I am going to be a Lead Developer in some web design company and right now they are using classic ASP and ASP.NET.
As far as I can see ASP.NET MVC is not mature. Should I recommend Python & Django stack? If it's not possible to run Python on IIS what do you think I should do?
Can I run IIS and Apache in parallel?
View 2 Replies
Nov 8, 2010
I have a GetList.py file which consumes Web Service and saves the output in XML on the server.
How do I invoke GetList.py so it saves the output in XML on the server before displaying the XML output in .ASPX page?
View 3 Replies
May 21, 2010
break up the pros and cons of using python django vs asp.net mvc besides the maturity level of its framework. I have intermediate experience with JAVA. As of right now, i'm leaning towards python but i just wanted to make sure i am making the right choice.
I find myself limited with books available on asp.net web developments. I am aware that there is the storefront example on the official asp.net site. However, that tutorial was a little hard for me to follow. I've done a research around and was python could be my next available choice. There are more tutorials available online for python anyways.
View 1 Replies
Jun 9, 2010
I'm building an ASP.NET MVC (C#) site where I want to implement STV (Single Transferable Vote) voting. I've used OpenSTV for voting scenarios before, with great success, but I've never used it programmatically.
The OpenSTV Google Code project offers a Python script that allows usage of OpenSTV from other applications:
import sys sys.path.append("path to openstv package")
from openstv.ballots import Ballots
from openstv.ReportPlugins.TextReport import TextReport
from openstv.plugins import getMethodPlugins
(ballotFname, method, reportFname) = sys.argv[1:]
methods = getMethodPlugins("byName")
f = open(reportFname, "w")
try:
b = Ballots()
b.loadUnknown(ballotFname)
except Exception, msg:
print >> f, ("Unable to read ballots from %s" % ballotFname)
print >> f, msg
sys.exit(-1)
try:
e = methods[method](b)
e.runElection()
except Exception, msg:
print >> f, ("Unable to count votes using %s" % method)
print >> f, msg
sys.exit(-1).......
Is there a way for me to make such a Python call from my C# ASP.NET MVC site?
View 2 Replies
Apr 7, 2010
I would like to use some code written in python (it uses built in modules) in a regular ASP.NET/C# web application. I am a newbie in python and have heard of IronPython and how ASP.NET now allows us to create IronPython apps.
Any thoughts on which way I should proceed? The python code is the on here
View 1 Replies
Jun 18, 2010
When user click Build button Phython script needs to run.I execute this by using process but so many processes created when so many clicked the Build button, server hang.
View 1 Replies
Mar 3, 2011
I'm trying to get a CodeExpression from a string of python. It seems possible since some of the DLR docs mention CodeDom classes but I haven't been able to find any examples or better docs. Any insight or psuedocode appreciated...
FYI, the use case here is to make an Asp.Net ExpressionBuilder that can evaluate python to use in SharePoint 2010. Let me know if there's anything like this out there!
View 1 Replies
Jan 28, 2011
I am using iTextSharp for convert aspx content to pdf . But facing problem for https site where page is secured and can't don lad as pdf. and geting error like "Internet explorer was not able to open this internet site. The requested site is either unavailable or cannot be found . Please try again later"
View 2 Replies
Jun 2, 2010
Does the DLR only work for console applications?
I would like to embed Python code in an ASP.NET Web application.
View 11 Replies
Jun 30, 2010
How do I tell an ASP.NET Web application to look in the current directory for a .pkl file? This file is read in an embedded Python script.So far, I get a FileNotFound exception stating:
[Could not find file 'C:Program Files (x86)Common FilesMicrosoft SharedDevServer10.0 2.pkl'.]
View 1 Replies
Jul 13, 2010
trying to parse an excel file. its structure is very complex. The possible way i know are.
Use Office introp libraries
Use OLEDB provider and read excel file in a dataset.
But the issue is of its complexity like some columns,cells or rows blank etc.
What are the best possible ways to do this ?
View 5 Replies
Nov 19, 2011
We have a site that was scraping a site to gather all models available of a product. The 3rd party site recently changed the website so it now uses ajax for users to select the manufacturer and then once they select that it loads a dropdown with products using ajax.
I currently was using httpwebrequest for all requests (see below).
Code:
Public Function fnRequest(ByVal sPOSTData As String, Optional ByVal bAutoRedirect As Boolean = False) As String
Dim uriSite As Uri
Dim sReturn As String
Dim srReader As StreamReader
Dim sTemp As String
[Code] ....
Now in fiddler the post appears to be done with ajax. I tried to send the post data the normal way but it didn't like that. Any example of how to do this? To get an idea go to [URL] ... and see Mount finder and select Projector.
View 5 Replies
Jun 2, 2010
I want to know to get content(in IE "View->Source" that produce current page content) of current aspx page in asp.net.
View 2 Replies
Apr 11, 2010
I'm trying to parse a page that has different sections that are loaded with a Javascript __doPostBack() function.
An example of a link is: javascript:__doPostBack('ctl00$cphMain$ucOemSchPicker$dlSch$ctl03$btnSch','')
As soon as this is clicked, the browser doesn't fetch a new URL but a section of webpage is updated to reflect new information.
What would I pass into a urllib function to complete the operation?
View 1 Replies
Mar 18, 2010
I've created a web application in asp.net so far. where i've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but i haven't accessed the secure page yet. Every time the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.
View 1 Replies
Jul 6, 2010
I'm trying to php/curl scrape data from an .NET site (those with __VIEWSTATE, __EVENTVALIDATION). I monitor headers and post vars using Tamper Data so I'm pretty sure I haven't missed anything. My approach is to micmic the post back when the user click on one of the links and parse the response. But the response I'm getting is a page redirect to "Unable to validate data".
View 1 Replies
Jan 7, 2010
I'm trying to write a small application to collect(Scrape) one piece of data from a web site. I would like to be able to simply run the app and it will open the page, find the one piece of data and display it. So far so good...my problem is that the web site is a secure site, meaning I have to provide a user name and password. I've searched all over the web, found many discussions but have yet to find anything that provides specifics on how to accomplish this. I understand a little bit about tokens etc, but I'm really looking for a detailed description of how to do this. Please feel free to direct me to a different forum if I'm in the wrong place.
View 3 Replies
Jan 18, 2010
I am building a site that need to scrape information from a partner site. Now my scraping code works great with other sites but not this one. It is a regular .html site. My thoughts is that it might be generated some how with php (site is build with php).
If it matters here is my code I use. The htmlDocument is htmlAgilityPack but that has nothing to do with it. Result is null on the site I try.
[Code]....
this is from the w3 validator, might have something with this? The site checked is this
[URL]
I am unable to validate this document because on line 422 it contained one or more bytes that I cannot interpret as utf-8 (in other words, the bytes found are not valid values in the specified Character Encoding). check both the content of the file and the character encoding indication.
The error was: utf8 "xA9" does not map to Unicode""
View 4 Replies
Dec 22, 2010
we need to compare the content of 2 identical gridviews and extract rows that differ in a third gridview, is this doable?at our office we take daily backup of ASP.net application ms access backendfor the next few days we need to evaluate the changes made to records in the database tablesat the end of each day i want to compare 2 access databases first database is the backup of yesterday and second database is the backup of todayi thought of the following algorithm, please read carefully and tell me how to proceed to compare the datatables / gridviewsi need to display th rows / cells containing the differences / updates / deleted datacomparing two ms access backend databases of an asp.net web application
View 2 Replies
Mar 18, 2010
I've created a web application in asp.net so far. where I've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but I haven't accessed the secure page yet. Every time
the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.
View 7 Replies
Aug 31, 2010
I need to screen scrape a web page and change its style to match the look and feel of the site where it will be displayed in. Is this possible? I'll be using asp.net to do the screen scraping.
View 1 Replies
