Security :: Screen Scraping On A Secure Site?
Jan 7, 2010
I'm trying to write a small application to collect(Scrape) one piece of data from a web site. I would like to be able to simply run the app and it will open the page, find the one piece of data and display it. So far so good...my problem is that the web site is a secure site, meaning I have to provide a user name and password. I've searched all over the web, found many discussions but have yet to find anything that provides specifics on how to accomplish this. I understand a little bit about tokens etc, but I'm really looking for a detailed description of how to do this. Please feel free to direct me to a different forum if I'm in the wrong place.
View 3 Replies
Similar Messages:
Mar 18, 2010
I've created a web application in asp.net so far. where i've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but i haven't accessed the secure page yet. Every time the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.
View 1 Replies
Jan 18, 2010
I am building a site that need to scrape information from a partner site. Now my scraping code works great with other sites but not this one. It is a regular .html site. My thoughts is that it might be generated some how with php (site is build with php).
If it matters here is my code I use. The htmlDocument is htmlAgilityPack but that has nothing to do with it. Result is null on the site I try.
[Code]....
this is from the w3 validator, might have something with this? The site checked is this
[URL]
I am unable to validate this document because on line 422 it contained one or more bytes that I cannot interpret as utf-8 (in other words, the bytes found are not valid values in the specified Character Encoding). check both the content of the file and the character encoding indication.
The error was: utf8 "xA9" does not map to Unicode""
View 4 Replies
Mar 18, 2010
I've created a web application in asp.net so far. where I've tried to get some data(site scraping) from secure page of a web site.I've used the HttpWebRequest class for this functionality but I haven't accessed the secure page yet. Every time
the login pages was scraped not secure page.I have the site user id and password and don't know that which language site has been developed in.
View 7 Replies
Apr 8, 2010
Every night I have a program that runs that creates a natural gas report. In this report is something called "expert analysis." The is just text which is a commentary of the current Natural Gas market and because it's a volatile market there is new commentary every day. So also every day, an employee in our office runs a public webpage which displays this text, copies and pastes it into a webpage that is part of our system and it gets saved to a file which gets pulled into our nightly report.
The page that is in our system is as ASP page and it uses FileUp to create the file. We want to break our dependancy on FileUp (it's a SoftArtisans product, an excellent vendor, but we are trying to save some money). Right now we don't have FileUp because our trial expired. So every day I run the public webpage and copy the text and paste it to my expert.txt file all manually. Another problem is that the expert analysis is updated after our office staff has all gone home,
Here is my question - can I write a program in .NET that runs a page, parses the resulting HTML and extracts out the expert analysis all automatically? I won't need FileUp anymore becaus I can use the .NET upload control. And I won't have to log on every night to manually copy and paste the expert analysis. Is that what web/screen scraping is? What happens then the public page's format changes because they do maintenance on their site? Is it just like if I were to run [URL] and try to read the Moderator column of the ASP.NET forum? I'd be okay for a while (would I?) but then if the layout changed I wouldn't find the Moderator anymore?
View 14 Replies
Aug 31, 2010
I need to screen scrape a web page and change its style to match the look and feel of the site where it will be displayed in. Is this possible? I'll be using asp.net to do the screen scraping.
View 1 Replies
Feb 8, 2010
I don't think the site has a web service available, so I need to pass the login credentials and read the xml file (e.g.https://www.somesite.com/needthisfile.xml). What do I use to do this?
View 1 Replies
Nov 11, 2010
I am new to web programming. I am developing a web program using asp.net(vb) that scrapes data of a certain website. I am using System.Net.HttpWebRequest and System.Net.HttpWebResponse to read the HTTP codes.My problem is I can not retrieve the codes of certain frame/container where the data that I needed is located in the website.I understand that iframe has its own URL or link aside from the main URL of the website
View 1 Replies
Apr 8, 2010
Requirment is to stop more than one user to login at a time using same user id.I am storeing user id in session. I searched the internet and found some ways but somehow they do not work in these situations:If javascript in brower is turned off. If user do not click logout and directly close brower.
View 3 Replies
Oct 28, 2010
I have a doubt regarding secure cookie.
I have four servers 2 App Server(behind firewall) and 2 webservers and each server is accessing through Load Balancer.
App Server is a modules for Data Access layer and webserver is for Presentation layer.
My Issue is My Load Balancer has Secure certificate installed but certificate is not installed on servers and I want to make cookie as secure in site , as per my understanding " cookie should be set as secure only for SSL site other ways new session will get create every time" so should I install SSL certificate on webserver to make my cookie as secure or will it work properly even if only Load Balance has SSL.
View 5 Replies
Jul 28, 2010
I have 2 sites, one the main product site the other a secure site featuring the basket and checkout processes.
When a user clicks "add item to basket" I am creating a basket in a database and then adding the items to the basket. The theory was then to store the id from the database for the basket (the basketid) in a cookie, redirect the user to secure site, use the basket id from the cookie and display the contents in the basket.
However I am having trouble getting the secure site to use the same cookie. When I redirect the users to the secure site, the basket appears empty. I need to go back to the main site and then back to secure before the cookie appears.
Here is the code for the cookie
[Code]....
And then on the secure site this is how it gets the basket ID from the cookie
[Code]....
But like I say it's having problems. I've tried replacing the cookies with session variables as well but it didn't work.
View 1 Replies
Oct 24, 2010
I have a custom mini login user control that I have embedded in the top of my website which shows on every page. These pages are non-secure HTTP://. I would like to avoid having to redirect the user to a HTTPS page to perform the login but I definitely don't want to send login credentials to the server in plain text.
I am trying find a method to send the user's login credentials encrypted via https from a non-secure (http) page.
I tried to set the postbackurl for the login button to itself but in https, but the user's input is not retained and the buttonLogin_click is not fired when I set the button postbackurl property. My ASP.net web application is VB.Net framework 4.0
I am assuming this can be done because I see lots of websites where login fields are on available on every page and they are running http and I can believe they are not encrypting the login credentials.
View 3 Replies
Jan 31, 2011
I have precompiled and encrypted my webconfig. What else could I do to make my site secure.
View 2 Replies
Nov 23, 2010
I want to develop a site with secure login. i followed various tutorials, and created a site, based on the default master page. I believe i set up the asp.net configurations correctly. under security, changed authentication type to "from the internet", added roles, added users based on those roles. when i compile the project and test the log in, it works fine. but, in visual studio 2010's Solution Explorer the App_Data folder is grayed out and i cannot get into it to view the database. how do i view the tables and create new tables?
View 4 Replies
Jul 14, 2010
I need to get the calling page url referrer and host name. I works for me if the calling pages are within the same network. When a page is called from an internal site that are outside of my network, I got the following error:
Object reference not set to an instance of an object.
It failed on this line:
string userURL = Request.UrlReferrer.ToString();
View 1 Replies
Apr 24, 2010
How can I prevent my asp.net 3.5 website from being screen scraped by my competitor? Ideally, I want to ensure that no webbots or screenscrapers can extract data from my website. Is there a way to detect that there is a webbot or screen scraper running ?
View 7 Replies
Nov 19, 2010
I have a internal site used by employees of my company that uses ASP.Net 3.5, Ajax and the Ajax Control Toolkit and is secured using https and requires a client certificate to gain access to the site. We are getting the following error when updating drop down lists on the very first page of an application wizard we have built. The page uses an UpdatePanel with PartialPagePostbacks enabled.
Sys.WebForms.PageRequestManagerServerErrorException:
An unknown error occurred while processing the request on the server. The status code returned from the server was: 413
ScriptResource.axd Line: 5
Code: 0 Char: 62099.
I have increased the maxRequestLength value to 8192Kb (which is way overkill), the readAheadBuffer to 4096Kb, set all the timeouts I can find, and this problem keeps showing up for my users. I can reproduce the error if I let the initial page site idle for 2-3 minutes before doing the partial page postback to move to the next segment. I have been chasing this now for a couple of weeks, and keep coming back to the same sites with the same answers.
View 4 Replies
Oct 1, 2010
I have 2 sites, one the main product site the other a secure site featuring the basket and checkout processes.
When a user clicks "add item to basket" I am creating a basket in a database and then adding the items to the basket. The theory was then to store the id from the database for the basket (the basketid) in a cookie, redirect the user to secure site, use the basket id from the cookie and display the contents in the basket.
However I am having trouble getting the secure site to use the same cookie. When I redirect the users to the secure site, the basket appears empty. I need to go back to the main site and then back to secure before the cookie appears.
Here is the code for the cookie
[Code]....
And then on the secure site this is how it gets the basket ID from the cookie
[Code]....
But like I say it's having problems. I've tried replacing the cookies with session variables as well but it didn't work.
View 6 Replies
Jun 7, 2010
i am creating a web app for a small buisness enterprise in which i have added a node in my menu (stocks) accessible to limited users
every time a user try to access this node , it will ask for his/her credentials. iam using asp.net login control( in built ) for this . Also i used
<authentiaction mode='forms' />
<authorization>
<deny users='?'>
</authorization>
for these pagesin my web.config file now apart from that , how can i made it more secure using asp.net membership provider.
View 2 Replies
Jan 13, 2011
I'm creating a ASP.NET/SQL Server 2005 Express app which requires users to upload supporting attachment or documents in .docx and .pdf format. The web server is IIS 6/Win 2003. I'm trying to come up with a good way of securing these documents so anonynous users cannot see them and creating a separate Windows login for every user is not an option. SQL Server Express does not have blob data types. I'm storing each user's authentication information in the SQL database and not really using forms authentication.
View 1 Replies
Apr 9, 2010
I would like to send email securely to third party with attachments containing sensitive info. The website is configured with SSL. I'd like to know how to send email securely via asp.net app.
View 3 Replies
Jan 24, 2010
i have asp.net 2.0 web site and in the main root i have some secure pages so if any unauthenticated user try to visit my page the browser redirect him to login.asp page (it was correct) but in my web i have subfolder named (Admin) and i want to secure all pages under this folder and redirect any user to new login page. how can i do it?
View 7 Replies
Apr 30, 2010
I've got IIS 6 with a self-signed certificate installed. This is now securing the whole site (all urs are HTTPS). How do I only apply HTTPS to logged in areas, leaving publicly viewable data with HTTP?
View 8 Replies
Apr 22, 2010
I'm auto-generating a form in my ASP.NET page. This is already tested and working. I want to know if:If there are any security problems with storing the database ID as part of my controls ID? I can see think of 2 issues: the id will be visible in page source (not really important in this case), and the possibility someone could change the name of the control somehow? This second possibility is more serious. Is this a potential problem and how to void it?If there would be a better preferred way to associate a unique data with any type of control? Is it possible to store a custom item in the viewstate for the control?
View 5 Replies
Mar 11, 2010
I'm building a forgot password feature for my first secure backend.
I want to email a url containing an encrypted parameter to the user which will lead them to a password reset form that can only be accessed via the url with the encrypted parameter.
But what if the email gets intercepted?? then anyone who intercepts it will have the link to the reset page.
Is there a way I can send a secure email? I know nothing about email really. Is it even possible to send a secure email? Can I encrypt the email, will that help? But if it is encrypted then how does the recipient read it?
View 4 Replies
