C# - Crawler Webresponse Operation Timed Out
May 18, 2010
I have built a simple threadpool based web crawler within my web application. Its job is to crawl its own application space and build a Lucene index of every valid web page and their meta content. Here's the problem. When I run the crawler from a debug server instance of Visual Studio Express, and provide the starting instance as the IIS url, it works fine. However, when I do not provide the IIS instance and it takes its own url to start the crawl process(ie. crawling its own domain space), I get hit by operation timed out exception on the Webresponse statement. Could someone please guide me into what I should or should not be doing here? Here is my code for fetching the page. It is executed in the multithreaded environment.
private static string GetWebText(string url)
{
string htmlText = "";
[code]...
View 1 Replies
Similar Messages:
Jan 15, 2010
I am trying to load the digg favorites rss items. But getting error as operation timed out.Please let me know if anyone have solved this issue before.
Public Shared Function HasRssItems() As Boolean
Dim doc As New XmlDocument
doc.Load("http://digg.com/users/Lovelezz/history/favorites.rss")
Return doc.SelectNodes("rss/channel/item").Count > 0
End Function
View 1 Replies
Feb 16, 2011
I´m making a gridview with a sql query where I also do an upload to an FTP address for each column. It works and several files are being send but after 2 minutes I got "the operation has timed out". I have tried setting this in the web.config but it does not solve it:
httpRuntime maxRequestLength="6024000"
What can I possible to? Is the page timing out due to the sql connection or just the webpage request?
View 8 Replies
Nov 1, 2010
I have a web page that uses System.Net.HttpWebRequest class that used to work perfectly fine for 2 years, and then all of a sudden I started to get the following error.[WebException: The operation has timed out] System.Net.HttpWebRequest.GetResponse() +5314029Also, I tried the application on my local machine and it works fine, do you think it is a setting change on my hosting server?I already tried the response.close and it didn't work.
View 10 Replies
Apr 16, 2012
sometimes i get timed out & sometimes error sending mail error when i am trying to send even only one mail frm yahoo to gmail or gmail to yahooÂ
i have a function fr emailing where in i pass the parameters needed
View 1 Replies
May 3, 2010
I know this has been asked umpteen times but i cant find an answer to my problem. Very simply, i have a webservice client which is trying to call a webservice. The webservice may take any time from 5mins to 30mins (or more) for downloading message data from pop3 server . However after 2mins i get the following error:
System.Net.WebException: The operation has timed-out
This is an issue with the webservice client timing out calling the webservice - as the underlying webservice still runs in the background
Sequence osf steps when the error occurs:
1.User logins with his credentials
2.Webservice client calls webservice to connect to pop3 server and download messages for that particular user.
Code in webservice for downloading data from pop3 server where the exception occurs
private string ReadOutput(bool bMultiline)
{
try
{
StreamReader srReader = new StreamReader(conn);
string strRet = srReader.ReadLine();
if (bMultiline)
{
// Example #4: Append new text to an existing file
string strLine = "";
while (true)
{
// In a multiline message the . indicates end of message
if (strLine == ".")
break;
// Relay will convert any line with just a . in it to ..
if (strLine == "..")
strLine = ".";
strRet += "
" + strLine;
}
}
return strRet;
}
catch (Exception e)
{
return null;
}
}
3.Exception is caught in webservice client while webservice continues to run.This exception occurs only when the user receives a large message may be above 1mb because i tested with 1MB msg file where i dint have any problem receiving or downloading from
pop3 server where as with 5 MB message file i got this exception .
After going through couple of forums i tested my luck by changing web.config in both webservice and webservice client where i set the attribute of httpruntime Execution timeout to 300 seconds with no success <httpRuntime
executionTimeout="300"></httpRuntime>.
I have this webservice client and webservice running on localmachine(OS-Windows Vista)
View 3 Replies
Mar 7, 2010
in my case, It is the windows service which in running the long running insert/update on a table and I need to cancel the operation from my ASP.NET application. In the above link Burnsys suggests that one should kill the Sql server session. Is that really only way and a goood practice to do this? Also, in the same poset can use SqlCommand.Cancel to cancel. However, I am not sure how can I cancel the command from the windows service from ASP.NET application.
View 1 Replies
Mar 26, 2010
I want to write crawler for screen scrapping What I want is, I want to get price of particular hotel from a website, like here is website e.g. In the above URL, there is list of hotels and its price. I want to get the price of the beaufort
View 3 Replies
Dec 6, 2010
I implemented : <%@ OutputCache Duration="43200" VaryByParam="none" location="Server" VaryByCustom="RawURL" %>
I have got a sitemap.xml with all url(about 12000) possible in my site.
I would now if it's judicious to create an application that will parse my sitemap in
order to request all url?
The Goal is caching all my web site to increase velocity.
I precise my question :Each page take about 10 sec to be created and the cache duration and cache duration is 12 hours
View 1 Replies
Oct 6, 2010
I would like to state literally that I really do not want to reinvent the wheel, but as you know some times we get some unique "Must-requirement(s)" that will hinder us to re-use the open source codes.I want a proper, flawless and consistent Web Crawler. Basically, I want this Crawler (As a Web app NOT desktop app - Of course based on asp.net and C#) to grab the pages of any website and store them locally (Including resources like images and CSS...etc), download them locally and adjust any resource hyperlinks to point to the locally downloaded resources.
I saw HTTrack (http://www.httrack.com/), and it seems quite excellent, but the problem is that I want this Crawler to be a part of a system which includes other features and process. So, I really can't have this Crawler as an external tool.Main challenges:1) User should be able to specify to which level s/he wants to crawl, which means: User might specify a sub-site and want to crawl everything underneath it and not the upper level. You see ? So, there should be full crawling for the entire site and partial crawling.
2) URLs and how to deal with them? I faced some weird URLs and it was hard for me to identify the actual page because there is no file name. How to handle that? For example: (http://www.blue1.com/en/uk/Travel-info/At-the-airport/Security-control/) this is a URL of a website that is built on EPiServer (.Net based) but as can be seen from the URL that there is not actual aspx page, Therefore, how to deal with such URLs ?I have already started developing a POC using HttpWebRequest class, but frankly I am totally dissatisfied with it. It is inconsistent and the generated static content misses a lot of images and styles. Besides, the threads act up sometimes strangely.I would greatly and sincerely appreciate any input (Approaches, source codes, ideas , links...etc)P.S. I already saw: (http://www.codeproject.com/KB/IP/Crawler.aspx) and (http://www.codeproject.com/KB/aspnet/ZetaWebSpider.aspx).
View 1 Replies
Dec 7, 2010
I have an .Net Framework #4.0 application that makes a large number of web requests using the WebRequest/WebResponse classes , as i see it has memory leak (or maybe i am doing something wrong)I Wrote some small simple application that demonstrates this:
class Program
{
public static void Main(string[] args)[code]...
The only one solution i came up with is use GC.Collect() (unmarked in example) , All the object are disposed , all streams are closed , am I missing something ?I found something but i don't understand the reason , if i minimize Console the memory usage decreases and looks O.K , what can be the reason for that is there a problem with Conosole or WinForm .
View 1 Replies
Sep 26, 2011
I'm trying to make a webpage that get's a url from another web page (this other webpage works fine in a browser, both IE and FF) but when I make the webrequest call the returned page is a 500 internal server error. Here's the code:
Imports System.Net
Imports System.IO
Partial Public Class _Default
Inherits System.Web.UI.Page
Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
[Code] ....
Here's the error I'm getting:
Code:
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at SW_GetDownloadFile._Default.GetDownloadFile(String Url) in C:Documents and SettingsvmMy DocumentsVisual Studio 2008ProjectsSW_GetDownloadFileSW_GetDownloadFileDefault.aspx.vb:line 24
The DownloadHandler.jsp page simply displays a text line being a text file download link, there's no html formatting or anything and when I paste that text into the browser it shows the text file contents just fine, I need my asp webpage to get that text file link so it can turn around and open it to parse the info.
I'm not in control of the DownloadHandler.jsp page so i can't alter that, the Java team wont make any changes to it either.
View 6 Replies
Nov 12, 2010
I've just updated my MVC2 project to run MVC3 (RC). Everything is working as expected except for one problem.
I'm running DotNetOpenAuth, but when I go to authenticate, my page renders the string
DotNetOpenAuth.Messaging.OutgoingWebResponseActionResult
instead of authenticating (which worked in the MVC2 app)
I found this question elsewhere on SO, and I did what was suggested, but to no avail.
Here is a clip of my Web.Config
</configSections>
<runtime>
<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
<dependentAssembly>
[Code]....
View 1 Replies
Oct 27, 2010
After a call to initial HttpWebResponse.GetResponseStream() and reading through the stream, that stream is done for and cannot be reused.
I have a situation where I need to examine the content of the response and if it is of a certain data, get another page and then pass the new response down the line. Otherwise, pass down the original response as is. The only problem is that after examining the response to check for this "special data", that response is no good to the downstream code.
The only way, I can think of, to make this transparent to the downstream code, is to create a derived class of HttpWebResponse, and somehow cache the data streamed, and pass that cached stream down the line instead of the initial stream. I'm not sure if that's even feasible since I haven't looked into it further.
View 3 Replies
Apr 19, 2010
In an ASP.NET website, how would I run a process, such as send an email, on a timed interval?
Lets say send myself an email every 4 hours. And assume there is no activity at all on the website during that period.
The only way I can think of doin it is open a web page on the site and have it refresh at a timed interval that is less than the server time out setting. When the page refreshes check to see if it is time to send the email, if so send it.
View 1 Replies
Nov 7, 2010
i have two pages a1.aspx a2.aspx....
if i do nothing in a1.aspx for 1minute...it should redirect me to a2.aspx...how to do this in mvc...or javascript
View 6 Replies
Aug 6, 2010
This is my base class for all pages except EndSession.aspx
override protected void OnInit(EventArgs e) {
base.OnInit(e);
if (Context.Session != null)
{
//check the IsNewSession value, this will tell us if the session has been reset.
[Code]....
But on EndSession I try to navigate back to, say default.aspx, and then this code above just redirects be back to EndSession.aspx.
So for better clarification:
Step 1: Go to mypage.aspx
Step 2: Wait for timeout
Step 3: try to navigate away
Step 4: get redirected to EndSession.aspx
Step 5: try to navigate away
Step 6: GoTo set 4
Setp 6 should be actually being able to navigate away...
View 1 Replies
Dec 21, 2010
I have deployed my .NET application on IIS and Iam facing the Same Problem " the connection has timed out and the server is taking too long to respond" My OS is Windows XP and IIS 5.1.
View 1 Replies
Jan 16, 2011
I am getting
[code]....
When I run the SAME query in the SQL query program I get result within seconds, and in LINQ it takes forever and fails.
View 1 Replies
Mar 15, 2011
is there a way to create a new session if it has timed out ? in mobile appication developing using asp.net?getting this error:the page requires session state that is no longer available. either the session has expired, the client did not send a valid session cookie, or the session state history size is too small. try increasing the history size or session expiry limit
<sessionState cookieless="UseCookies" cookieName="Mobile_Session" regenerateExpiredSessionId="true">
</sessionState>
View 2 Replies
Jul 25, 2010
I am working on a marketing module. This module will allow the admin to setup marketing plans; emails and sms sending to a list of receipients. The admin will select a specific date and time on which the marketing campaign will execute (i.e send emails and sms to a list of users).
Now the problem is that I have no idea of what I shall use to monitor the server time and how I can run a timer to check the date and check campaigns dates.
View 2 Replies
Mar 31, 2010
I am calling web service which takes very long time to execute due to which i get request timed out exception....
I hav kept my web method processing in try catch block....
even i have handled httpException....but it gives Request timed out exception..
View 1 Replies
Mar 15, 2011
is there a way to create a new session if it has timed out ?
View 3 Replies
Nov 1, 2010
I have a thread class which makes a web request. After 20-30 urls request, its throwing exception: System.Web.HttpException: Request timed out. My code is below where it is throwing exception:
httpReq.AllowAutoRedirect = false;
httpReq.KeepAlive = false;
httpReq.Headers.Add("Location", "");
httpReq.Timeout = this.HttpRequestTimeout;
httpRes = (HttpWebResponse)httpReq.GetResponse();
In last line : httpRes = (HttpWebResponse)httpReq.GetResponse(); it is throwing exception. "The remote server returned an error: (403) Forbidden."
View 1 Replies
Aug 19, 2010
I'm looking at automating some reporting that I'm going to be generating. I want the reports to be generated and emailed at a particular time every day. These times will vary, and so will the types of reports.I was wondering if there were any better solution to creating a routine with a timer that executes at a set time every day? If I include this class and routine in the Application_Start of my Global.asax, will it always be running, or is there some awesome EventListener that I'm not aware of?
View 3 Replies
