Web Crawler To Generate Output Cache?
Dec 6, 2010
I implemented : <%@ OutputCache Duration="43200" VaryByParam="none" location="Server" VaryByCustom="RawURL" %>
I have got a sitemap.xml with all url(about 12000) possible in my site.
I would now if it's judicious to create an application that will parse my sitemap in
order to request all url?
The Goal is caching all my web site to increase velocity.
I precise my question :Each page take about 10 sec to be created and the cache duration and cache duration is 12 hours
View 1 Replies
Similar Messages:
Nov 2, 2010
I need to enable caching in my asp.net application, but I do not want to use the webserver's memory for holding cache objects. If I add the page directive for output caching will the page be stored in the asp.net cache object?
View 2 Replies
Oct 28, 2010
You know I have the way to Cache the data I've got from the SQL Server over data caching. In addition I can output cache web user controls.Whats about a web user control contains data from a SQL database? Does it make sense to cache the data and also cache the control?What is the best solution for the combination of these two components?
View 1 Replies
Jun 26, 2010
I'm running an ASP.NET MVC website where several page controllers are decorated with an output cache attribute like:
[OutputCache(Duration = someduration..., VaryByParam = "*")]
These are typically forms where some lookup data are drawn from a database. These data change rarely, so the cache duration is typically set to 24 hours. Anyway, my question is: when the sql server is updated with new data in this area, is there a way of forcing such pages to "refresh" rather than keeping the cached output? Or if this cannot be done, is there at least a way of clearing the whole cache for the website?
View 1 Replies
May 11, 2010
I've got a website that has a number of host headers. Depending on the host header, the results are different - both visually (theme'd) and data.
So lets imagine i have a website called 'Foo' - that returns search results (original, eh?).
Now, the same code runs both sites. It is physically the same server/website (using Host Headers) :-
[URL]
And the data is different for the same search result, based on the host name (ie. us results for [URL] results for [URL]
SO .. if i wish to use OutputCaching .. can this be handled / differ by the host name?
View 2 Replies
Sep 9, 2010
Suppose i have an controller action that i would like to output cache.
public class HomeController : Controller
{
[OutputCache(Duration=10, VaryByParam="none")]
public ActionResult Index()[code]....
I need to dynamically set dependency to an external file inside the code. How can i do that?
View 2 Replies
Mar 25, 2010
I have different urls that points to the same code
www.url1.com
www.url2.com
I need to use the cache, but if the asp net cache is enabled when someone access to www.url1.com next person accessing www.url2.com could get the previously cached data (www.url1.com)
I need to have ALL caches activated except this one.
View 1 Replies
Mar 18, 2011
IIS7, WINDOWS WEB SERVER 2008, ASP.NET 4.0 Does the output cache work when using webgarden?
View 1 Replies
Jul 20, 2010
My web client requires client-side output caching since it makes use of its own Back and Forward buttons (which basically just call the browser back/forward buttons).
The issue I'm having, however, is (of course) when the user signs out of the app. The client-side output cache remains so anyone can click the "back" button and it appears to be signed in again. I've researched a few solutions, but most involve server-side output caching or no output caching at all.
I've had success with server caching and HttpResponse.RemoveOutputCacheItem, but unfortunately I need the client side caching.
Is it at all possible to clear this client side output cache upon logging out?
View 1 Replies
Dec 23, 2010
On our website we use standard asp cache with duration set to 5h.
It works fine, but sometimes the publisher add some special content that need to be showed impatiently on many different sub-pages (example some promoted article).
That's what I need to do it's easy to use page like this:
[URL]
I want to clear SERVER SIDE CACHE.
View 2 Replies
May 18, 2010
I'm trying to cache a complex page with lots of controls on it so that if the user navigates to it later it will look like it did when he last saw it.The page has controls which post back and other controls are populated depending on the selections.
I'm using [Code]....
after the <@ Page directive.
What happens is that a postback gets the cached page, so no processing and it looks the same regardless of user selections. So far so good. But if I browse to another page and return by a link, the cached version is ignored and the default page is created again. This is more or less the opposite to what i want.
I attempted to use the validation callback to ensure that a postback resulted in a new version, and that worked, but it still ignored the cached page if it wasn't a postback.I realise that it will need more work to ensure each user gets their version of the page from the cache, but why bother if this doesn't work.
View 3 Replies
Mar 22, 2010
UserControl: How to set Output Cache duration programaticaly?
View 2 Replies
Apr 15, 2010
I want to use an Enum value for the types of VaryByCustom parameters I will support, is it possible to do this?
I tried setting it in the page itself
<%@ OutputCache Duration="600" VaryByParam="none"
VaryByCustom='<%=VaryByCustomType.IsAuthenticated.ToString(); %>' %>
But this returned the entire literal string "<%=VaryByCustomType.IsAuthenticated.ToString(); %>" inside my global.asax is there any way to do this either on the page itself or from the codebehind? Or is this just something I have to accept is purely magic strings and nothing I can do to add type safety to it?
View 1 Replies
May 17, 2010
I would like to use output caching with WCF Data Services and although there's nothing specifically built in to support caching, there is an OnStartProcessingRequest method that allows me to hook in and set the cacheability of the request using normal ASP.NET mechanisms.
But I am worried about the worker process getting recycled due to excessive memory consumption if large responses are cached. Is there a way to specify an upper limit for the ASP.NET output cache so that if this limit is exceeded, items in the cache will be discarded?
I've seen the caching configuration settings but I get the impression from the documentation that this is for explicit caching via the Cache object since there is a separate outputCacheSettings which has no memory-related attributes.
Here's a code snippet from Scott Hanselman's post that shows how I'm setting the cacheability of the request.
[code]....
View 1 Replies
Feb 4, 2011
It's great to be able to output cache child actions, but how do I invalidate that cache?
I've been trying to use [Code]....
View 3 Replies
Oct 7, 2010
I wrote a Simple Control, that implements ScriptControl. This is holder for JQuery framework:
/// <summary>
/// Generic control with client behavior handled via jQuery
/// </summary>
[code]...
View 1 Replies
Jan 20, 2010
I create a menu control in a user contol, named as RightUserrControl.ascx.
and i kept this RightUserrControl.ascx in nearly in 200 pages.
every thing is working fine , but when ever i am going to any page the page loading is very slow, nearly each page is taking 1 minute to load.
set caching for the user control, if this is the case , do i need to declare output cache in all 200 pages or is there any alternative way to resolve this issue.
View 8 Replies
Aug 27, 2010
I have set the output cache for 5 pages(5 minutes)
I want to clear all these pages from cache on some nutton click
View 2 Replies
Jan 22, 2011
I am developing a webiste in ASP.NET. On one of the pages page named menu.aspx, with the help of a certain login, I am getting the output in this format
foo({"1":"Jeff","2":"Mik","5":"Tom"});
I am using this code to get the output on the JSON form.
[Code]....
However, I must get the output only in simple JSON on the page without any HTML. For eg: Here's the screenshot showing the source of the output page.[URL] Notice that it has only the JSON string as the output. And here's the screenshot showing the output page source (with HTML). [URL] .This HTML is what I have to remove. How can do I do this? The full code of menu.aspx.cs is putup here.[URL]
View 1 Replies
Aug 26, 2010
i have 2 questions about output cache:1.is it just me , or does it seems that it is used only after the second time that the user reaches the webpage? if so, why, and how can i tweak it to my needs? my guess is that it wouldn't be logical to cache every time a user reach a webpage, but only when it happens enough times.2.i know that i can use "varybyparam" for using the output cache when the parameter doesn't change , but it seems that it would use the output cache even if there is no parameter , or if the specified paramter is not there. is there a way to overcome this?3.an old question that somehow marked as "answered" , yet i didn't solve : suppose i created a user control and i gave it values through the CTOR via the aspx file . is it possible to show the values from within the split/design mode? remember that i've talking about ASP.NET and not simply C# winforms . i write this because this question was on :[URL]and the answer that i got was for C# and winforms . i've tried to do the same for ASP.NET , but i've failed.
View 6 Replies
Mar 26, 2010
I want to write crawler for screen scrapping What I want is, I want to get price of particular hotel from a website, like here is website e.g. In the above URL, there is list of hotels and its price. I want to get the price of the beaufort
View 3 Replies
May 18, 2010
I have built a simple threadpool based web crawler within my web application. Its job is to crawl its own application space and build a Lucene index of every valid web page and their meta content. Here's the problem. When I run the crawler from a debug server instance of Visual Studio Express, and provide the starting instance as the IIS url, it works fine. However, when I do not provide the IIS instance and it takes its own url to start the crawl process(ie. crawling its own domain space), I get hit by operation timed out exception on the Webresponse statement. Could someone please guide me into what I should or should not be doing here? Here is my code for fetching the page. It is executed in the multithreaded environment.
private static string GetWebText(string url)
{
string htmlText = "";
[code]...
View 1 Replies
Oct 6, 2010
I would like to state literally that I really do not want to reinvent the wheel, but as you know some times we get some unique "Must-requirement(s)" that will hinder us to re-use the open source codes.I want a proper, flawless and consistent Web Crawler. Basically, I want this Crawler (As a Web app NOT desktop app - Of course based on asp.net and C#) to grab the pages of any website and store them locally (Including resources like images and CSS...etc), download them locally and adjust any resource hyperlinks to point to the locally downloaded resources.
I saw HTTrack (http://www.httrack.com/), and it seems quite excellent, but the problem is that I want this Crawler to be a part of a system which includes other features and process. So, I really can't have this Crawler as an external tool.Main challenges:1) User should be able to specify to which level s/he wants to crawl, which means: User might specify a sub-site and want to crawl everything underneath it and not the upper level. You see ? So, there should be full crawling for the entire site and partial crawling.
2) URLs and how to deal with them? I faced some weird URLs and it was hard for me to identify the actual page because there is no file name. How to handle that? For example: (http://www.blue1.com/en/uk/Travel-info/At-the-airport/Security-control/) this is a URL of a website that is built on EPiServer (.Net based) but as can be seen from the URL that there is not actual aspx page, Therefore, how to deal with such URLs ?I have already started developing a POC using HttpWebRequest class, but frankly I am totally dissatisfied with it. It is inconsistent and the generated static content misses a lot of images and styles. Besides, the threads act up sometimes strangely.I would greatly and sincerely appreciate any input (Approaches, source codes, ideas , links...etc)P.S. I already saw: (http://www.codeproject.com/KB/IP/Crawler.aspx) and (http://www.codeproject.com/KB/aspnet/ZetaWebSpider.aspx).
View 1 Replies
Apr 18, 2010
First off, suggest better ways if you want rather than patch up this code. I am just starting this project and it is first time I have tried to code a web site so anything you suggest is very much welcomed. I have spent the last 2 and 1/2 days trying to find workable answers to this but none I have found and tried seem to fit. If needed I can email screens shots or code. I am trying to construct a website that will have the same main theme throughout as far as the header, navbars sitemap, and footer go. Naturally the content will vary from page to page. I want to use a single master page and css stylesheet for this main theme and I will change the content format as needed per page because each page may vary somewhat.
However, about 15 of the pages will all use the same format for their content and this format will differ from the rest of the site but the format of the main theme will stay the same. So I am trying to create a nested master page to use to format the just the content area for all these pages while retaining the main theme for the header, etc.. I believe I should use a separate css file with the nested master page to handle the formatting of the content areas for these 15 pages. I have code that looks like it works when viewed in web developer but design view but does not work when viewd through a browser (IE, Chrome, Firefox). So far I have the following done in web developer.
If it helps the code and screen shots follow. Master Page called "Parent.master". For all theme throughout site Nested Master Page called "Lab.master". For the 15 like pages Primary stylesheet file for main theme called "StylesheetNew.css" For site theme Secondary stylesheet file to syle the 15 like formatted pages. It is called Labmaster.css Labs.apsx file to use as first of the 15 like pages.
Finally screens shots from web developer and browser. Sorry try as I might I could not capture screenshots and put them into this post. The problem is that web developer shows all the area (many lines high) with gray background and with the XXX text that I want to allocate for content in the 15 pages. Yet all the browser show is one line of text o a white background followed by the footer. It looks like the LabStyleSheet works in Web Developer but not in a browser.
Parent.master
<%@ Master Language="VB" CodeFile="Parent.master.vb" Inherits="MasterPage" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "[URL]">
<html xmlns="[URL]">
<head id="ParentHeader" runat="server">
<title>HX5</title>
<asp:ContentPlaceHolder runat="server" id="headerPlaceHolder" />
<link href="~/StyleSheetNew.css" rel="stylesheet" type="text/css" />
</head>
<body>
<form id="form1" runat="server">
<div id="wrapper" >
<div id="Header" >
<table class="hdrtbl1" >
<tr>
<td id="hdrtd1" style="width:175px; height:100px;" >
<img id="logo" alt="Logo" src="../Images/logo.png" style="width: 136px; height: 87px" />
</td>
</tr>
</table >
<table class="hdrtbl2" >
<tr><td id="hdrtbl2td1"><b> Providing Professional Technical, Manangement and Business Solution</b></td></tr>
<tr><td id="hdrtbl2td2"> <b> Services since 2004 </b></td></tr>
<tr> <td id="hdrtbl2td3" align="center" ><img id="rdln" alt="RedLine" style=" height:3px;" src="../Images/RedLine.png" /></td></tr>
<tr><td id="hdrtbl2td4" > <b>Committed to Excellence In All We Do</b></td></tr>
</table>
</div>
<div id="navbardiv">
<ul id="topnav">
<li><a href="../pages/about.aspx" >About Us <img style=" border: none; " src="../Images/arrow-down.gif" /></a>
<!--Subnav Starts Here-->
<span>
<a style="font-weight:lighter; " href="#" >Who We Are </a> |
<a href="#">Locations</a> |
<a href="#">Business Classifications</a> |
<a href="#">Contact Us</a> |
</span>
</li>
<!--Subnav Ends Here-->
<li><a href="../pages/services.aspx">Services <img alt="arrow" style="border: none; " src="../Images/arrow-down.gif" /></a>
<!--Subnav Starts Here-->
<span>
<a style="font-weight:lighter; " href="#" >Capabilities</a> |
</span>
<!--Subnav Ends Here-->
</li>
<li><a href="#">Customers <img style=" border: none; " src="../Images/arrow-down.gif" /></a>
<span>
<a style="font-weight:lighter; " href="#" >Subnav Link</a> |
<a href="#">Customer Locations</a> |
<a href="../pages/labs.aspx">Labs</a> |
</span>
</li>
<li><a href="#">Careers <img style=" border: none; " src="../Images/arrow-down.gif" /></a>
<!--Subnav Starts Here-->
<span>
<a style="font-weight:lighter; " href="#" >Joining the HX5 Team</a> |
<a href="#">Current Opportunities</a> |
</span>
</li>
</ul>
</div>
<div id="subnavbar" >
<!-- <asp:SiteMapPath ID="SiteMapPath" runat="server">
</asp:SiteMapPath> -->
</div>
<div id="topContent" style="border-bottom:solid 1px #191970;">
<table>
<tr><td style="height:30px; width:900px;" >
<asp:ContentPlaceHolder id="TopPlaceHolder" runat="server">
</asp:ContentPlaceHolder>
</td>
</tr>
</table>
</div>
<!-- <div id="mainContent"> -->
<asp:ContentPlaceHolder id="MainPlaceHolder" runat="server" />
<!-- </div> -->
<!-- <div id="lowerContent"> -->
<asp:ContentPlaceHolder id="LowerPlaceHolder" runat="server">
</asp:ContentPlaceHolder>
<!-- </div> -->
<div id="footerContent" style=" height:50px; width:900px; text-align: right; background-color: #ccc; ">
<span style=" font-family: Arial, Helvetica, sans-serif; font-size: xx-small; font-weight: bold; color: #000080; margin: 30px 20px 0px 0px;"> Copyright 2010 HX5, LLC All rights reserved</span></div>
</div>
</form>
</body>
</html>
View 4 Replies
Mar 29, 2010
Is it correct to implement my caching object like this in my controller :
[code]....
And I Use it like this :
[code]....
View 1 Replies
