How To Scale Document Storage System
Feb 1, 2011
I maintain a web application (ASP.NET/IIS7/SQL2K8/Win2K8) that needs to access documents, actually hundreds of thousands of documents, and growing. Currently, they are all on a Windows 2K8 Server fileshare, being accessed by UNC path (SMB). The files are in a single flat directory and I'm trying to plan how to best improve this solution. I don't want to use the SQL Filestream attribute as it would be significant effort to migrate it all into that, and would really lock in to SQL Server. I also need to find a way to replicate the data for disaster recovery, so perhaps a solution can help with that too. Options could be: Segment files into multiple directories? application would add metadata for which directory it's on (or segment by other means) Segment files into separate servers? (virtualize)Backup becomes more complicated. Application would add metadata for which server it's on
NAS Storage
SAN Storage
Put a service (WCF) in front of the files and have the app talk to the service bonus of being reusable across many applications Assuming I'm going to store on filesystem and not in database (I've read those disccusions here), which would be a more scalable solution?
View 1 Replies
Similar Messages:
Dec 21, 2010
In our organization we had to write code to store and retrive documents in many apps. Now I would like to write a service/library that can be used my other devs. Not sure how to start about that? What should be my first step.
View 5 Replies
Apr 16, 2010
I want to build a custom document management web application that ties in with sharepoint for the actual document versioning and storage. I'm hoping for something like a sharepoint widget that I can plug into my web application that would allow me to tie in with sharepoint and download documents, make edits to them, and upload them back to sharepoint, with sharepoint handling all of the versioning and storage. Basically I'm looking for a sharepoint API. If WSS is the answer to this, are there licensing issues that I need to consider?
View 4 Replies
Jan 11, 2010
What is the best approach to storing images in a file system. Currently I am separating the images by day, month and then year. I am also hashing file name so that no two files will be stored with the same name. Lastly, how would I go about using the image file address in a forum?
View 3 Replies
Nov 6, 2010
Currently, I am storing images and video clips in my database as a varbinary(max).
The problem is that when my front-end goes to retrieve and play the video, it takes an excessive amount of time to download.
For example, I have a video file: test.avi (5mb). If I attempt to read from database and play avi, it takes about 40 seconds. If I attempt to play this test.avi from the filesystem, it takes about 10 seconds to load.
I have investigated compression/decompression, but those seem to be as time-consuming as just loading up the raw data from the database.
Should I scrap storing video clips in the database, and go for a file system approach. I would really like to keep everything clean, secure, and organized. Which is why I am storing images, video clips, etc. in database.
View 4 Replies
Feb 10, 2011
How can i retrieve a document saved in system folder whose path is stored in sql db...
View 3 Replies
Jun 16, 2010
Is it a good idea to create a document managament system that stores all kinds of documents inside sql server database. This will have infrastructure to accomodate all kinds of applicartions..
View 2 Replies
Apr 5, 2010
We have created an ASP.NET web app that upload files to WSS Doc Libary. The files get added under 'SYSTEM ACCOUNT' in the library. The FullTextSqlQuery class is used to search the document libary items. But it only searches files that has been uploaded by a windows user account like 'Administrator' and ignores the ones uploaded by 'SYSTEM ACCOUNT'. As a result the search results are empty even though we have the necessary data in the document library. What could be the reason for this?
The code is given below:
public static List GetListItemsFromFTSQuery(string searchText)
{
string docLibUrl = "http://localhost:6666/Articles%20Library/Forms/AllItems.aspx";
List items = new List();
DataTable retResults = new DataTable();
SPSecurity.RunWithElevatedPrivileges(delegate
{
using (SPSite site = new SPSite(docLibUrl))
{
SPWeb CRsite = site.OpenWeb();
SPList ContRep = CRsite.GetListFromUrl(docLibUrl);
FullTextSqlQuery fts = new FullTextSqlQuery(site);
fts.QueryText =
"SELECT Title,ContentType,Path FROM portal..scope() WHERE freetext('" +
searchText +
"') AND (CONTAINS(Path,'"" +
ContRep.RootFolder.ServerRelativeUrl + ""'))";
fts.ResultTypes = ResultType.RelevantResults;
fts.RowLimit = 300;
if (SPSecurity.AuthenticationMode != System.Web.Configuration.AuthenticationMode.Windows)
fts.AuthenticationType = QueryAuthenticationType.PluggableAuthenticatedQuery;
else
fts.AuthenticationType = QueryAuthenticationType.NtAuthenticatedQuery;
ResultTableCollection rtc = fts.Execute();
if (rtc.Count > 0)
{
using (
ResultTable relevantResults =
rtc[ResultType.RelevantResults])
retResults.Load(relevantResults, LoadOption.OverwriteChanges);
foreach (DataRow row in retResults.Rows)
{
if (!row["Path"].ToString().EndsWith(".aspx"))
//if (row["ContentType"].ToString() == "Item")
{
using (
SPSite lookupSite =
new SPSite(row["Path"].ToString()))
{
using (SPWeb web = lookupSite.OpenWeb())
{
SPFile file = web.GetFile(row["Path"].ToString());
items.Add(file.Item);
}
}
}
}
}
} //using ends here
});
return items;
}
View 1 Replies
Mar 28, 2011
based on a recent article on MySpace blaming the microsoft stack not being able to easily scale here.Is this true? Is scaling an app built on the microsoft stack (asp.net mvc here) nearly impossible? does php/LAMP really scale better than microsoft?
View 1 Replies
Feb 7, 2010
I'm thinking of using ASP.NET and MySQL together on a large project. Does anyone have any experience in using these two on a large scale and are there any aspects I should be wary of? If there is a chance that there could be compatibility issues, then I may be prepared to purchase an MSSQL license, but I'd rather go the free route right now. In theory it's possible, but how easy is it to transfer a MySQL database to and MSSQL database in the future?(I'll make this a CW if requested).
View 2 Replies
Nov 8, 2010
Okay in my WebApplication i have a webform showing some pictures.The pictures - or better - the picturepath i get from a database.
With this paths i am loding the pictures into my webform.
Now i want to render a scale into these pictures. For example lets say 1px = 1cm.It should look something like in this sample picture:
Is it possible to realize somethin like that and how?!
I tryied to google something but i could not find anything!
View 6 Replies
Jan 25, 2010
I am using abcPdf to convert an HTML report into a pdf file. The pdf has to be a single landscape A4 page.Do you know if there is any way to tell abcPdf to scale the HTML page to fit on a single page in the pdf? I tried using the Magnify() method, and it scales the content but still breaks it into pages, even though it would fit on one page. I've been scratching my head on this for a while now
View 2 Replies
Mar 4, 2010
i wants to upload images on server with small scale, large scale and medium scale on server hard disk when user uploads picture on server by upload image click..
View 1 Replies
Feb 21, 2011
I have to do the following.
There is a database (SQL Server) with locations (longitude and latitude values). I need to plot the route on the map (Google or otherwise) based on those locations extracted from database. Plotting lines and map are on ASP.NET webpage.
What would be the most and cost-effective way to accomplish the task above?
View 1 Replies
Feb 20, 2010
I am working on a site where I will need to be able to track ad impressions. My environment is ASP.Net with IIS using a SQL Server DMBS and potentially Memcached so that there are not as many trips to the database. I must also think about scalability as I am hoping that this application becoming a global phenom (keeping my fingers crossed and working my ass off)! So here is the situation:
My Customers will pay X amount for Y Ad impressions These ad impressions (right now, only text ads) will then be shown on a specific page.The page is served from Memcached, lessening the trips to the DB When the ad is shown, there needs to be a "+1" tick added to the impression count for the database
So the dilemma is this: I need to be able to add that "+1" tick mark to each ad impression counter BUT I cannot run that SQL statement every time that ad is loaded. I need to somehow store that "+1" impression count in the session (or elsewhere) and then run a batch every X minutes, hours, or day.
Please keep in mind that scalability is a huge factor here.
View 2 Replies
Jun 7, 2010
On my X Axis, I have months. The chart shows up to 11 points, i.e. Jan - Nov of the same year, but when I add 12 points (Jan - Dec), it will do an auto label thing and change the interval for every 4 months.
How can I change the graph so that it shows 12 months before it does the auto labels?
Here is the server control code I am currently using.
[code]....
View 1 Replies
Jan 24, 2014
<iframe src="http://docs.google.com/viewer?url=filename&embedded=true" style="width:600px; height:600px;margin-left:20px;" frameborder="0"></iframe>
I am using above code to show our document online.
But when we inspect element it is showing url of file which we dont't want.
it is possible to hide this url or any other alternate to show our document to user without exposing our physical file path.
View 1 Replies
Aug 16, 2010
I have xml document and wanting to pass it through wcf restfull service, i am able to do that by using string format i mean i can pass the xml as string param in the restful service and it worked but now my xml document becomes huge so if i pass it as string then it gives URI long error. any way i can pass the huge xml document through the WCF RESTFul service.
View 1 Replies
May 9, 2012
I want when user upload a doc file it convert into pdf. How can i do this. I read that itextsharp convert it.
View 1 Replies
Sep 15, 2010
I am launching a pdf document using either:
1. Response.Redirect("filepath/document.pdf"), or
2. Response.ContentType = "Application/pdf"
Response.WriteFile("filepath/document.pdf")
Response.End()
Both methods successfuly launch the PDF document displaying the document path and title in the title bar. For security reasons I'd like to HIDE the document title. Can someone advice how this can be done? (How can a PDF document be launched without showing the actual document's name)
View 14 Replies
Jan 10, 2010
A few years ago, when first being introduced to ASP.net and the .NET Framework, I built a very simple online file storage system.This system used Rijndael encryption for storing the files encrypted on the server's hard drive, and an HttpHandler to decrypt and send those files to the client. Being one of my first project with ASP.net and databases, not understanding much about how the whole thing works (as well as falling to the same trap described by Jeff Atwood on this subject), I decided to store freshly generated keys and IVs together with each file entry in the database.
To make things a bit clearer, encryption was only to protect files from direct access to the server, and keys were not generated by user-entered passwords. My question is, assuming I don't want to keep one key for all files, how should I store encryption keys for best security? What is considered best practice? (i.e: On a different server, on a plain-text file, encrypted). Also, what is the initialization vector used for in this type of encryption algorithm? Should it be constant in a system?
View 2 Replies
Dec 26, 2010
User images - database vs. filesystem storage
Where should i store photos for a facebook-like app ? locally on the server in a folder or upload them to a database ? I am not intending to make this app commercial .
View 2 Replies
Oct 13, 2010
Currently I am looking to move my websites images to a storage service. I have two websites developed in PHP and ASP.NET.
Using Amazon S3 service we can host all our images and videos to serve web pages. But there are some limitations using S3 service when we want to serve images.
If website needs different thumbnail images with different sizes from original image, it is tough. We have again need to subscribe for EC2 also. Though the data transfer from S3 to EC2 is free, it takes time for data transfer before processing image resize operation. Uploading number of files in zip format and unzipping in S3 is not possible to reduce number of uploads. Downloading multiple files from S3 is not possible in case if we want to shift to another provider. Image names are case sensitive in S3. Which will not load images if image name does not match with request. Among all these first one is very important thing since image resize is general requirement. Which provider is best suitable to achieve my goal. Can I move to Google AppEngine only for the purpose of image hosting or is there any other vendor who can provide above services?
View 3 Replies
Feb 2, 2011
Im using Visual studio 2008 in a Windows 7 (32 bit) operating system. Ive just started getting this error message randomlyNot enough storage is available to process this commandIve googled the message and it seems like the solution is to increate the amount of RAM visual studio usesIve tried to follow the instructions herehttp://stevenharman.net/blog/archive/2008/04/29/hacking-visual-studio-to-use-more-than-2gigabytes-of-memory.aspxbut when I try to use bcdedit, I get this errorboot configuration data store could not be opened access is deniedso I tried the suggestion herehttp://social.answers.microsoft.com/Forums/en-US/w7repair/thread/9e995bc8-141c-4ed0-9b17-2dbe92369202Im confused by this linec. At the command prompt, the following line, and then press ENTER:bcdedit /set {current} Description "name you want"and press Enter.
View 7 Replies
Jun 10, 2010
I want to convert to Word Document in a PDF document. I do not want to use any third party control. I want to do it on my own. Can somebody tell me how to start with it?
View 7 Replies
