C# - .NET Automated & Manual ETL Process?
Oct 28, 2010
Here's an interesting problem. I have an ETL script written in c# that I have been running manually on a somewhat regular basis. It is used to update my web app's database.
I want to automate the ETL process AND create an interface for the web app admins to manually start the ETL process.
I could have sql server kick off the ETL process on a schedule and implement a button or something on a web page that will do the same thing, but I don't want to put my code in 2 different places because I don't want to update it in 2 places when it changes. But I don't know how to make my web app tell SQL server to manually start a scheduled process. Can this be done? OR
I could somehow implement the scheduling in the web app itself, but by now most people are familiar with the problems that are faced when trying that (app may not be running at certain times, must wait on request to start a process (without some trickery)). Also, since the ETL process takes a while, I don't want to make some poor end user wait on a response, so it would definitely have to use a new thread.
View 1 Replies
Similar Messages:
Sep 7, 2010
I've written a aspx.net(C#) page that after the user has done what they do on the page they hit a Submit button. The Submit button
1) takes the work the users was producing on the page and writes it to a file.
2) attempts to execute a batch file.
The call to the batch file is executed via:
protected void Execute123EDI()
{
try
{
string File = @"c:80sAdminSendV80s.bat";
lblCancelled.Text = lblCancelled.Text + File;
Process proc = new System.Diagnostics.Process();
proc.StartInfo.UseShellExecute = false;
proc.StartInfo.LoadUserProfile = false;
proc.StartInfo.UserName = "administrator";
proc.StartInfo.RedirectStandardOutput = true;
System.Security.SecureString secPass = new System.Security.SecureString();
string paswd = "123abc";
for (int i = 0; i < paswd.Length; i++)
{
secPass.AppendChar(paswd[i]);
}
proc.StartInfo.Password = secPass;
proc.StartInfo.FileName = File;
proc.Start();
FileStream fs = new FileStream(@"c:80sAdminSendV80sOutput.log", FileMode.Append);
StreamWriter sw = new StreamWriter(fs);
sw.Write(proc.StandardOutput.ReadToEnd());
proc.WaitForExit();
sw.Close();
proc.Close();
}
catch(Exception ex)
{
lblDebug.Text = lblDebug.Text + ex.Message + "<br/>";
}
If I watch the processes tab in Task Manager on the web server I see "cmd.exe" under the context of 'administrator' but it just hangs. For test purposes c:SendV80s.bat: copy c: oot.ini c:zzz.txt
If I logon onto the webserver's console and execute SendV80s.bat it works and exits without issue. But when I execute the same batch file via the Submit button it gets stuck executing in Task Manager/Process. I believe this has something to do with the fact that cmd is not running in a full environment/desktop context. I just noticed this on the actual console of the webserver (not in my RDP console but console 0 instead)
A pop-up box stating: CMD.exe Application error The application failed to initialize properly (0xc0000142). Click OK to terminate the application. And when I click on the OK button my ASPX page's WaitForExit is satisfied and the continues processing normally.
View 2 Replies
Jul 21, 2010
I know that similar questions have been asked all over the place, but I'm having trouble finding one that relates directly to what I'm after.
I have a website where a user uploads a data file, then that file is transformed and imported into SQL. The file could be up to 50mb in size, and some times this process can take 30 minutes or sometimes even longer.
I realise I need to palm off the actual work to another process, and poll that process on the web page. I'm wondering what the best approach would be though? Being a web developer by trade, I'm finding all this new Windows Service stuff a bit confusing, and I just wanted somewhere to start.
So:
Can I do / should I being doing this with a windows service? if so, how?
Should I use WCF? If this runs under IIS, will I have problems with aspnet_wp.exe recycling and timing out my process?
clarifications
The data is imported into sql, there's no file distribution taking place.
If there is a failure, it absolutely MUST be reported to the user. The web page will poll every, lets say, 5 seconds, from the time the async task begins, to get the 'status' of the import. Once it's finished another response will tell the page to stop polling for status updates.
queries on final decision
ok, so as I thought, it seems that a windows service is the best idea. So as to HOW to get it to work, it seems the 'put the file there and wait for the service to pick it up' idea is the generally accepted way, is there a way I can start a process run by the service, without it having to constantly be checking a database table / folder? As I said earlier, I don't have any experience with Windows Services - I wondered if I put a public method in the service, can I call it somehow?
View 2 Replies
Feb 26, 2010
I'm working on an ASP.net app I'm trying to execute a process remotely , using System.Diagnostics.Process class here's my code:
ProcessStartInfo startInfo = new ProcessStartInfo(@"C:TestCommand.exe");
startInfo.Domain = "myDomain";
startInfo.UserName = "MyUserName";
SecureString sec = new SecureString();
foreach (char item in "MyPassword")
{
sec.AppendChar(item);
}
sec.MakeReadOnly();
startInfo.Password = sec;
startInfo.UseShellExecute = false;
Process.Start(startInfo);
I keep receiving an exception with the message "Logon failure: unknown user name or bad password". Im absolutelly sure that i'm submiting my correct username/pwd
View 1 Replies
Apr 3, 2010
I've developed a popup email .aspx used on our intranet based web app that is auto generated with .pdf's attached. I'm developing with VS 2008 ASP.Net 3.5 C# and System.Net.Mail.MailMessage. I can create and send the email with no issues. The problem is with any attempt to open or delete the attachments I get the above error. The .pdf's a copied with the following code:
FileStream fsr = new FileStream(inFilename, FileMode.Open, FileAccess.Read, FileShare.Read);
BinaryReader reader = new BinaryReader(fsr);
byte[] bytes = new byte[fsr.Length];
reader.Read(bytes, 0, bytes.Length);
FileStream fsw = new FileStream(outFileName, FileMode.Create, FileAccess.Write, FileShare.Write);
BinaryWriter writer = new BinaryWriter(fsw);
writer.Write(bytes, 0, bytes.Length);
// clean up
writer.Flush();
writer.Close();
writer = null;
fsw.Close();
fsw.Dispose();
fsw = null;
reader.Close();
reader = null;
fsr.Close();
fsr.Dispose();
fsr = null;
Later after sending the email I:
mailMessage.Dispose();
mailMessage = null;
foreach (string fileName in attachments)
{
if (File.Exists(fileName))
File.Delete(fileName);
}
The error occurs at: the File.Delete(fileName);
how I can delete or reopen these files after sending the email?
View 9 Replies
Dec 25, 2010
Dear ladies and sirs.
I have a .NET infra code running both within the IIS worker process and within a desktop client app. How can the .NET code determine whether it is running within an IIS worker process?
I know that I could check the name of the process (w3wp.exe, for instance), but I would like a more robust approach. I wish to make a side note. This is not a production need. I need this information to enable certain scenarios useful during the development and testing phase. Specifically to ease the testing of secure vs non secure configurations.
View 3 Replies
Jan 26, 2010
When I hit the run button (in my Default.aspx), a process starts (this process contacts a webservice to get some files, etc). How do I: Ensure that only a single process is running at a time (i.e. if I refresh the browser, I don't want to start the process a second time)Track progress - there are 4 points of the process (at 25%, 50%, 75%, 100%) that I want to track, and when each part completes, I want to update the progress bar. I have a status object for the running process, but the question is how to update the progress bar automatically? Do I need to use threads to achieve the above two?
View 1 Replies
Nov 6, 2010
My code is that I want to create a log file and log it upon a new user browsing the site. However, what i did was I put in a 6 second delay and then used another browser to access the page. And it threw an Exception saying it is being used by another process which is true. So how come I set it so that, if IT is being used by another process, WAIT and retry every 500 milliseconds until it becomes free/available?
here's the code:
protected void Page_Load(object sender, EventArgs e)
{if (!IsPostBack) // if this is the first time page loads, set k to 1
{ lognewuser();
}}
[Code]....
View 1 Replies
Mar 28, 2011
I am having the text file which is used to track all the ip address which is available in the network and replace the content from"Reply from 172.29.116.3: bytes=32 time=1ms TTL=255" to 172.29.116.3.
For this task i am having 2 functions 1.runCMD() Function is used to create a file which ping all the ip address between 3 to 254 ("Reply from 172.29.116.3: bytes=32 time=1ms TTL=255").
2.Another function textFileReplace() which is used to replace the text from "Reply from 172.29.116.3: bytes=32 time=1ms TTL=255" to 172.29.116.3.
This process will continue every 30 minutes..
But i am having the error while accessing the function textFileReplace() as The process cannot access the file 'C:inetpub' because it is being used by another process.
View 4 Replies
Aug 23, 2010
User registers for the siteAn email is sent to the user confirming their registration am using : Asp.net 3.5(C#)itefinity as a CMSMS SQL 2008 server r2My question is what is the best way to automatically send the email?Should it be done in ASP.net or as a database trigger using something like xp_sendmail
View 3 Replies
Nov 10, 2010
to create automated sms reminder.
View 2 Replies
Sep 24, 2010
I have a bunch of C# functions with string, int and bool arguments that serve as data entry interfaces. I'd like to be able to create a webform for each method; a textbox for each string/int and a checkbox for each bool.
Is there a way to automate this process?
View 1 Replies
Apr 23, 2010
I have to login with a username / password with a c# program with asp.net form.
I have already do that with HttpUtility on PHP website but how to do that with ASP.NET website ?
In ASP.NET, I must handle postback and so on...
I have to privilegied access to this website. I have to login like any other visitor. I don't know if it clarify enough the question
View 3 Replies
Feb 25, 2010
I am not able to send mails thru my ASP.net pages. If I host site with 172.0.0.1 IP on other hosting server then it will work but if I set the 202.71.148.84 IP provided by my Hosting Provider that time I can't if any other information is required to address this issue.
View 2 Replies
Mar 12, 2010
I'm working on a site that will offer a free service. It's a very simple site: A visitor will choose some text files from their PC, and once he has finished choosing them, he will click on the "Submit" button. This "Submit" button will simply upload the text files to X folder on the server and run a 3rd-party executable that will process these files. That's it. The exe will be doing all the work and creating a new file that the user will be downloading. This is the problem: I've found out the hard way that shared hosting plans don't allow executables on their servers. So it's been suggested that I purchase a VPS or dedicated server plan for this, but the cheapest one I found is $39/mo.
Although my site will have a "premium" service, it's mostly free, and I wasn't expecting to spend too much on hosting. As a matter of fact, I was planning on using my current shared plan for this. My question is: is there any other way of being able to do this? It would need to be automated; being free, tons of people (hopefully) will go to the site, start uploading text files, and wait for the result. I could do it manually (download the text files from the server, run the EXE locally, and finally upload the result and email the user), but that's almost impossible.
View 2 Replies
Apr 5, 2011
Iīm trying to automate a confirmation email going out to a newly registered user.
I've added reference to the interop.outlook library and I import it to the asp webpage. It actually works, so thatīs fine.However I donīt really know whatīs going on here, so maybe you guys can tell me?
1. What needs to be installed on the server for this to work?
2. Who will be the sender? When I do this locally, Iīm the sender, but on the site??
3. Will it work?
View 4 Replies
Sep 9, 2010
These days I started new ground in learning intellectual website for example I want to understand the number and name and type of tables and columns and make button and interface of website with all aspx and cs and classes and to cut long story short blank solution and sth like this but i do not know where I should start .I prefer to have suggestion of some good books whith functional example I searched in alot of website but I could not find anything ,now i am about losing my time and energy
View 1 Replies
Feb 10, 2011
I have a class with many properties - strings, booleans, etc. This class was created from Entity Framework.
Now I want to quickly create the controls for the properties. I dont care about linking the controls to an object - I can do this later. I just find it a pain to create 50+ controls on the webpage when the class could be used to create a 'template'.
View 1 Replies
Jun 4, 2010
What are some good automated tools for load testing (stress testing) web applications, that do not use record and replay of HTTP network packets?I am aware that there are numerous load testing tools on the market that record and replay HTTP network packets. But these are unsuitable for my purpose, because of this:he HTTP packet format changes very often in our application (e.g. whenwe optimize an AJAX call). We do not want to adapt all test scripts just becausethere is a slight change in HTTP packet format
View 5 Replies
Mar 22, 2011
I'm developing a system that should send emails to clients on specified dates. i don't have much idea about it.i have created a simple windows service .how to get the email address from the sql server database and attach to windows service?
View 2 Replies
Feb 6, 2010
I have a website which needs to send automated emails. I have the emailing set up fine but what I haven't managed to do is embed an image in the messages. I have tried the following code :
[Code]....
All this does is is place the standard "box with red cross" in the email body and attach a '.dat' file.
View 3 Replies
Mar 7, 2011
I have a problem with using menu control in ASP.NET. I am using a maser page with web.sitemap and menu control. I am designed the whole thing with css but it not looks what I want. I have noticed when I build the page it generates some css in the head section and that srcrew my whole design. How can I reset this automated injected css for my menu to I could use pure CSS for my project.
View 4 Replies
Aug 19, 2010
The data that we display in gridview and details view can easily be scraped using automated tools. My question is if someone uses some automated tool and logs into the website and scraps data how can we find that? How can we detect whether a human is viewing the site or a tool? For example one way is by calculating the time up to which a user stays in page from which we can detect whether human intervention is involved. I do not know how to implement that but just thinking about this method. how to detect and prevent automated tools from scraping data from my website? I have used security image in login section, the user has to log in by entering username, password and security image displayed in the login section. Even then in home page a human may type security image and log in to the site and then use an automated tool to scrap data from the site!!When the recaptcha image appears after a period of time then that time alone the user may type the security image and again use automated tool to scrap data from the site. Actually, I have even developed a tool to scrap data from another site. So only i want to prevent this from happening in my site!
View 7 Replies
Mar 7, 2016
I am new to asp.net.
How do I automatically update a Label.text (with javascript) ?
View 2 Replies
Jul 23, 2010
I have a stored procedure which fetches data after joining 8-9 tables and inserts that into a temp table. It was running fine till now, but now when the amount of data fetched have exceeded 20000, the SP is breaking. I have debugged the sp and found that this main query is failing after returning arround 15000-16000 records.
The error message says Transaction (Process ID) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. I know what is deadlock, but when i checked with SP_LOCK stored proc i only found that my process is running on database only. So how is this getting deadlocked when there are no other processes running simultaneously on server.
View 4 Replies
