i have flat files in some direcotry. each flat file contains around 60,000 rows of data. earlier we were using excel sheets to upload data. but now the challenge is it is taking more time to upload data into DB tables. which is the best way to upload such a huge data into db tables. can we use SSIS for uploading data. if so, how can we achieve that.
View 1 Replies
I have a flat file with text that has a special character like this: "RIBA® Generic Strip Labeling." so the registerd symbol is coming as square when i am simply reading the file in c#. is there a way to read the symbol as is and print on screen? i am working in on a console application. Also other german characters, not all but a few are showing as squares as well.
View 3 Repliesiam Working on Asp.net Web apliation using c#.net. in system folder (Ex: E://Santhu) i have few Flat files and Excels Files .So My Application should read those files automatically and should store the data into my database..
View 1 Repliesretrieving data from sql server 2000 database which has four tables. 3 of them has 256 datafields. And I need to retrieve all columns from all table for last 3years. All tables are related through a coman primary key.
So, Total data is 300,000.
I am looking at upgrading a realtime program ASP.NET C#, that takes very frequently updated data and moves it from one database to another.
Currently using a middle man app, that pulls from one and inserts into another using SqlBulkCopy.
Is it better to have the source db server write a flat file and the middle man collect from that flat file?
I'm using a log file to document info while getting and processing info from a form
1. if not post back, open the log file, build page, send form. 2. get submit back from form, possible post back if errors, possible writes to log file, if no errors definitely writes to log file, and send confirm page. 3. postback from confirm page can either be going back to form or finalizing, and send acknowledge page, then done.
so I need to keep the logfile object between direct postbacks to the form, or posting back to orignal form page after gettting a postback from a confirm page.
this type of file setup works fine, it creates a single log file per day:
[Code]....
then successive writes are like this:
[Code]....
when the form submit comes back from the original send/submit, lf from above is 'nothing'
saw the post about 9 ways to maintain state, not sure at all which would be most appropriate to keep the lf object accessible between possible postbacks and submits to the form.
I am working on an SSIS package that needs to read a flat file and load the data into a SQL table. The problem is, the flat file name changes because it incorporates a date similar to this:
fileName_2010-04-12.txt
I found this site that shows how to set up a dynamic flat file connection:
[URL]
I understand the principle, but what I'm struggling with is using it in my data flow task. I need to configure the column mappings. In the PROPERTIES of the flat file connection, under expressions, I set up a ConnectionString with the following:
@[User::DataLoadDir] + @[User::priceFileName]
where [User::DataLoadDir] is the path where the file resides and [User::priceFileName] is a variable that should have the name of the specific file I want to read.
However, when I try to set the column mappings in the flat file connection, it's pulling the column headings for a different file in the same directory.
I have to get input from user in flat file format.then convert it to xml. Then, i have to store the entries into a datatable in the database. say 1,2,3 a,b,c be the user entries. it shall be stored into a table with 2 columns.how to do this?
View 1 RepliesI know this is probably a pretty easy thing to do and it is if I can upload the file and store it onto the hard drive of the server. What I need to do is read the text file into memory and then parse through it one line at a time. Anyone have any code that demonstrates that?
View 8 RepliesI am trying to export images from my SQL server to SSIS's Flat file destination but keep getting the same error: Error 0xc0208030: Data Flow Task: The data type for "input column "image_full" (105)" is DT_IMAGE, which is not supported. Use DT_TEXT or DT_NTEXT instead and convert the data from, or to, DT_IMAGE using the data conversion component. (SQL Server Import and Export Wizard) The data type is IMAGE in SQL DB. I would like to export the image and save the same in local folder. Is their any way to perform this task using SSIS flat file destination? What is the best and easiest way to do this? What format should I be exporting to?
View 1 RepliesI have a csv file with 5 columns. I need to read this file using c# and have to write the contents to the table in database which has 5 columns to it.
if anyone know the code or steps to do this.
I urgently need to find a way of reading data from a .CSV or .xls(Excel) file into an sql database so that I can use it in my asp.net 2.0 application.
View 4 RepliesI would like ask you for some ideas how can I write mechanism that will be automatically read files from folder. I don't know how to schedule the write task. I want to check for new file everyday and if the new file is in the folder read it.
View 2 RepliesI have the following data that basically I only need a few bits of information from:
Resource:X - Y;Z - Ã…;Type:(all) From Date:
07/12/2010 - To Date 07/12/2010 Sort
by:Time Include Referring
source/physician:No Footer:Default
Criteria:None
","Appointments","X,
Y","ZAssociates","Monday, July 12,
2010","Time","Patient Name","Patient
ID","Appt. Type","Ref. Source/
Physician","Phone","Type","DOB
","Z, X","Y","7/12/2010
12:00:00AM","Time","Patient
Name","Patient ID","Appt.
Type","Phone","Type","DOB "," 7:30
[snip]
The only things I need from this are:
Patients Name
Drs Name
Patients Phone Number
Appt Time
Appt Date
and the rest of the information I can discard. A customer uploads this as a .csv file (even though it really isn't as you can see) and I'd like to parse the needed information and post that to my SQL database and discard the rest. I think I can do this with a dataset but I've never built that before. The fields from the customer will always be the same and the fields I will need will always be the same. Also, the date time has to be in the format of yyyy/mm/dd:hhmm and the phone number always has to have 512 as a prefix. Here is the code I currently have for my site:
Imports System.IO
Imports System.Data
Imports System.Data.SqlClient
Partial Class _Default
Inherits System.Web.UI.Page
Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
End Sub
Protected Sub Submit1_Click(ByVal sender As Object, ByVal e As System.EventArgs)
Dim SaveLocation = "\xxxWEB3wwwrootWebfile1RemindersDoug_Ancildoug.csv"
If UploadFile(SaveLocation) Then
'the file was uploaded: now try saving it to the database
SaveToDatabase(SaveLocation)
End If
End Sub
Private Function UploadFile(ByVal SavePath As String) As Boolean
Dim fileWasUploaded As Boolean = False 'indicates whether or not the file was uploaded
'Checking if the file upload control contains a file
If Not File1.PostedFile Is Nothing And File1.PostedFile.ContentLength > 0 Then
Try
'checking if it was .txt file BEFORE UPLOADING IT!
'You used to upload it first...but the file could be a virus
If File1.FileName = ("doug.csv") = False Then
'The file is not the expected type...do not upload it
'just post the validation message
message.Text = "Sorry, thats not the correct file."
message2.Text = "Please locate and upload 'doug.csv'"
Else
'The file is a .txt file
'checking to see if the file exists already
'If it does exist Deleting the existing one so that the new one can be created
If IO.File.Exists(SavePath) Then
IO.File.Delete(SavePath)
End If
'Now upload the file (save it to your server)
File1.PostedFile.SaveAs(SavePath)
'After saving it check to see if it exists
If File.Exists(SavePath) Then
'Upload was sucessful
message.Text = "Thank you for your submission."
fileWasUploaded = True
Else
'the file was not saved
message.Text = "Unable to save the file."
End If
End If
Catch Exc As Exception
'We encountered a problem
message.Text = "Your file was not in the correct format. Please contact Customer Service at xxxx-xxxx-xxxx."
End Try
Else
'No file was selected for uploading
message.Text = "Please select a file to upload."
End If
Return fileWasUploaded
End Function
Private Sub SaveToDatabase(ByVal SavePath As String)
Try
Dim sqlQueryText As String = _
"BULK INSERT dialerresults " + _
"FROM '" & SavePath & "' " + _
"WITH ( FIELDTERMINATOR = ',' , ROWTERMINATOR = '
' )"
' and bulk import the data:
'If ConfigurationManager.ConnectionStrings("Dialerresults") IsNot Nothing Then
'Dim connection As String = ConfigurationManager.ConnectionStrings("Dialerresults").ConnectionString
Dim connection As String = "data source=10.2.1.40;initial catalog=IVRDialer;uid=xxxx;password=xxxxx;"
Using con As New SqlConnection(connection)
con.Open()
' execute the bulk import
Using cmd As New SqlCommand(sqlQueryText, con)
cmd.ExecuteNonQuery()
End Using
End Using
'Else
'message.Text="ConfigurationManager.ConnectionStrings('Dialerresults') is Nothing!"
'End If
Catch ex As Exception
message.Text = "Your file was not in the correct format. Please contact Customer Service at xxxxxxx."
End Try
End Sub
End Class
1) I need to make reading CSV file from the fileupload and then retrieve the data.
2) I need to load CSV and the 3 columns rise two functions.
Load CSV data and then stored in a database, but I need to save storage space vary according to 3 column.
CSV Example:
1 column - Cat, 2 column - Black, 3 column - 1
1 column - Dog, 2 column - Brown, 3 column - 2
1 column - Cat, 2 column - Black, 3 column - 1
1 column - Dog, 2 column - Brown, 3 column - 2
Now I need to load the CSV in C # and column 3 to distinguish what the operation is carried out.When the load line in column 3 contain the value 1, so will save the table 'Cats' when the value 2 and is saved in the table "Dogs".(Note: Data are separated ";")
I would like to read some data from a .XSLT file to populate my dropdown, the outline is as follows
file name StaffDirectory.xslt
<xsl:stylesheet version="1.0" extension-element-prefixes="msxsl" exclude-result-prefixes="msxsl js dl" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:js="urn:custom-javascript" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:dl="urn:datalist">
[Code].....
So id like to retrieve all values for ="IDAKRS0J" and populate a dropdown with this, is this possible ?
I have a 1.5 GB file into a table. I want to read It from server and create a file stream with it for save.
When I use DataAdapter.Fill(DataSet); code OutOfMemoryException occurs.
What should I do for read a hage file from data base?
I was trying to go through this tutorial:
[URL]
but when I tried to add the service reference to the silverlight app, the Add Service Reference window would show me that there was a service file, but the node wouldn't expand to show the rest of the files that contain the services and what not. I got this error instead:
"An error occurred during the processing of a configuration file required to service this request. review the specific error details below and modify your configuration file appropriately."
"Parser Error Message:There is no service behavior named 'AdventureWorks_WebServer.Service1Behavior'."
"Metadata contains a reference that cannot be resolved: 'http://localhost:55579/Service1.svc'.
The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.
If the service is defined in the current solution, try building the solution and adding the service reference again."
here's my web.config file:
[Code]....
I am writing code to insert values from an xml file. I check for a duplicate before inserting and delete if there is a match. Is there a more effecient way to prevent duplicates before insert a record? my code: private void LoadFData() { SqlConnection oConn = new SqlConnection(ConnectionString); SqlCommand oCommand = new SqlCommand(); oCommand.Connection = oConn; oCommand.CommandType = CommandType.Text; oConn.Open(); // Load committee master files string[] strFiles = System.IO.Directory.GetFiles("fff", "ffff.dta", System.IO.SearchOption.AllDirectories); for (int i=0; i
View 2 RepliesI am using VS 2005.I want to include Chart reading values from an XML file.
View 3 RepliesI am using reading data from excel file and inserting into sql server using following article, it seems to be working correctly except when I do page refresh normally, it again execute the the btnUpload_click. I want that once the record imported and if user refresh the page again it should ask for enter file name again. let me know how do I resolve this? [URL]
View 8 Repliesi like to create a web application,in that i need to get the resumes from the user,then i need to publish that resume to the manager in date wise..
View 5 RepliesIn web service reading *.txt file .this file contains some special characters like "" .so its shows error while reading .
string fullPath = Server.MapPath("sample.txt");
string reconciliationData = string.Empty;
StreamReader ObjReconciliationDataStream = new StreamReader(fullPath, Encoding.UTF8);
string strReconData = ObjReconciliationDataStream.ReadToEnd();
ObjReconciliationDataStream.Close();
ObjReconciliationDataStream = null;
return strReconData;
above code i am using .
I would like to find a way to read a txt file (in my case delimited by ¤). I need to change the order of the columns and also add and remove some columns. My output file should be a txt file delmited by ;.
I tried Jet.OLEDB to read the file and put it into a datatable and then used a stringBuilder and streamwriter to get an output file in .txt format. However. This does not me since I´, reading the txt file from start to end and my output will be the same. It does not seem like I can have a custom sql statement when reading the file. The only query that works is
Dim da As New System.Data.OleDb.OleDbDataAdapter("Select * from 1.txt", TextConn)
How can I possible modify a txt file?
This is my code for now:
[Code]....
I can directly use to read and display Excel or CSV files using asp web control(GridView, sqldatasource etc). I do NOT want to upload the file first to an sql database before reading them, I just want to read the files directly. I am using asp.net 2.0 and c#.
View 3 Replies
