Importing 20 Million Records

[Dan Ridinger]

Hello Das,

If you want to do it in Omnis code you could do it with AppleScript.  For example:

Begin text block

Text: tell application “Terminal" (Carriage return)

Text:delay 6 (Carriage return)

Text: 	do script “sqlite3" (Carriage return)

Text: 	do script ".open [database_name]" in window 1 (Carriage return)
Text: 	do script ".mode csv" in window 1 (Carriage return)
Text: 	do script ".import [import_file_name] members" in window 1 (Carriage return)
Text: 	do script "select count() from members;" in window 1 (Carriage return)
Text: 	end tell (Carriage return)
End text block

Get text block iScript

Do $root.$runapplescript(iScript,iErrCode,iErrText)

Quit method iErrCode


This will load the data from the csv into the SqLite database. It will probably take some tweaking to get it to work.

Dan
> On Jun 25, 2021, at 9:38 AM, Das Goravani <goravanis at gmail.com> wrote:
> 
> 
> Dear Friends,
> 
> In OMnis I am familiar with importing, but never 20 million records.
> 
> If you have a huge file with 20 million records in it, how do you import that?
> 
> Omnis works with a list.. you import to a list.. Omnis can handle between 1 and 2 million rows.. from my experience
> 
> Do you cut up the file.. do you get some utility for that?
> 
> Because you can’t open such a file in any program like Word or Excel or Numbers or Pages
> 
> So you’d need some utility meant for cutting up large files
> 
> Or is there a way to tell Omnis to only import part, then next part, then next.. I’ve never seen that. 
> 
> Thanks
> 
> Das Goravani
> _____________________________________________________________
> Manage your list subscriptions at http://lists.omnis-dev.com
> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com