Importing 20 Million Records

[Alex Clay]

Hi Das,

Omnis is a great tool, but not the right one for this job.

IIRC, you use sqlite. Try searching for a tool that bulk loads data into a sqlite datafile. Or use the native import command: https://www.sqlitetutorial.net/sqlite-import-csv/ <https://www.sqlitetutorial.net/sqlite-import-csv/>

If you want to use Omnis, use a command to break up the file into chunks, then import each one. Skip the GUI tools like Numbers and Pages as you don't need to see this data.

The split command would do the trick nicely: https://www.linode.com/docs/guides/divide-files-with-split/ <https://www.linode.com/docs/guides/divide-files-with-split/>

Alex

> On Jun 25, 2021, at 12:38, Das Goravani <goravanis at gmail.com> wrote:
> 
> 
> Dear Friends,
> 
> In OMnis I am familiar with importing, but never 20 million records.
> 
> If you have a huge file with 20 million records in it, how do you import that?
> 
> Omnis works with a list.. you import to a list.. Omnis can handle between 1 and 2 million rows.. from my experience
> 
> Do you cut up the file.. do you get some utility for that?
> 
> Because you can’t open such a file in any program like Word or Excel or Numbers or Pages
> 
> So you’d need some utility meant for cutting up large files
> 
> Or is there a way to tell Omnis to only import part, then next part, then next.. I’ve never seen that. 
> 
> Thanks
> 
> Das Goravani
> _____________________________________________________________
> Manage your list subscriptions at http://lists.omnis-dev.com
> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com