Importing 20 Million Records

Ben Butler 760.f563 at gmail.com
Fri Jun 25 21:10:08 UTC 2021


Hi,

I would write a loader routine that was responsible for reading in the 
data and putting it into the database / whatever you want to do with 
it.  For that I would call from Omnis to a python script. Python is able 
to read files line at a time without having to load it all into ram.

https://www.kite.com/python/answers/how-to-read-a-large-file-without-loading-it-into-memory-in-python

This mean you can load in any amount of source records / size of import 
file.

My 2c, python plus Omnis is a very powerful combination and easy to 
integrate and exchange messages in both directions.

Ben

On 25/06/2021 19:51, Andrew Stolarz wrote:
> Das,
>
> Sometimes when I need to open very large text,csv,sql files and do
> something manually with them (10+gb with 20+ million records).
>
> My go to software is Hex Fiend. https://hexfiend.com/
>
> Its licensed under BSD too.
>
> It's extremely fast and handles a lot of data very well so you can break up
> your file into smaller chunks etc.
>
>
>
> Andrew
>
>
> On Fri, Jun 25, 2021 at 12:38 PM Das Goravani <goravanis at gmail.com> wrote:
>
>> Dear Friends,
>>
>> In OMnis I am familiar with importing, but never 20 million records.
>>
>> If you have a huge file with 20 million records in it, how do you import
>> that?
>>
>> Omnis works with a list.. you import to a list.. Omnis can handle between
>> 1 and 2 million rows.. from my experience
>>
>> Do you cut up the file.. do you get some utility for that?
>>
>> Because you can’t open such a file in any program like Word or Excel or
>> Numbers or Pages
>>
>> So you’d need some utility meant for cutting up large files
>>
>> Or is there a way to tell Omnis to only import part, then next part, then
>> next.. I’ve never seen that.
>>
>> Thanks
>>
>> Das Goravani
>> _____________________________________________________________
>> Manage your list subscriptions at http://lists.omnis-dev.com
>> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com
>>
> _____________________________________________________________
> Manage your list subscriptions at http://lists.omnis-dev.com
> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com


More information about the omnisdev-en mailing list