General Success with things

[Mike Matthews - Omnis]

Good news.

Mike


[Lineal Software Solutions]         <http://www.lineal.co.uk>

[Twitter]<https://twitter.com/linealsoftware> [Facebook] <https://www.facebook.com/linealsoftware?fref=ts&ref=br_tf>  [Google +] <https://plus.google.com/112982815000763473226/about>  [LinkedIn] <https://www.linkedin.com/company/1859834?trk=tyah&trkInfo=clickedVertical%3Acompany%2Cidx%3A1-1-1%2CtarId%3A1437729256906%2Ctas%3Alineal%20s>

[https://s3.amazonaws.com/htmlsig-assets/spacer.gif]

Mike Matthews
omnis at lineal.co.uk<mailto:omnis at lineal.co.uk>

Lineal Software Solutions
01271 375999
Commercial House, The StrandBarnstaple, Devon, EX31 1EU
www.lineal.co.uk<http://www.lineal.co.uk>


[http://www.lineal.co.uk/stuff/EmailSig/MicrosoftPartner.png]<http://www.lineal.co.uk/windows-support/> [http://www.lineal.co.uk/stuff/EmailSig/Office_365.png] <http://www.lineal.co.uk/microsoft-office-365/>  [http://www.lineal.co.uk/stuff/EmailSig/Cisco.png] <http://www.lineal.co.uk/wifi-networks/>  [http://www.lineal.co.uk/stuff/EmailSig/Meraki.png] <http://www.lineal.co.uk/wifi-networks/>  [http://www.lineal.co.uk/stuff/EmailSig/Dell.png] <http://www.lineal.co.uk/it-systems/>  [http://www.lineal.co.uk/stuff/EmailSig/Gamma.png] <http://www.lineal.co.uk/gamma-horizon-voip/>  [http://www.lineal.co.uk/stuff/EmailSig/Kerio.png] <http://www.lineal.co.uk/kerio/>  [http://www.lineal.co.uk/stuff/EmailSig/Apple.png] <http://www.lineal.co.uk/apple-upgrades-repairs/>  [http://www.lineal.co.uk/stuff/EmailSig/Eset.png] <http://www.lineal.co.uk/eset-security-software/>  [http://www.lineal.co.uk/stuff/EmailSig/mitel.png] <http://www.lineal.co.uk/shoretel/>




On 27 Aug 2020, at 18:35, Das Goravani <goravanis at gmail.com<mailto:goravanis at gmail.com>> wrote:


I’m done with my big data imports and massages now.

Had import of accent characters problem.. found resolution in saving the original text file as UTF-8 WITH BOM, it was that WITH BOM part that made the difference.

Could load 1.7 million records to list for import, could not load the records to list after import due to limitations.

All 1.7 million records imported in about 3 hours.

Importing, Inserting, is faster than updating in a loop with a fetch.

I kept the speed up by logging off and logging back on every 5,000 records. They got noticeably slower after about 2,000 inserts or updates.. and would become much faster again after these periodic log offs. Rudolph and Doug may have suggested other SQL options rather than logging off.. well I accomplished it that way.  Since I’m done I won’t pursue what is better at this point.

I can load the data in chunks.. just can’t load all 1.7 million records to a list.

Had a win where I wrote a complex piece of code and it worked the first time testing it. Yay. It finds nearby cities to the one the user selects.. I have to borrow data from the nearby cities.. I was pleased that longitude and latitude work like x, y grid so perfectly.. I do

If less than or equal to longitude plus 1 is greater than or equal to longitude minus 1, this defines a range that the other city can be in relative to the one they chose.. 1 degree about equals 60 miles I think.. do the same with latitude.. when you do both you effectively have defined a little box of area that the other cities can be in.. then do larger if you don’t find any.. I found this method exhilarating to write because it deals with real data, real cities, real x,y searching.. and my SQL where clauses worked the first time.. I’m getting the hang of writing where clauses.

So I’ve got the whole world in my program now, I’m really jazzed about that.

My “running in the CRB” is going well.. no issues except it IS a little slower to do because you have that extra step of putting the data into the CRB.. depending on how you do it.. I do it the slow but sure way.. with a repeat loop on a field by field basis.. redefining the SQL list or row is another way.. I would do that but I found my biggest table corrupted upon the redefinition.. the data would shift columns.. only on my very largest table..

So spinning the data into the CRB FIELDs and spinning it back up again into the row to do insert, update, etc.. does take time and slows down my app a little bit.  When I was doing my importing I would work with the row and lists without the CRB and it was faster of course.

My generic methods slow things down a bit too as I always define the row or list with each time I call it rather than re using an already defined row or list.. so it’s ALL STEPS EVERY TIME which is slower than doing only those steps you have to.. re using defined lists, faster..

I love how you can simply make a method for SQL errors.. that is always calls this method every time there is a SQL error.. I have it in my master table class which is perfect.. it’s called for any row any SQL error.. I put a Sound Bell there so I know anytime I have an error.. beep.. love it.. excellent.

I’ve never shipped a version to users with SQL yet.. nervous about it but going to do it soon, it’s time.

My app server runs like a charm, just sits there and serves.. do NOT have a heavy load at all, but what little traffic I get is served rightly.. it’s stable.. just sits there day after day.. was easier to setup than I thought it would be..


_____________________________________________________________
Manage your list subscriptions at https://linkprotect.cudasvc.com/url?a=http%3a%2f%2flists.omnis-dev.com&c=E,1,G1We2qbJw7lG4K9SIUy-0yHOoYEgKXmxsOKJesAbYq0nh6M3Nd7pbt7stjmsJaSCy7eooFBeP4pAS29XrpuEqkozqfje1Gi58Oaeu2kCVGMaaQU,&typo=1
Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com