load balancing experience

Marc De Roover omnis1 at arcict.com
Wed Dec 18 04:39:49 EST 2013


Vik,

> Doug: if you don't mind sharing with us/community your 
> findings/experiences too please. It could help ease newcomers into the 
> 6.x experience. Plus having operating stats is more beneficial than 
> theoretical info/assumptions.

One more very good reason to convince your boss to let you attend 
Euromnis. :-)

I had the pleasure to attend some great sessions on these topics and 
observe elaborate visionary discussions over a beer ( or two )



Marc


On 18 Dec 2013, at 9:33, Vik Shah wrote:

> Lars & Doug,
>
> Curious...
>
> We are solely into webclient solution, still with studio 4.x and 
> eyeing 6.x.
>
> Lars: if you could elaborate on what you are attempting to achieve, 
> thin/ultra-thin/mobile-JS solution and the level of complexity of the 
> app (a highly subjective question, but I had to ask)?
>
> Doug: if you don't mind sharing with us/community your 
> findings/experiences too please. It could help ease newcomers into the 
> 6.x experience. Plus having operating stats is more beneficial than 
> theoretical info/assumptions.
>
> I'd love to know more.
>
> Regards,
>
> Vik
>
>> On 18 Dec 2013, at 0:37, "Doug Easterbrook" <doug at artsman.com> wrote:
>>
>> hi Andy, thanks for the vote of confidence, and yes we've done it.
>>
>> for Lars:
>>
>> With some creativeness in our approach we've built our web services 
>> so that they are horizontally and vertically scaleable -- and we are 
>> now at the point where we believe we can really handle about 600 
>> simultaneous transactions per second off a quad core i7 machine.
>>
>> 600 simultaneous transactions is 600 people hitting enter at the same 
>> time and getting a response back, be it search for tickets, add 
>> tickets to cart, send emails, what have you.
>>
>> Note: we only use thin client.  Our web pages contains some 
>> javascript -- but we do not do Studio javascript client for a number 
>> of reasons ...  primarily we want to handle a request from any device 
>> in rest style API, deal with it and the disconnect the session --- 
>> until the next reuqest that is.
>>
>>
>>
>> some of the design keys to the success were:
>>
>> load balancing outside omnis
>> -- using apache 2.4 and the load balancer module to distribute 
>> transactions amongst application servers.
>> -- We have also used nginx for even more throughput
>>
>>
>> making heavy use of the worker concept that is part of the Studio 6 
>> dams.
>> -- we thread database i/o to load whatever we can in the background 
>> on a different thread for things like constant tables, event lists, 
>> etc
>> -- we made workers to handle things that block time like an email 
>> worker that does SMTP sends.  This means the main thread never sends 
>> emails, it just puts them into a table and a 'worker' gathers those 
>> and sends.
>> -- We also use background workers for any kind of clean up processes, 
>> synchronization process or what have you.
>>
>> The goal is ZERO latency for the web services - do the job,
>>
>>
>> Take advantage of technology and caching;
>> -- we are moving away from polling for record changes.  i.e. we do 
>> not want to *Read* a record to see if it changed.  We created an 
>> external that does postgres listen notify .   Meaning, there is a 
>> trigger in the database that notices when a record changes.   if it 
>> does, it fires off a 'hey this is changed' notification to whomever 
>> cares.   Think of it as a big model-view-controller.   All part of 
>> postgres (needs an external for omnis).
>>
>> benefit for us -- we were reading some data every 10 seconds or every 
>> time a request came in.  but since some data is more (or less) 
>> constant, if we figure that the listen notify process could save a 
>> hundred thousand database reads in any one given day.    no need to 
>> read = faster.  Faster=more clients served.
>>
>> we also look to see if a page has changed and if we can, we cache 
>> entire pages and send them back.  example a home page, image, or list 
>> of events does not change for each request -- so we cache it and 
>> simply send back the cached page without any database hits.  This 
>> works for ultra-thin -- maybe not so much for javascript client.
>>
>>
>>
>> horizontal scaleability:
>>
>> We wrote an apache module for load balancing the omnis fatter client 
>> interface.   However, we only ever used it for thin client -- but it 
>> may work for javascript client as I believe that they all use the 
>> same interface from apache into omnis.   That does load distribution 
>> as well.  If balancer-manager in apache can't do the job, then we use 
>> this.
>>
>> the goal:   when a client is not busy, they use one machine.  When 
>> they get really busy, then can scale up by adding any number of 
>> machines to the server pool and the web server does load management 
>> and distribution -- if the machine is available or set up.   It just 
>> handles it without taking down or reconfiguration.
>>
>>
>>
>> Threading:
>> this is the other part that we do a lot of.  we take advantage of all 
>> the cores on any given machine - multi threading or multi processing 
>> if you can.
>>
>> When we did an omnis only solution on a 4 core machine, we were 
>> getting as many as 20 real requests per second (4 copies of omnis 
>> processing requests in 200 milliseconds each, with internal caching 
>> and all sorts of tricks).        Changing how things were coded a 
>> bit, caching, threading, worker concept, avoiding latency in all 
>> forms, apache load balancing, postgres listen-notify and avoid 
>> database polling, using some other technologies to assist, and a 
>> clear focus in our design on avoiding bottlenecks and connections -- 
>> got us to some requests happening in 5 milliseconds or less.   Some 
>> requests still take a while (the first time) .. but we cache every 
>> little bit we can, parts of searches, parts of pages, parts of 
>> queries -- you name it...
>>
>>
>> we are working with some venues that want to sell out concerts in 
>> minutes -- and we are fairly sure that having 3 or 4 mac mini's 
>> handle 4*600=2400 real connections a second, all load balanced will 
>> let our clients do this at minimal cost.
>>
>>
>>
>> Warnings:
>> -- apache works pretty good.. but it will start to wither a bit under 
>> high load and start to drop some connections.   Best results have 
>> been nginx so far as the actual front end server.
>> -- use a mixture of tools like apache benchmark, jmeter, locust, or 
>> some home grown tools to hammer the hell out of your server.  you'll 
>> find issues faster than the clients can that way.
>> -- don't trust any one benchmarking or load creating tool..    Each 
>> will test some aspect of your servers -- but may not find all the 
>> bottlenecks.
>> -- any one technology choice will not solve all problems.    While we 
>> like postgres listen-notify and it will save a lot of i/o -- it can't 
>> be the only tool in our arsenal.
>>
>>
>>
>>
>> just my thoughts.      happy to explain more off list for anybody who 
>> is interested.
>>
>>
>>
>> Doug Easterbrook
>> Arts Management Systems Ltd.
>> mailto:doug at artsman.com
>> http://www.artsman.com
>> Phone (403) 536-1205    Fax (403) 536-1210
>>
>>> On Dec 16, 2013, at 10:00 AM, 
>>> omnisdev-en-request at lists.omnis-dev.com wrote:
>>>
>>> Lars
>>>
>>> I would speak in depth with Doug Easterbrook and his team - who have 
>>> a lot of experience at high workloads and load balancing of various 
>>> kinds.....
>>>
>>> Andy
>>
>> _____________________________________________________________
>> Manage your list subscriptions at http://lists.omnis-dev.com
> _____________________________________________________________
> Manage your list subscriptions at http://lists.omnis-dev.com



More information about the omnisdev-en mailing list