load balancing experience

[Phil Potter (ou)]

I'm just going to second this, Euromnis this year was great, and 
attending both Doug's and David's sessions on what Doug was talking 
about, was an eye opening experience.
Phil.

On 18/12/2013 09:39, Marc De Roover wrote:
> Vik,
>
>> Doug: if you don't mind sharing with us/community your 
>> findings/experiences too please. It could help ease newcomers into 
>> the 6.x experience. Plus having operating stats is more beneficial 
>> than theoretical info/assumptions.
>
> One more very good reason to convince your boss to let you attend 
> Euromnis. :-)
>
> I had the pleasure to attend some great sessions on these topics and 
> observe elaborate visionary discussions over a beer ( or two )
>
>
>
> Marc
>
>
> On 18 Dec 2013, at 9:33, Vik Shah wrote:
>
>> Lars & Doug,
>>
>> Curious...
>>
>> We are solely into webclient solution, still with studio 4.x and 
>> eyeing 6.x.
>>
>> Lars: if you could elaborate on what you are attempting to achieve, 
>> thin/ultra-thin/mobile-JS solution and the level of complexity of the 
>> app (a highly subjective question, but I had to ask)?
>>
>> Doug: if you don't mind sharing with us/community your 
>> findings/experiences too please. It could help ease newcomers into 
>> the 6.x experience. Plus having operating stats is more beneficial 
>> than theoretical info/assumptions.
>>
>> I'd love to know more.
>>
>> Regards,
>>
>> Vik
>>
>>> On 18 Dec 2013, at 0:37, "Doug Easterbrook" <doug at artsman.com> wrote:
>>>
>>> hi Andy, thanks for the vote of confidence, and yes we've done it.
>>>
>>> for Lars:
>>>
>>> With some creativeness in our approach we've built our web services 
>>> so that they are horizontally and vertically scaleable -- and we are 
>>> now at the point where we believe we can really handle about 600 
>>> simultaneous transactions per second off a quad core i7 machine.
>>>
>>> 600 simultaneous transactions is 600 people hitting enter at the 
>>> same time and getting a response back, be it search for tickets, add 
>>> tickets to cart, send emails, what have you.
>>>
>>> Note: we only use thin client.  Our web pages contains some 
>>> javascript -- but we do not do Studio javascript client for a number 
>>> of reasons ...  primarily we want to handle a request from any 
>>> device in rest style API, deal with it and the disconnect the 
>>> session --- until the next reuqest that is.
>>>
>>>
>>>
>>> some of the design keys to the success were:
>>>
>>> load balancing outside omnis
>>> -- using apache 2.4 and the load balancer module to distribute 
>>> transactions amongst application servers.
>>> -- We have also used nginx for even more throughput
>>>
>>>
>>> making heavy use of the worker concept that is part of the Studio 6 
>>> dams.
>>> -- we thread database i/o to load whatever we can in the background 
>>> on a different thread for things like constant tables, event lists, etc
>>> -- we made workers to handle things that block time like an email 
>>> worker that does SMTP sends.  This means the main thread never sends 
>>> emails, it just puts them into a table and a 'worker' gathers those 
>>> and sends.
>>> -- We also use background workers for any kind of clean up 
>>> processes, synchronization process or what have you.
>>>
>>> The goal is ZERO latency for the web services - do the job,
>>>
>>>
>>> Take advantage of technology and caching;
>>> -- we are moving away from polling for record changes.  i.e. we do 
>>> not want to *Read* a record to see if it changed.  We created an 
>>> external that does postgres listen notify . Meaning, there is a 
>>> trigger in the database that notices when a record changes.   if it 
>>> does, it fires off a 'hey this is changed' notification to whomever 
>>> cares.   Think of it as a big model-view-controller.   All part of 
>>> postgres (needs an external for omnis).
>>>
>>> benefit for us -- we were reading some data every 10 seconds or 
>>> every time a request came in.  but since some data is more (or less) 
>>> constant, if we figure that the listen notify process could save a 
>>> hundred thousand database reads in any one given day.    no need to 
>>> read = faster.  Faster=more clients served.
>>>
>>> we also look to see if a page has changed and if we can, we cache 
>>> entire pages and send them back.  example a home page, image, or 
>>> list of events does not change for each request -- so we cache it 
>>> and simply send back the cached page without any database hits.  
>>> This works for ultra-thin -- maybe not so much for javascript client.
>>>
>>>
>>>
>>> horizontal scaleability:
>>>
>>> We wrote an apache module for load balancing the omnis fatter client 
>>> interface.   However, we only ever used it for thin client -- but it 
>>> may work for javascript client as I believe that they all use the 
>>> same interface from apache into omnis. That does load distribution 
>>> as well.  If balancer-manager in apache can't do the job, then we 
>>> use this.
>>>
>>> the goal:   when a client is not busy, they use one machine. When 
>>> they get really busy, then can scale up by adding any number of 
>>> machines to the server pool and the web server does load management 
>>> and distribution -- if the machine is available or set up.   It just 
>>> handles it without taking down or reconfiguration.
>>>
>>>
>>>
>>> Threading:
>>> this is the other part that we do a lot of.  we take advantage of 
>>> all the cores on any given machine - multi threading or multi 
>>> processing if you can.
>>>
>>> When we did an omnis only solution on a 4 core machine, we were 
>>> getting as many as 20 real requests per second (4 copies of omnis 
>>> processing requests in 200 milliseconds each, with internal caching 
>>> and all sorts of tricks).        Changing how things were coded a 
>>> bit, caching, threading, worker concept, avoiding latency in all 
>>> forms, apache load balancing, postgres listen-notify and avoid 
>>> database polling, using some other technologies to assist, and a 
>>> clear focus in our design on avoiding bottlenecks and connections -- 
>>> got us to some requests happening in 5 milliseconds or less.   Some 
>>> requests still take a while (the first time) .. but we cache every 
>>> little bit we can, parts of searches, parts of pages, parts of 
>>> queries -- you name it...
>>>
>>>
>>> we are working with some venues that want to sell out concerts in 
>>> minutes -- and we are fairly sure that having 3 or 4 mac mini's 
>>> handle 4*600=2400 real connections a second, all load balanced will 
>>> let our clients do this at minimal cost.
>>>
>>>
>>>
>>> Warnings:
>>> -- apache works pretty good.. but it will start to wither a bit 
>>> under high load and start to drop some connections.   Best results 
>>> have been nginx so far as the actual front end server.
>>> -- use a mixture of tools like apache benchmark, jmeter, locust, or 
>>> some home grown tools to hammer the hell out of your server.  you'll 
>>> find issues faster than the clients can that way.
>>> -- don't trust any one benchmarking or load creating tool.. Each 
>>> will test some aspect of your servers -- but may not find all the 
>>> bottlenecks.
>>> -- any one technology choice will not solve all problems. While we 
>>> like postgres listen-notify and it will save a lot of i/o -- it 
>>> can't be the only tool in our arsenal.
>>>
>>>
>>>
>>>
>>> just my thoughts.      happy to explain more off list for anybody 
>>> who is interested.
>>>
>>>
>>>
>>> Doug Easterbrook
>>> Arts Management Systems Ltd.
>>> mailto:doug at artsman.com
>>> http://www.artsman.com
>>> Phone (403) 536-1205    Fax (403) 536-1210
>>>
>>>> On Dec 16, 2013, at 10:00 AM, 
>>>> omnisdev-en-request at lists.omnis-dev.com wrote:
>>>>
>>>> Lars
>>>>
>>>> I would speak in depth with Doug Easterbrook and his team - who 
>>>> have a lot of experience at high workloads and load balancing of 
>>>> various kinds.....
>>>>
>>>> Andy
>>>
>>> _____________________________________________________________
>>> Manage your list subscriptions at http://lists.omnis-dev.com
>> _____________________________________________________________
>> Manage your list subscriptions at http://lists.omnis-dev.com
> _____________________________________________________________
> Manage your list subscriptions at http://lists.omnis-dev.com
> .
>
>
> -- 
> Phil Potter
> Based in Chester in the UK.