load balancing experience
Marc De Roover
omnis1 at arcict.com
Wed Dec 18 04:39:49 EST 2013
Vik,
> Doug: if you don't mind sharing with us/community your
> findings/experiences too please. It could help ease newcomers into the
> 6.x experience. Plus having operating stats is more beneficial than
> theoretical info/assumptions.
One more very good reason to convince your boss to let you attend
Euromnis. :-)
I had the pleasure to attend some great sessions on these topics and
observe elaborate visionary discussions over a beer ( or two )
Marc
On 18 Dec 2013, at 9:33, Vik Shah wrote:
> Lars & Doug,
>
> Curious...
>
> We are solely into webclient solution, still with studio 4.x and
> eyeing 6.x.
>
> Lars: if you could elaborate on what you are attempting to achieve,
> thin/ultra-thin/mobile-JS solution and the level of complexity of the
> app (a highly subjective question, but I had to ask)?
>
> Doug: if you don't mind sharing with us/community your
> findings/experiences too please. It could help ease newcomers into the
> 6.x experience. Plus having operating stats is more beneficial than
> theoretical info/assumptions.
>
> I'd love to know more.
>
> Regards,
>
> Vik
>
>> On 18 Dec 2013, at 0:37, "Doug Easterbrook" <doug at artsman.com> wrote:
>>
>> hi Andy, thanks for the vote of confidence, and yes we've done it.
>>
>> for Lars:
>>
>> With some creativeness in our approach we've built our web services
>> so that they are horizontally and vertically scaleable -- and we are
>> now at the point where we believe we can really handle about 600
>> simultaneous transactions per second off a quad core i7 machine.
>>
>> 600 simultaneous transactions is 600 people hitting enter at the same
>> time and getting a response back, be it search for tickets, add
>> tickets to cart, send emails, what have you.
>>
>> Note: we only use thin client. Our web pages contains some
>> javascript -- but we do not do Studio javascript client for a number
>> of reasons ... primarily we want to handle a request from any device
>> in rest style API, deal with it and the disconnect the session ---
>> until the next reuqest that is.
>>
>>
>>
>> some of the design keys to the success were:
>>
>> load balancing outside omnis
>> -- using apache 2.4 and the load balancer module to distribute
>> transactions amongst application servers.
>> -- We have also used nginx for even more throughput
>>
>>
>> making heavy use of the worker concept that is part of the Studio 6
>> dams.
>> -- we thread database i/o to load whatever we can in the background
>> on a different thread for things like constant tables, event lists,
>> etc
>> -- we made workers to handle things that block time like an email
>> worker that does SMTP sends. This means the main thread never sends
>> emails, it just puts them into a table and a 'worker' gathers those
>> and sends.
>> -- We also use background workers for any kind of clean up processes,
>> synchronization process or what have you.
>>
>> The goal is ZERO latency for the web services - do the job,
>>
>>
>> Take advantage of technology and caching;
>> -- we are moving away from polling for record changes. i.e. we do
>> not want to *Read* a record to see if it changed. We created an
>> external that does postgres listen notify . Meaning, there is a
>> trigger in the database that notices when a record changes. if it
>> does, it fires off a 'hey this is changed' notification to whomever
>> cares. Think of it as a big model-view-controller. All part of
>> postgres (needs an external for omnis).
>>
>> benefit for us -- we were reading some data every 10 seconds or every
>> time a request came in. but since some data is more (or less)
>> constant, if we figure that the listen notify process could save a
>> hundred thousand database reads in any one given day. no need to
>> read = faster. Faster=more clients served.
>>
>> we also look to see if a page has changed and if we can, we cache
>> entire pages and send them back. example a home page, image, or list
>> of events does not change for each request -- so we cache it and
>> simply send back the cached page without any database hits. This
>> works for ultra-thin -- maybe not so much for javascript client.
>>
>>
>>
>> horizontal scaleability:
>>
>> We wrote an apache module for load balancing the omnis fatter client
>> interface. However, we only ever used it for thin client -- but it
>> may work for javascript client as I believe that they all use the
>> same interface from apache into omnis. That does load distribution
>> as well. If balancer-manager in apache can't do the job, then we use
>> this.
>>
>> the goal: when a client is not busy, they use one machine. When
>> they get really busy, then can scale up by adding any number of
>> machines to the server pool and the web server does load management
>> and distribution -- if the machine is available or set up. It just
>> handles it without taking down or reconfiguration.
>>
>>
>>
>> Threading:
>> this is the other part that we do a lot of. we take advantage of all
>> the cores on any given machine - multi threading or multi processing
>> if you can.
>>
>> When we did an omnis only solution on a 4 core machine, we were
>> getting as many as 20 real requests per second (4 copies of omnis
>> processing requests in 200 milliseconds each, with internal caching
>> and all sorts of tricks). Changing how things were coded a
>> bit, caching, threading, worker concept, avoiding latency in all
>> forms, apache load balancing, postgres listen-notify and avoid
>> database polling, using some other technologies to assist, and a
>> clear focus in our design on avoiding bottlenecks and connections --
>> got us to some requests happening in 5 milliseconds or less. Some
>> requests still take a while (the first time) .. but we cache every
>> little bit we can, parts of searches, parts of pages, parts of
>> queries -- you name it...
>>
>>
>> we are working with some venues that want to sell out concerts in
>> minutes -- and we are fairly sure that having 3 or 4 mac mini's
>> handle 4*600=2400 real connections a second, all load balanced will
>> let our clients do this at minimal cost.
>>
>>
>>
>> Warnings:
>> -- apache works pretty good.. but it will start to wither a bit under
>> high load and start to drop some connections. Best results have
>> been nginx so far as the actual front end server.
>> -- use a mixture of tools like apache benchmark, jmeter, locust, or
>> some home grown tools to hammer the hell out of your server. you'll
>> find issues faster than the clients can that way.
>> -- don't trust any one benchmarking or load creating tool.. Each
>> will test some aspect of your servers -- but may not find all the
>> bottlenecks.
>> -- any one technology choice will not solve all problems. While we
>> like postgres listen-notify and it will save a lot of i/o -- it can't
>> be the only tool in our arsenal.
>>
>>
>>
>>
>> just my thoughts. happy to explain more off list for anybody who
>> is interested.
>>
>>
>>
>> Doug Easterbrook
>> Arts Management Systems Ltd.
>> mailto:doug at artsman.com
>> http://www.artsman.com
>> Phone (403) 536-1205 Fax (403) 536-1210
>>
>>> On Dec 16, 2013, at 10:00 AM,
>>> omnisdev-en-request at lists.omnis-dev.com wrote:
>>>
>>> Lars
>>>
>>> I would speak in depth with Doug Easterbrook and his team - who have
>>> a lot of experience at high workloads and load balancing of various
>>> kinds.....
>>>
>>> Andy
>>
>> _____________________________________________________________
>> Manage your list subscriptions at http://lists.omnis-dev.com
> _____________________________________________________________
> Manage your list subscriptions at http://lists.omnis-dev.com
More information about the omnisdev-en
mailing list