NO and Studio Customer Complaints - Web based apps and network Issues -- or designing to make web apps look faster than they really are

Doug Easterbrook doug at artsman.com
Mon Mar 6 14:42:51 UTC 2023


hi Steve:

you say:

> Developers are really
> losing control over the performance of their apps due to the vagaries of
> the bandwidth at any point in time.

and

> or do you just not worry about it and just take the calls from
> irate customers?

and I answer — yes to both, and no, you can mitigate response but you have to design your apps differently.

so, I have examples of:
— an analogy for you
— how we take time to educate our customers and find flaws in their equipment (its first
— and specific technical things we do in our application to improve performance, to mitigate what throttles are put on web traffic that we can’t control.






I always look for analogies, so lats take a British country road, hedgerows and all.

on a clear day with no traffic, you can go as fast as you want down them.   it is fun with unrestricted performance.    Then sunday comes and everybody is out driving.   life is not fast.

so who do we blame?
- the people who constructed the road many years ago
- the improvements in manufacture of the automobile that allows them to go faster than a horse
- the increase in affluence so that there are more cars on the road,
- the weatherman that said on the newscast the previous evening ’tomorrow will be sunny and a great day for a drive’
- the arbourists and farmers that didn’t chop down the hedgerow to allow better sight and anticipation of other drivers (out of pure self interest to keep the cows in the field).

the answer is likely all of them contributed in some way to the slow speed that is being experienced.   and we as humans, who wanted to have that beautiful super fast experience to ourselves, naturally want to blame somebody else. because we got on the road at the same time as a humongous amount of traffic.   The refrain is ‘of course, this is not my fault,   its the other persons fault for being on my private road way’


the one thing you CANNOT guarantee is unrestricted access to the roadway (or internet)





Slow Response = investigation and  education

With web traffic, when people get slow response, we have a cast of characters that we like to blame and have had to spend time educating our customers thats its not our issue, but theirs.   I”ll give examples.

first to blame:  their ISP.     as in ‘you are the only customer complaining, so lets do some side by side testing

look for other bad actors that they don’t tell you about:     Two specific examples here:

—  customers say its fine when they are the only person usin the app, but when people start getting into the office, it slows down.    We know it ca’t be our servers, because we have some hundreds of CPU’s, spread over multiple machines, gigabytes of ram and a lot of M.2 high performance ssd’s.    What do we typically find out?    everybody in the office is on some slow wifi router and each new employee basically divides their internal bandwidth by two.   so 4 employees is 1/4 the speed of 1 employee.

— they have no QOS (Quality of service) set up on their routers.    This really happened to us.  A large hockey venue said our app was slow.  We said no way — but asked when.  They said it was fine in the morning, but slow when customers were in watching hockey.   Instant clue there.  so we asked them to try the application tethered to their phone and they said it was miraculously much faster.

we discovered that they allowed some 6,000 hockey fans to use their corprorate internet connection as a customer service gesture without prioritizing their own business traffic.    They said, still our fault.  and we responded ‘interesting, you have cut your speed by 1/6000 during games - so please use your phones, or implement QOS.  if its not faster, we’ll look again’


— we do have the discussion that 'the internet is not reliable and is designed to break and recover’. (because it is).  so we use various analogies like 'cell phone dropping in a tunnel', or ‘ever notice that even google does not always give you answers fast’ or that ‘you ever see pages for others load slowly’  they always respond yes ‘hey I’ve seen that’.  

— or.  did you call your ISP and ask if they had any equipment failures.

— or, did you reboot your own routers and did things get better.

traffic shaping and limits:    Some people have internet plans that throttle traffic after a certain rate of data transfer (eg ,first 1 meg is fast, then it goes slow on any request), or after a certain limit. (50 gigs/day, then go slow), etc.   If we identify those, we help people find a better ISP.



all in all, slowness is usually addressed with one of:
finding a fault wit their network
searching for internet outages on their ISP
helping with new ISP's
and and educational approach about what to look for

YES, we are always blamed,  but I find that most times, some education to find root cause gets customers on your side as they remember that failure or slow performance is ‘once in a while or rare’ and not all the time.







And as for performance improvements

there are a couple of tools in the toolshed that you can do to make things faster.  Also some examples:

Caching is a big one.    We write our apps so that data, once requested, can reside on the machine and we employ algorithms for determining when and how often to flush the cache.    we use:

— CDN’s (Content delivery networks) to provide the foundational boostrap code that is part of our web experience

— HTTPS/2 as much as possible (its faster than HTTP/1.1)

— we use gzip compression on all our web pages.   You tell the web server (we use nginx to compress the page before sending) if the browser supports it.  it means less total volume of traffic and a faster response to the end user.  Apache supports gZip as well — its a parameter setting.   If the user’s browser does not support gzip, the web server won’t do it.    These days, every browser does it except the really oles ones like IE, so we don’t support them at all and get away with it because we require at least TLS1.2 that is available in all modern browsers.

— streaming load times —  as in we break our request up the right way so that people see some data and let the rest load in the background while they see something

— images as pushed forward from the app into our web server (we use NGINX) so that they do not burden our app when they re requested over and over

— fully cache some sort of static pages — for example, people ask for lists of events off our web site.  so we track cache keys (eg all events, todays events, this weeks events) and if two people ask for the same page, then we built is once, and store it in ram.  the second person to ask, gets the same page out of memory without any disk i/o.  difference in performances is as much as 8 to 10 times greater.   and we nullify that cache key every 30 seconds or a minute since it is not materially different.  It means one long request periodically, and may short ones.

a request that is made by the user suffers a longish request, then many fast ones.    Of course, if the data needs to be instantly up to date, those pages go slower

— we also use the cache-control http header to give the page a bit of a timeout without needing to reset it.   browsers can check to see if a page has changed (new header) and if not, display the old page.

— we break up our data into changeable parts and constant parts.  For example, we show maps of a venue where people sit.    There are four components to that map.   These are ‘physical seat location’, ‘which seats are taken’, ‘background map’, ‘and javscript to load the three parts and merge the data’.      We saw an 8 fold increase in displaying seating location to the customer when when realized that the only part that really changed was the ‘which seats’ are taken.   so once a person wants to find tickets, we download all 4 parts of the venue display to the customers browser, then only reload the ‘which seats are taken’ as they decide to change seats.

— sometimes a customer will complain that a certain set of conditions will cause a report or response to go slow.  I pay attention to those.  Over the course of the past 6 years since we put our app on the cloud, I’ve rewritten a bunch of SQL calls to gather sets of data (one call), rather than a record at a time (many calls), since each SQL call introduces latency.    It has improved the application a lot and a slow spot goes away.

how do I find these in omnis.   I use the trace log and the option to ’Log Diagnostic messages’ (right click on tracelog window).    That sets the trace log to only show messages that are fabricated somewhat like below:

Send to trace log (Diagnostic message) [tim(#T,'h:N:S.s')] this is the sql [SQLStatement]

 and then any SQL call appears in my tracelog.   if I have one or two, SQL is not the issue.   if I have many, then ‘Houston, we have a problem’.   I also use it to track timers, or long running routines if thats what I]m trying to diagnose.    Its been very helpful finding sever side issues.

— background workers:  we have database logs.  I use the postgres workers to write logs in the background so that it doesn’t impact the ’things that need to happen now’.   basically defer what you can and only do the necessary stuff now.  

Another example of this is sending emails.  that makes web applications seem slow.  so we use the worker design pattern to break up creating emails and sending emails.   Creating an email is writing a record to the database.  then we have some worker that wakes up every 15 seconds and looks for emails to send — and sends them in the background, from another machine.     This makes your application appear faster — by separating out essential work and work that can be done later after a delay when things are quieter or on another machine.


— and I look at nginx logs and other logs we have put int he database to see where time goes.





— and there are more.

the point is: there are tools that you need to know about and employee to make your web app
— compress data
— cache data
— send only what it needs (and not resend entire payloads)
— minimize disk access at the server that slows page creation
— push static pages forward from the application to the web browser
— use CDN’s to deliver high static payloads (bootstrap is one that we use) without touching your web site ..  those tend to be highly cached by various ISP’s and distributed around the world to improve performance
— stream some large data without requiring the entire download to occur before you see the thing of interest
— display text with images to follow and load later as separate requests


all the above are key reasons why we use stateless connections and ultra thin web pages.





hope that helps.  building web apps is not the same as building client server apps.  You have to design for latency in your application and find ways to split up things that you might normally do in a linear fashion.


hopefully the examples and analogy help.    However, you will get blamed since thats society work these days.   Its always your fault, never mine and thats where education of customers comes in.




Doug Easterbrook
Arts Management Systems Ltd.
mailto:doug at artsman.com
http://www.artsman.com
Phone (403) 650-1978

> On Mar 5, 2023, at 8:59 PM, Stephen Miller <stephenmiller1958 at gmail.com> wrote:
> 
> Hi All
> 
> I have been thinking with respect to web-based apps. Developers are really
> losing control over the performance of their apps due to the vagaries of
> the bandwidth at any point in time.
> 
> I, for example, have 5G wireless, and am guaranteed 100 mbps and normally
> get 180 mbps. But today I am getting 20 mbps.
> 
> Got me thinking that developers must be using some independent method of
> testing the bandwidth at any point in time which led me to speedtest dot
> net.
> 
> Ookla has an api for developers which runs on linux and MAC which led to
> this posting.
> 
> I was curious how others do it. Do you just ping your server and a known
> reliable web-site - I use www.ibm dot com to get a performance base
> normally,  or do you just not worry about it and just take the calls from
> irate customers?
> 
> Kind Regards,
> 
> Stephen Miller
> _____________________________________________________________
> Manage your list subscriptions at https://lists.omnis-dev.com
> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com 



More information about the omnisdev-en mailing list