I want to point out to potentially serious performance issue when importing large number of images into MG.
We have well tuned web servers with several large, complicated MODX sites. Properly using MODX caching and elaborated site design, we have long time record of both good user experience, while web servers keep running around 10% CPU on average with plenty of free memory as a safety margin.
The only exception is importing many files into MG by users due to nature of thumb generation in manager: for every uploaded image browser calls MG connector that creates thumb, resulting in hundreds of concurrent CPU and memory intensive requests to server. Even worse, if you define default crop or crops in system settings, for every uploaded image those additional crops are generated as well.
This can kick out even server with enough spare resources for any normal operation. Depending on server config, this can result in killing background processes, unresponsive server, reboot…
While user uploading images is authenticated in manager and those requests are legitimate, there is a few defenses that can be done on server side. You don’t want to restrict number of requests per single user, so any normal request are served as fast as possible. Depending on server architecture you can limit max number of PHP workers (e.g. fast-cgi processes), but again this will impact normal operations.
The best soulution would be if MG can somehow queue request for thumb generation, sending only limited number of requests at a time.
For those experiencing this issue, here are few tips:
ask your editors to upload only limited number of images at once (frustrating and unconvenient)
don’t define default crops in MG system setting. Use phpthumbof (contrary to MG documentation advice). For a single front-end page even with many images, only one php task is created. This is slow, but only the first time, as results are cached. Even if this task exceeds max_execution_time and is killed, subsequent reload continues to create more thumbs. You can event try to raise max_execution_time if you are confident there are no other time consuming scripts used
pre-process images to minimum necessary size before importing to MG. This can reduce memory and cpu usage
I’m referring to act of uploading images to Gallery resource (for example drag-and-drop many files into fresh Gallery). Every uploaded image calls immidiately:
/assets/components/moregallery/connector.php?action=mgr/images/upload
that creates all necessary thumbs (more than one per image if crops are defined).
MODX Media Browser might seem to have similar issue, and sometimes it really can cause troubles. But:
the big difference here is lazy-load used for thumbs. So Media Browser calls phpthumb only for visible images. This, even on large screens is about several dozens. Sure, fast scrolling down the folder makes a lot of requests
only one 100px wide thumb is produced for each image
Oh, really? This is the first I’ve heard that the upload itself is enough to overwhelm a server, and I’ve stress-tested it in the past on non-optimized servers.
Because you mentioned “hundreds of concurrent requests”, are you actually uploading hundreds of images into a single gallery at once here? And not just as a one-time thing, but regularly? If so I’m definitely interested in learning more about your use case, cause that seems like a ton of assets.
I did take a quick look at our code. I thought we already had some queuing system active for uploads that limits the number of requests going at the same time, but that doesn’t seem to be the case. However, the library we handle the multi-upload with does have an option to enable a queue, so I’ve made a note in our tracker to see if that works well.
We do have a couple of MoreGallery features and improvements that are pending release for after my vacation time, so if that works as advertised, we’ll squeeze that in.
.
FWIW, MoreGallery creating the thumbnail and crops on upload is an optimization feature so it doesn’t have to do that later. While it’s possible to argue for different approaches in different situations, in general we decided that letting the admin user wait a second or two longer for the upload, and doing that just once, is better than forcing that on front-end users repeatedly with phpthumb.
If you set up your crops properly for all your different types of frontend usage and have those generated at upload time, those turn into simple static file references rather than needing any sort of processing, which keeps your front-end humming along nicely.
Well, usually we don’t have troubles either. During years we have tested many different setups and approaches, so I guess we understand quite well underlying tech. While we fully manage our servers, we also take into account apache setup, fast-cgo config, user separation etc.
Our latest project that we run since this spring is a bit different. It’s Slovak Golf Association website, where some parts are public, so you can have a look: https://www.skga.sk/ (sorry slovak only at the moment). The core functionality is heavily based on presenting and editing all data aspects of playing golf in Slovakia (players, tournaments, resorts, clubs, handicap history etc.). This is implemented using REST API on external server, not really MODX stuff.
But there is also “editorial” stuff, like everyday news, documents and of course GALLERIES! We combine MoreGallery with Collections as I mentioned here Moregallery first image of the resource rendered in the collections extra? - #9 by hrabovec.
These galleries are uploaded by PR agencies usually after some tournaments, one or two per weekend. There is usually about 200 images, but for example this one Slovenská golfová asociácia have more than 500. This triggers more that 1000 concurrent requests for thumb generation (mgr thumb plus one 300x200 for front-end overview). Large images for lightbox we generate using phpthumbof, otherwise there would be even more trouble.
It also should be mentioned that it’s a big difference if someone uploads files coming directly from his DSLR (like 20Mpx), or process them in Lightroom or so to max. HD size. In this case limiting uloaded file size is not quite a measure, because filesize is dependent on pixel dimensions, bit depth and compression, while memory usage while processing the file is only dependent on pixel size and bit depth.
One possible optimization mentioned in docs is using mgr thumb directly in template or as a base for phpthumb. But it’s unable to setup custom mgr thumb generation (e.g. zoom-crop settings), so we need to generate extra thumb with own settings. And we also had trouble setting up correctly the large image generation in crop settings (but this is separate issue).
To sum it up: generating all thumbs and sizes on upload is generally a good idea, but devil hides in details. So if you can somehow implement queuing that would be great. We are here to test it heavily!
