macOS external drive - Omnis DataBridge / Postgres issues

Mon Feb 17 08:38:04 UTC 2025

Thanks for the feedback, Doug and Andy.

I think I found the culprit: Apple Spotlight.
By default, we turn Spotlight off on our servers with the mdutil command.
But I think that doesn't apply to any newer drives that are connected afterwards.

By excluding the external drive in the Spotlight settings, the issue no longer occurs.
Postgres has been running smoothly for 3 days now, so I think that is it.

Cheers,

Nick Renders

On 12 Feb 2025, at 1:45, Doug Easterbrook via omnisdev-en wrote:

> time machine behaviour  is interesting.
>
> I have had much better luck using network drives (like Synology DS 1861 and a truenas server) that connect as SMB shares.    I have some old AFP time machines (the apple white ones) and I was using a drobo for quite ages up until two months ago.
>
> In the Synology and Truenas servers, I have system  level read/write caches using SSD’s before data gets committed to the old spinning disks and I have to say that they are now FAST, and always complete  — with no hint of problems between backup sessions.
>
>
> I had problems on the old apple time machine boxes and the Drobo one because they just were not fast enough and invariably something, someplace would go to sleep, leaving me with a message like ’backups not complete’, or ’some files could not be backed up’.
>
> I no longer have this issue…   but speed of drives and wakeness (if thats a word) seems to play a factor.    highly recommend a decent NAS box with some decent SSD’s for read/write cache.
>
>
>
> I don’t know about the specifics of your machine, nor the external drives — but I’ve seen the Volume disconnected on slower drives and when my machine has gone into screensaver mode.   I have an M1 Macbook Pro and I tell it never to sleep…. but it does seem to go to low power mode or something that affects writing to drives, even though I’ve turned off all plaes that I know of that allow things to disk or network sleep'
>
>
>
> Doug Easterbrook
> doug at artsman.com
> Phone (403) 650-1978
>
>> On Feb 11, 2025, at 4:05 PM, Andy McVeigh via omnisdev-en <omnisdev-en at lists.omnis-dev.com> wrote:
>>
>> Not sure if this is relevant but using external drives for Time Machine backups and since OS 15 upgrade the 'Volume has been disconnected' message regular comes up and then it just reconnects later on
>>
>> I have checked all cables and replaced and tried different drives and the pattern continues but the TM backups get done
>>
>> Maybe it is this intermittent loss of connection to external drives
>>
>> Andrew McVeigh
>> Surfway Real Solutions
>> Phone 02 44412679 Mobile 0418428016
>> www.surfway.com.au
>> www.berrarabeach.com.au
>>
>> <http://www.surfway.com.au/>
>>> On 12 Feb 2025, at 2:48 am, Doug Easterbrook via omnisdev-en <omnisdev-en at lists.omnis-dev.com> wrote:
>>>
>>> hi Nick.
>>>
>>> You should talk to apple — I truly suspect it is a gate keeper problem.    Apple (and their penchant for uber security) has done a disasterous job allowing software running on a machine to access just about anything.
>>>
>>> Yesterday Ben Weinberg reported a problem writing print files or pdf’s to external drives.     In a short test of my own. I tried to CD into an external mounted volume and got forced to give permission to terminal (as in apple’s terminal app that is part of the root/base BSD unix shell) to allow it to access any external volume.
>>>
>>> Others on the list have complained as well for other reasons, always with Mac OS, and worse on apple silicon machines.
>>>
>>>
>>>
>>> you might try giving postgres 'full disk access' in system processes.
>>>
>>>
>>>
>>> on the other hand, I’d suggest NOT using external drives with databases (or other 24x7 processes) on MAC os.     You might want to use linux or free BSD instead.
>>>
>>>
>>> I have had zero issues with postgres on OSX when the data is on the local machine, and I suspect thats because postgres talks to everything within its own directory, which apple seems fine with.
>>>
>>>
>>>
>>>
>>>
>>> the other thought I might have is that perhaps postgres crapped out on you.   The postmaster.pid file is a small file that indicates postgres is running, or if it shut down cleanly.   its is normally in the data directory (on the local machine).    If you’ve put the entire data directory on an external drive ….   might be that postgres restarted itself, or did something with the file.
>>>
>>> https://www.crunchydata.com/blog/postgres-postmaster-file-explained
>>>
>>>
>>>
>>>
>>> bottom line, postgres is so light (i.e. doesn’t take up much space on a drive) that there is no reason not to run it on a local drive, vs a network drive.
>>>
>>> if you want data on a network drive, why not run postgres on that machine ???
>>>
>>>
>>>
>>>
>>> Doug Easterbrook
>>> doug at artsman.com
>>> Phone (403) 650-1978
>>>
>>>> On Feb 11, 2025, at 1:32 AM, Nick Renders via omnisdev-en <omnisdev-en at lists.omnis-dev.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> We are having intermittent issues with external drives on macOS Sonoma (14) and Sequoia (15).
>>>> Sometimes, the privileges of certain ( or maybe all? ) files seem to momentarily change.
>>>>
>>>> The issue is most noticeable when we have a Postgres data directory located on an external volume. The Postgres service is running fine, then all of sudden, we are no longer able to connect to the database. In the Postgres log, the following is logged every minute:
>>>>
>>>> 	2025-02-10 22:12:06.198 CET [10267] LOG:  could not open file "postmaster.pid": Operation not permitted; continuing anyway
>>>> 	2025-02-10 22:13:06.223 CET [10267] LOG:  could not open file "postmaster.pid": Operation not permitted; continuing anyway
>>>> 	2025-02-10 22:14:06.225 CET [10267] LOG:  could not open file "postmaster.pid": Operation not permitted; continuing anyway
>>>> 	2025-02-10 22:15:06.227 CET [10267] LOG:  could not open file "postmaster.pid": Operation not permitted; continuing anyway
>>>> 	2025-02-10 22:16:06.230 CET [10267] LOG:  could not open file "postmaster.pid": Operation not permitted; continuing anyway
>>>> 	...
>>>>
>>>> This continues until we restart the Postgres service (pg_ctl restart).
>>>>
>>>> The same issue occurs with Omnis Datafiles: when we run an Omnis DataBridge with the datafiles located on an external volume, at some point, the files are no longer accessible. The ODB service continues running, but trying to open a Datafile will result in the following error in messages.txt of the ODB:
>>>>
>>>> 	Loading user config file...done
>>>> 	Client IP = 127.0.0.1: Omnis Bridge Error: Unable to open specified segment
>>>> 	System Error 1 : Operation not permitted
>>>> 	Loading user config file...done
>>>> 	Client IP = 127.0.0.1: Omnis Bridge Error: Unable to open specified segment
>>>> 	System Error 1 : Operation not permitted
>>>> 	Omnis Bridge Error: Error while waiting for messages
>>>> 	System Error 54 : Connection reset by peer
>>>>
>>>> Restarting the ODB service fixes the issue again.
>>>>
>>>>
>>>> We've noticed this issue with 3 different volumes, two being a Promise Pegasus RAID, the third being a simple SSD connected with USB-C. The servers have been both Intel and Silicon Macs (i7, M1, M2) running Sonoma or Sequoia.
>>>> The external volumes have "Ignore ownership" turned on in the Info window in the Finder. We have tried adjusting the user privileges/ownership of specific folders, but no change.
>>>>
>>>> The issue is very intermittent. The server can be running great for a couple of hours, then it happens out of the blue. As you can see in the Postgres log above, it first occurred at 22:12, when nothing else was happening on the machine.
>>>>
>>>>
>>>> Does this sound familiar to anyone?
>>>> Any ideas or suggestions?
>>>>
>>>>
>>>> Kind regards,
>>>>
>>>> Nick Renders
>>>> _____________________________________________________________
>>>> Manage your list subscriptions at https://lists.omnis-dev.com
>>>> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com
>>>
>>> _____________________________________________________________
>>> Manage your list subscriptions at https://lists.omnis-dev.com
>>> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com
>>
>> _____________________________________________________________
>> Manage your list subscriptions at https://lists.omnis-dev.com
>> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com
>
> _____________________________________________________________
> Manage your list subscriptions at https://lists.omnis-dev.com
> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com