AW: Document Processing question

[Rudolf Bargholz]

Hi Paul,

To be honest, Andrew's solution will probably be the most performant solution to your problem, however you mention the starting point of your problem, the end result, but are glossing over the important middle part that would really help to give you decent information. What are the rules to determine if a file needs to be archived or not? Using robocopy will likely only work if you can tell robocopy what the rules are to archive/move files. If the rules for archiving cannot be mapped to the robocopy command, perhaps an easier solution would be to write the source and destination paths into a table on your database, then in an asynchronous process use the data from this table to create a batch file that does the moving of the files:

move  /Y "c:\source\file1.txt" "z:\destination\file1.txt"
move  /Y "c:\source\file2.txt" "z:\destination\file2.txt"

Depending on your rules on how to determine which files need to be batched, you could even create a simple SQL to mark the files to archive, then copy the files to the copy table, and then delete the old file records, e.g.

Begin transaction
1)
UPDATE DMS set ARCHIVEFLAG=1,GROUPTIMESTAMP=@[MyTimeStamp] WHERE CREATEDATE<@[Date]

2)
merge into ARCHIVE dest
using (
select SEQ
,SOURCEFILE
from DMS
where
ARCHIVEFLAG=1 and GROUPTIMESTAMP=@[MyTimeStamp]
) as source(SEQ, SOURCEFILE)
on source. SEQ = dest. SEQ
when not matched
then
insert (
SEQ
, SOURCEFILE
)
values (
source. SEQ
,source.BDEA_LEISTTYP
,source. SOURCEFILE
)

3)
DELETE from DMS where ARCHIVEFLAG=1,GROUPTIMESTAMP=@[MyTimeStamp]

Commit if everything above succeeded

A second async process could then process the ARCHIVE table to copy the files from the old path to the archive path, with whatever workflow you decide on.

By splitting up the workflow into two steps, you will take away a lot of the pressure to have this process work as fast as possible.

Regards

Rudolf Bargholz

-----Ursprüngliche Nachricht-----
Von: omnisdev-en <omnisdev-en-bounces at lists.omnis-dev.com> Im Auftrag von Paul Manning via omnisdev-en
Gesendet: Mittwoch, 7. September 2022 19:41
An: OmnisDev List - English <omnisdev-en at lists.omnis-dev.com>
Cc: Paul Manning <paulrmanning58 at icloud.com>
Betreff: Document Processing question

Background. I am working with a company that handles a massive amount of documents.  The system is set up to store these documents on a drive on the network. They are wanting to start archiving/deleting old records.  Due to time requirements they will need to archive to a separate drive then come back later and delete those.

It presently goes through data records to determine which have docs to archive and does a fileOps.$move… to move files from the working directory to the archive directory. Well this takes a bit of time when you are dealing with thousands of files.  Might there be a way to batch these moves?

Paul

************************************************
Paul R. Manning
paulrmanning58 at icloud.com