Extracting binary response from HTTP call within Omnis

[TBS]

Paul

Many many thanks (also to Reg Paling too who replied off list !)

I was so close its annoying !!!

I needed to basically morph my two calls into one - so :

HTTPPage (URL,,kFalse) Returns lvBuffer

And then the secret was really knowing to subtract 1 from the bytemid final parameter :

Calculate lvBuffer as bytemid(lvBuffer,binsearch(headerEndString,lvBuffer)+4,binlength(lvBuffer)-1)

Et voila :) Job done - thank you guys for your prompt and insightful responses !

Andy Hilton
Totally Brilliant Software Inc
Phone (US) : (863) 409 4870
Phone (UK) : 0207 193 8582
Web : www.totallybrilliant.com
Helpdesk : http://totallybrilliant.kayako.com
Email : andyh at totallybrilliant.com
On Nov 6, 2022, 7:40 PM -0500, Paul Mulroney via omnisdev-en <omnisdev-en at lists.omnis-dev.com>, wrote:
> Hi Andy,
>
> This is what we use to download binary files using Omnis. This is in Studio 8.x. We call this method httpDownload() with the URL of the file to download, and the path to where we want to save it. Hope this helps!
>
> Regards,
> Paul.
>
>
> ; Download the given URL to the given filename
> ; IN: psURL, psFilePath
>
> ; - Open the file to write
> Calculate vsTempPath as con(psFilePath,'.part')
> Do FileOps.$setunixpermissions(vsTempPath,'-rwxrwxrwx')
> Do FileOps.$deletefile(vsTempPath) ;; Delete the previous file to be sure to be sure.
> Do voFile.$createfile(vsTempPath) ;; Create the file for writing
> Calculate vnFilePos as 0
> Calculate vbDownloadOK as kTrue ;; Assume that there's no error
>
> ; - Get the file name for the progress bar.
> Do FileOps.$splitpathname(psFilePath,vsDrive,vsDir,vsFile,vsExt)
> Calculate vsLibName as con(vsFile,vsExt)
> Calculate vsHeaderDivider as binfromhex('0D0A0D0A') ;; Cr/Lf x 2
> HTTPSplitURL (psURL,vsServer,vsURI) ;; Split the URL into a server address and URI component.
> If pos('https://',psURL)<>0 ;; Using a secure connection
> HTTPGet (vsServer,vsURI,,,,kTrue,kFalse,kFalse) Returns vnSocket ;; Send the request to the server, get a socket back. True=Secure, False=Don't verify (this might be a problem, but for now leave it)
> Else ;; Not secure
> HTTPGet (vsServer,vsURI) Returns vnSocket ;; Send the request to the server, get a socket back
> End If
> If vnSocket>=0 ;; We were able to connect OK
> Calculate vbDone as kFalse ;; We're not done downloading the file yet.
> Calculate vbHeader as kFalse ;; We've not yet received the header.
> ; - Get the header first. From that we can get the file size and give a reasonable working message.
> Repeat
> TCPReceive (vnSocket,vsBuffer,1000) Returns vnReceived
> If vnReceived>0 ;; We received some data
> Calculate vsHeader as bytecon(vsHeader,vsBuffer) ;; Append it to the header buffer
> Else If vnReceived=0 ;; 0=Done
> Calculate vbDone as kTrue
> Else If vnReceived=-10035
> ; Do nothing, this is really a "data not yet available" message.
> Else If vnReceived<0 ;; Error
> Calculate vsResult as con('An error occurred downloading the file. Error code is ',vnReceived)
> Calculate vbDone as kTrue
> Calculate vbDownloadOK as kFalse ;; We bombed out.
> End If
> Calculate vbHeader as binsearch(vsHeaderDivider,vsHeader)>=0 ;; Keep going until we received CR/LF x2. Not found = -1
> Until vbHeader|vbDone
> If not(vbDone) ;; Not done, so we must have received the header OK, now download the rest
> ; - We got the header, split into the key/value pairs so we can look at the headers we care about easily.
> Calculate vnSplit as binsearch(vsHeaderDivider,vsHeader)
> Calculate vsBuffer as bytemid(vsHeader,vnSplit+4,binlength(vsHeader)-1) ;; Get the start of the download here. Note that with bytemid, the first byte starts at position 0, so lengths need to be offset by 1 also.
> Calculate vsHeader as bytemid(vsHeader,0,vnSplit-1)
> Do voFile.$writefile(vsBuffer,vnFilePos) Returns vnErr
> Calculate vnFilePos as vnFilePos+binlength(vsBuffer)
>
> ; - Work out the file length, by extracting all the headers
> Do vlHeader.$define(vsKey,vsValue)
> Calculate vsHeaderAsText as utf8tochar(vsHeader) ;; I'm not sure this is kosher, but it seems to be able to convert a binary field into a character field that we can successfully parse.
> Calculate vsLine as strtok('vsHeaderAsText',chr(13,10)) ;; First line is eg HTTP/1.1 200 OK
> While len(vsHeaderAsText) ;; Keep going until we process the entire header.
> Calculate vsLine as strtok('vsHeaderAsText',chr(13,10))
> Calculate vsKey as strtok('vsLine',':')
> Calculate vsValue as vsLine
> If vsKey<>'' ;; We got something valid, add to the list
> Do vlHeader.$add(vsKey,vsValue)
> End If
> End While
> Do vlHeader.$search(vsKey='Content-Length',kTrue,kFalse,kFalse,kFalse) ;; Look for the length of the file to receive.
> Calculate vnFileLength as vlHeader.vsValue ;; In bytes. Used in our progress message below.
>
> ; - OK, get the rest of the download.
> Calculate vnProgressRange as vnFileLength/1000 ;; We're downloading in 1k blocks, so that should be the number of times we're going around the loop.
> Repeat
> TCPReceive (vnSocket,vsBuffer,1000) Returns vnReceived ;; We're downloading 1Kb at a time
> If vnReceived>0 ;; Received something, write it to disk
> Working message Updater/-1073735799,-1073735795;15;0;0 {Downloading [vsLibName]//Received [rnd(vnFilePos/vnFileLength*100,0)]%}
> Redraw working message
> Do voFile.$writefile(vsBuffer,vnFilePos) ;; Write it to disk now.
> Calculate vnFilePos as vnFilePos+vnReceived ;; Increment our position in the file.
> Else If vnReceived=0 ;; 0=Done
> Calculate vbDone as kTrue
> Else If vnReceived=-10035
> ; Do nothing, this is really a "data not yet available" message.
> Else If vnReceived<0 ;; Error
> Calculate vsResult as con('An error occurred downloading the file. Error code is ',vnReceived)
> Calculate vbDone as kTrue
> Calculate vbDownloadOK as kFalse ;; We bombed out.
> End If
> Until vbDone
> Close working message
> End If
> HTTPClose (vnSocket) ;; We're done with the connection.
> Else ;; Socket error
> Calculate vsResult as con('An error occurred downloading the file. Error code is ',vnSocket)
> Calculate vbDownloadOK as kFalse ;; We bombed out.
> End If
> ; - Tidy up.
> Do voFile.$closefile ;; Finished downloading the temporary file
> If vbDownloadOK ;; We got it OK.
> Do FileOps.$setunixpermissions(psFilePath,'-rwxrwxrwx')
> Do FileOps.$deletefile(psFilePath) ;; Make sure there isn't one there already
> Do FileOps.$rename(vsTempPath,psFilePath) ;; Rename it back to what it should be.
> Calculate vsResult as '' ;; No errors.
> Else ;; Didn't download OK, delete the temp file
> Calculate vsResult as pick(vsResult='',vsResult,'An error occurred downloading the file') ;; A generic error
> Do FileOps.$setunixpermissions(vsTempPath,'-rwxrwxrwx')
> Do FileOps.$deletefile(vsTempPath)
> End If
> Quit method vsResult
> ; --- history
>
>
>
> > On 7 Nov 2022, at 6:24 am, TBS <andyh at totallybrilliant.com> wrote:
> >
> > Folks
> >
> > Trying to extract a binary file from an HTTP response from an Omnis method HTTP call…..the call (when made in a browser) simply spits back an excel file, and that’s what I am trying to get back here !
> >
> > I use :
> >
> > HTTPGet (lvHost,lvURI) Returns SocketNum
> >
> > To initiate the connection and get my socket, all good.
> >
> > Then I have 2 calls going on (more just for testing here !) :
> >
> > HTTPPage (URL) Returns PageContent
> > And
> > HTTPRead (SocketNum,lvBuffer) Returns lvByteCount
> >
> > In the first line - that issues the call that I want to make to the server and I get a text based response in PageContent that starts with :
> >
> > HTTP/1.1 200 OK
> > Date: Sun, 06 Nov 2022 22:07:00 GMT
> > Server: WSGIServer/0.2 CPython/3.9.7
> > Content-Type: application/vnd.ms-excel
> > Content-Disposition: attachment;filename=DetailedWhereabouts_06-Nov-2022.xlsx
> > X-Content-Type-Options: nosniff
> > Referrer-Policy: same-origin
> > X-Frame-Options: DENY
> > Content-Length: 6445
> > Vary: Cookie
> >
> >
> > And then it appears to have the attachment - presumably as a binary but of course in text its gobbledy gook - not sure if I can extract the binary from the text here ?
> >
> > The second one (HTTPRead) returns a binary field (lvBuffer) that ‘looks’ like it may have the binary file I want plus a little bit more (I am suspecting it is actually the same as the HTTPPage response but as a binary field ??)
> >
> > My guess is that the actual binary file that I am interested in follows a known binary string in the binary field returned (like maybe after headerEndString - chartoutf8(chr(13,10,13,10)) ) - but this is where my knowledge comes to an end ! Does anyone have knowledge of the make up of HTTPRead binary ?? i.e. if I can use something like :
> >
> > If binsearch(headerEndString,lvBuffer)>0
> > Calculate lvBuffer as bytemid(lvBuffer,binsearch(headerEndString,lvBuffer)+4,binlength(lvBuffer))
> > End If
> >
> > To get my actual binary file being returned…..
> >
> > I’m good when I get it - just need to get it :)
> > Any and all help appreciated…..
> >
> > Andy Hilton
> > Totally Brilliant Software Inc
> > Phone (US) : (863) 409 4870
> > Phone (UK) : 0207 193 8582
> > Web : www.totallybrilliant.com
> > Helpdesk : http://totallybrilliant.kayako.com
> > Email : andyh at totallybrilliant.com
> > _____________________________________________________________
> > Manage your list subscriptions at https://lists.omnis-dev.com
> > Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com
>
>
>
>
> <https://www.logicaldevelopments.com.au/> Paul W Mulroney
> Logical Developments
> Customised Software Solutions
> Ph: 08 9458 3889
> We Don't Do Simple Pty Ltd
> trading as
> Logical Developments
> ACN 161 009 374
> <https://www.facebook.com/logicaldevelopmentswa/> <https://twitter.com/WeDontDoSimple> <https://www.linkedin.com/company/logicaldevelopments/> <https://www.youtube.com/channel/UCPldVVgWR05WX3cVrR5WUQw>
> _____________________________________________________________
> Manage your list subscriptions at https://lists.omnis-dev.com
> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com