Extracting binary response from HTTP call within Omnis

Paul Mulroney pmulroney at logicaldevelopments.com.au
Mon Nov 7 00:38:51 UTC 2022 

Hi Andy,

This is what we use to download binary files using Omnis.  This is in Studio 8.x.  We call this method httpDownload() with the URL of the file to download, and the path to where we want to save it. Hope this helps!

Regards,
Paul.


;  Download the given URL to the given filename
;  IN: psURL, psFilePath

;  - Open the file to write
Calculate vsTempPath as con(psFilePath,'.part')
Do FileOps.$setunixpermissions(vsTempPath,'-rwxrwxrwx')
Do FileOps.$deletefile(vsTempPath)     ;; Delete the previous file to be sure to be sure.
Do voFile.$createfile(vsTempPath)     ;; Create the file for writing
Calculate vnFilePos as 0
Calculate vbDownloadOK as kTrue     ;; Assume that there's no error

;  - Get the file name for the progress bar.
Do FileOps.$splitpathname(psFilePath,vsDrive,vsDir,vsFile,vsExt)
Calculate vsLibName as con(vsFile,vsExt)
Calculate vsHeaderDivider as binfromhex('0D0A0D0A')     ;; Cr/Lf x 2
HTTPSplitURL (psURL,vsServer,vsURI)      ;; Split the URL into a server address and URI component.
If pos('https://',psURL)<>0     ;; Using a secure connection
	HTTPGet (vsServer,vsURI,,,,kTrue,kFalse,kFalse) Returns vnSocket     ;; Send the request to the server, get a socket back.  True=Secure, False=Don't verify (this might be a problem, but for now leave it)
Else     ;; Not secure
	HTTPGet (vsServer,vsURI) Returns vnSocket     ;; Send the request to the server, get a socket back
End If
If vnSocket>=0     ;; We were able to connect OK
	Calculate vbDone as kFalse     ;; We're not done downloading the file yet.
	Calculate vbHeader as kFalse     ;; We've not yet received the header.
	;  - Get the header first.  From that we can get the file size and give a reasonable working message.
	Repeat
		TCPReceive (vnSocket,vsBuffer,1000) Returns vnReceived
		If vnReceived>0     ;; We received some data
			Calculate vsHeader as bytecon(vsHeader,vsBuffer)     ;; Append it to the header buffer
		Else If vnReceived=0     ;; 0=Done
			Calculate vbDone as kTrue
		Else If vnReceived=-10035
			;  Do nothing, this is really a "data not yet available" message.
		Else If vnReceived<0     ;; Error
			Calculate vsResult as con('An error occurred downloading the file.  Error code is ',vnReceived)
			Calculate vbDone as kTrue
			Calculate vbDownloadOK as kFalse     ;; We bombed out.
		End If
		Calculate vbHeader as binsearch(vsHeaderDivider,vsHeader)>=0     ;; Keep going until we received CR/LF x2.  Not found = -1
	Until vbHeader|vbDone
	If not(vbDone)     ;; Not done, so we must have received the header OK, now download the rest
		;  - We got the header, split into the key/value pairs so we can look at the headers we care about easily.
		Calculate vnSplit as binsearch(vsHeaderDivider,vsHeader)
		Calculate vsBuffer as bytemid(vsHeader,vnSplit+4,binlength(vsHeader)-1)     ;; Get the start of the download here.  Note that with bytemid, the first byte starts at position 0, so lengths need to be offset by 1 also.
		Calculate vsHeader as bytemid(vsHeader,0,vnSplit-1)
		Do voFile.$writefile(vsBuffer,vnFilePos) Returns vnErr
		Calculate vnFilePos as vnFilePos+binlength(vsBuffer)
		
		;  - Work out the file length, by extracting all the headers
		Do vlHeader.$define(vsKey,vsValue)
		Calculate vsHeaderAsText as utf8tochar(vsHeader)     ;; I'm not sure this is kosher, but it seems to be able to convert a binary field into a character field that we can successfully parse.
		Calculate vsLine as strtok('vsHeaderAsText',chr(13,10))     ;; First line is eg HTTP/1.1  200 OK
		While len(vsHeaderAsText)     ;; Keep going until we process the entire header.
			Calculate vsLine as strtok('vsHeaderAsText',chr(13,10))
			Calculate vsKey as strtok('vsLine',':')
			Calculate vsValue as vsLine
			If vsKey<>''     ;; We got something valid, add to the list
				Do vlHeader.$add(vsKey,vsValue)
			End If
		End While
		Do vlHeader.$search(vsKey='Content-Length',kTrue,kFalse,kFalse,kFalse)     ;; Look for the length of the file to receive.
		Calculate vnFileLength as vlHeader.vsValue     ;; In bytes. Used in our progress message below.
		
		;  - OK, get the rest of the download.
		Calculate vnProgressRange as vnFileLength/1000     ;; We're downloading in 1k blocks, so that should be the number of times we're going around the loop.
		Repeat
			TCPReceive (vnSocket,vsBuffer,1000) Returns vnReceived     ;; We're downloading 1Kb at a time
			If vnReceived>0     ;; Received something, write it to disk
				Working message Updater/-1073735799,-1073735795;15;0;0 {Downloading [vsLibName]//Received [rnd(vnFilePos/vnFileLength*100,0)]%}
				Redraw working message
				Do voFile.$writefile(vsBuffer,vnFilePos)     ;; Write it to disk now.
				Calculate vnFilePos as vnFilePos+vnReceived     ;; Increment our position in the file.
			Else If vnReceived=0     ;; 0=Done
				Calculate vbDone as kTrue
			Else If vnReceived=-10035
				;  Do nothing, this is really a "data not yet available" message.
			Else If vnReceived<0     ;; Error
				Calculate vsResult as con('An error occurred downloading the file.  Error code is ',vnReceived)
				Calculate vbDone as kTrue
				Calculate vbDownloadOK as kFalse     ;; We bombed out.
			End If
		Until vbDone
		Close working message
	End If
	HTTPClose (vnSocket)      ;; We're done with the connection.
Else     ;; Socket error
	Calculate vsResult as con('An error occurred downloading the file.  Error code is ',vnSocket)
	Calculate vbDownloadOK as kFalse     ;; We bombed out.
End If
;  - Tidy up.
Do voFile.$closefile     ;; Finished downloading the temporary file
If vbDownloadOK     ;; We got it OK.
	Do FileOps.$setunixpermissions(psFilePath,'-rwxrwxrwx')
	Do FileOps.$deletefile(psFilePath)     ;; Make sure there isn't one there already
	Do FileOps.$rename(vsTempPath,psFilePath)     ;; Rename it back to what it should be.
	Calculate vsResult as ''     ;; No errors.
Else     ;; Didn't download OK, delete the temp file
	Calculate vsResult as pick(vsResult='',vsResult,'An error occurred downloading the file')     ;; A generic error
	Do FileOps.$setunixpermissions(vsTempPath,'-rwxrwxrwx')
	Do FileOps.$deletefile(vsTempPath)
End If
Quit method vsResult
;  --- history



> On 7 Nov 2022, at 6:24 am, TBS <andyh at totallybrilliant.com> wrote:
> 
> Folks
> 
> Trying to extract a binary file from an HTTP response from an Omnis method HTTP call…..the call (when made in a browser) simply spits back an excel file, and that’s what I am trying to get back here !
> 
> I use :
> 
> HTTPGet (lvHost,lvURI) Returns SocketNum
> 
> To initiate the connection and get my socket, all good.
> 
> Then I have 2 calls going on (more just for testing here !) :
> 
> HTTPPage (URL) Returns PageContent
> And
> HTTPRead (SocketNum,lvBuffer) Returns lvByteCount
> 
> In the first line - that issues the call that I want to make to the server and I get a text based response in PageContent that starts with :
> 
> HTTP/1.1 200 OK
> Date: Sun, 06 Nov 2022 22:07:00 GMT
> Server: WSGIServer/0.2 CPython/3.9.7
> Content-Type: application/vnd.ms-excel
> Content-Disposition: attachment;filename=DetailedWhereabouts_06-Nov-2022.xlsx
> X-Content-Type-Options: nosniff
> Referrer-Policy: same-origin
> X-Frame-Options: DENY
> Content-Length: 6445
> Vary: Cookie
> 
> 
> And then it appears to have the attachment - presumably as a binary  but of course in text its gobbledy gook - not sure if I can extract the binary from the text here ?
> 
> The second one (HTTPRead) returns a binary field (lvBuffer) that ‘looks’ like it may have the binary file I want plus a little bit more (I am suspecting it is actually the same as the HTTPPage response but as a binary field ??)
> 
> My guess is that the actual binary file that I am interested in follows a known binary string in the binary field returned (like maybe after headerEndString - chartoutf8(chr(13,10,13,10)) ) - but this is where my knowledge comes to an end ! Does anyone have knowledge of the make up of HTTPRead binary ?? i.e. if I can use something like :
> 
> If binsearch(headerEndString,lvBuffer)>0
> Calculate lvBuffer as bytemid(lvBuffer,binsearch(headerEndString,lvBuffer)+4,binlength(lvBuffer))
> End If
> 
> To get my actual binary file being returned…..
> 
> I’m good when I get it - just need to get it :)
> Any and all help appreciated…..
> 
> Andy Hilton
> Totally Brilliant Software Inc
> Phone (US) : (863) 409 4870
> Phone (UK) : 0207 193 8582
> Web : www.totallybrilliant.com
> Helpdesk : http://totallybrilliant.kayako.com
> Email : andyh at totallybrilliant.com
> _____________________________________________________________
> Manage your list subscriptions at https://lists.omnis-dev.com
> Start a new message -> mailto:omnisdev-en at lists.omnis-dev.com 




  <https://www.logicaldevelopments.com.au/>	Paul W Mulroney
Logical Developments
Customised Software Solutions
Ph: 08 9458 3889
 	We Don't Do Simple Pty Ltd
trading as
Logical Developments
ACN 161 009 374
 	  <https://www.facebook.com/logicaldevelopmentswa/>   <https://twitter.com/WeDontDoSimple>   <https://www.linkedin.com/company/logicaldevelopments/>   <https://www.youtube.com/channel/UCPldVVgWR05WX3cVrR5WUQw>

More information about the omnisdev-en mailing list