Programatically download the latest 64 bit version of Firefox for Windows.

[AIR]

Script BASIC

Note: Doesn't meet the challenge requirement for displaying download status live. I'm not sure the user would have time to read it anyways.

How about something like this?


riveraa@nas:~/src$ ./sb64-x86_64.AppImage getFirefox.sb
Downloading Latest 64Bit Firefox (63.0) for Windows.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 42.3M  100 42.3M    0     0  71.2M      0 --:--:-- --:--:-- --:--:-- 81.3M

Firefox_Setup-63.0.exe Downloaded 44,400,072 Bytes at 74,754,643 Bytes/Second.



Add the following to your code before the final CURL::PERFORM:

Code: ScriptBasic

1.     curl::option(ch,"NOPROGRESS",0)

AIR.

[John]

Too cool!

The Script BASIC submission post has been updated.

I had this cURL progress display working in a example of long ago. I just couldn't remember how I did it.

Nice job AIR. This is what sets you apart from everyone else. Your curiosity, experience and being a generous contributor.

FWIW: I'm wondering if what curl::info(ch,"SIZE_DOWNLOAD") is the accumulation of all bytes downloaded since the INIT of the cURL library.

[John]

I download 'Firefox Setup 63.0.exe' directly from the web site and it matched (size) to what SB / cURL is saying for the download size. This solves the question I had about cURL accumulating its downloaded byte count giving a wrong value. It seems it's the cURL progress function not giving acurate file size info.

--compressed
(HTTP) Request a compressed response using one of the algorithms curl supports, and save the uncompressed document.  If this option is used and  the  server  sends  an unsupported encoding, curl will report an error.

I wonder If the progress function is seeing the compressed request byte count / rate.

[AIR]

I download 'Firefox Setup 63.0.exe' directly from the web site and it matched (size) to what SB / cURL is saying for the download size. This solves the question I had about cURL accumulating its downloaded byte count giving a wrong value. It seems it's the cURL progress function not giving acurate file size info.

I believe it's a timing issue.  The download completes during the callback cycle, and the value may not be updated.

--compressed
(HTTP) Request a compressed response using one of the algorithms curl supports, and save the uncompressed document.  If this option is used and  the  server  sends  an unsupported encoding, curl will report an error.

I wonder If the progress function is seeing the compressed request byte count / rate.

You have to actually enable that option (CURLOPT_ACCEPT_ENCODING), does SB have that enabled in the Module by default?

AIR.

[John]

It doesn't look like it based on interface.c for the cURL extension module.

To aid applications not having to bother about what specific algorithms this particular libcurl build supports, libcurl allows a zero-length string to be set ("") to ask for an Accept-Encoding: header to be used that contains all built-in supported encodings.

Seems nothing equals everything.

[John]

Tomaaz, I know you're out there!!!  Post it on RetroB if you want, I'm interested in seeing how you do this!

Tomaaz was a member here but deleted his own account. I think you are wasting your time on him. His only forum interest is tormenting Aurel and bitching about everything.

[John]

My first thought with this challenge was to use SB sockets to connect to get the Firefox download page. That was abruptly halted by Cloudfare. Not like the good old HTTP days.

[AIR]

I think you need to send a proper user agent , as well as add SSL support to Sockets.

Bacon has the same issue with it's built in Socket support although Peter coded something that incorporates what's needed as a proof of concept.

I think that most of the older BASICS will have this problem as well.  I know that MBC does, but I'm not gonna fix that since it's essentially dead now....

AIR.

[John]

I wonder if James Fuller is still doing his version of BCX?

[John]

I think it would be possible to use wget with a SB SYSTEM or EXECUTE to get the Firefox download page and do the file download with its built in progress display.

The whole challenge could be done with bash, wget and regex.

[AIR]

I already did a Bash version for work.....LOL.

Edit:

I originally wrote this to download the latest version for macOS; with a minor tweak it works for windows.

There was much more to this for the Mac, which I've stripped out.

Code: Bash

1. #!/bin/bash
2.  
3. # ff.sh
4. # Script to download the latest version
5. # of Firefox 64bit for Windows
6. #
7. # Written by Armando I. Rivera (AIR)
8. # Timeinc Global Technology Services
9.  
10. XPATH="string(//div[@id='outer-wrapper']/main/div[@id='other-platforms']/div[@class='content']/section[@class='section-other-platforms']/div[@id='download-platform-list-release']/ul[@class='download-platform-list recommended']/li[@class='os_win64']/a/@href)"
11.  
12. VERSION=$(curl -s https://www.mozilla.org/en-US/firefox/new/ | xmllint --html --xpath 'string(/html/@data-latest-firefox)' - 2>/dev/null)
13.  
14. curl -s https://www.mozilla.org/en-US/firefox/new/ | xmllint --html --xpath "$XPATH" - 2>/dev/null | xargs wget -O "Firefox Setup $VERSION.exe" -q --show-progress
15.  

[John]

I love it when there is more comment than code. 

[AIR]

I think XPATH can be replaced with:

Code: Bash

1. XPATH="string(//div[@id='outer-wrapper']/*/*/*/*/*/*/li[@class='os_win64']/a/@href)"

 

AIR.

[John]

It's amazing what can be done with Bash. Peter has a version of BaCon written in it.

[AIR]

Speaking of Peter/Bacon, here's my BACON Version

I've included a zip file with the updated Curl module, as well as a custom REGEX module that I wrote just for this.

Bacon's native REGEX function doesn't return any matches as strings/text.  You have to use the return value from that function, which is where in the file the hit begins, and use the REGLEN builtin to get the length.  Then you need to feed the source, the REGEX return value, and the REGLEN variable to MID$ to retrieve the actual hit.

My function (called ReGex) returns the actual text.  Like Peter's implementation, it's based on Posix Regex, which has it's own arcane regex syntax.

Interestingly, the regex that I feed to my function doesn't work with Peter's; I haven't looked into why that is since I don't think Peter would want to modify his implementation much (if at all) since a few other Bacon functions seem to rely on it.

Anyway, here's the Bacon source (again, the required modules are in the attached zip file)

Code: Text

1. TRAP LOCAL
2. INCLUDE "curl.bac"
3. INCLUDE "Regex-Plus"
4.  
5.  
6. ' We store our data in this variable
7. DECLARE data$, version_num$
8. DECLARE regx TYPE mREGEX_type
9.  
10. FUNCTION progress_callback(long clientp, double dltotal, double dlnow, double ultotal, double ulnow )
11.     LOCAL totaldotz, dotz, ii TYPE int
12.     LOCAL fractiondownloaded TYPE double
13.     LOCAL tmpStr$
14.  
15.     IF dltotal <= 0.0 THEN RETURN 0
16.     tmpStr$ = CONCAT$((STRING)clientp,"")
17.    
18.     totaldotz = 40
19.     fractiondownloaded = dlnow/dltotal
20.     dotz = ROUND(fractiondownloaded * totaldotz)
21.  
22.     PRINT fractiondownloaded * 100 FORMAT "%3.0f%% [";
23.  
24.     WHILE ii < dotz DO
25.         PRINT "=";
26.         INCR ii
27.     WEND
28.  
29.     WHILE ii < totaldotz DO
30.         PRINT " ";
31.         INCR ii
32.     WEND
33.  
34.     PRINT "] \r";
35.  
36.     RETURN 0
37. END FUNCTION
38.  
39. FUNCTION Save_Data(STRING buffer$, size_t size, size_t nmemb, void *userp)
40.     data$ = CONCAT$(data$, buffer$)
41.     RETURN size*nmemb
42. END FUNCTION
43.  
44. SUB getData$(STRING url) TYPE STRING
45.     LOCAL handle TYPE long
46.     LOCAL success
47.  
48.     handle = curl_easy_init()
49.     curl_easy_setopt(handle, CURLOPT_URL, url)
50.     curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 1)
51.     curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, Save_Data)
52.    
53.     success = curl_easy_perform(handle)
54.     curl_easy_cleanup(handle)
55. END SUB
56.  
57. SUB downloadFile(STRING url, STRING downloadFileName)
58.     LOCAL handle TYPE long
59.     LOCAL success
60.  
61.     PRINT "Downloading Latest 64Bit Firefox (",version_num$,") for Windows.\n"
62.    
63.     handle = curl_easy_init()    
64.     OPEN downloadFileName FOR WRITING AS download
65.     curl_easy_setopt(handle, CURLOPT_URL, url)
66.     curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION,1)
67.     curl_easy_setopt(handle, CURLOPT_WRITEDATA, download)
68.     curl_easy_setopt(handle, CURLOPT_NOPROGRESS,0)
69.     curl_easy_setopt(handle, CURLOPT_PROGRESSFUNCTION,progress_callback)
70.  
71.     success = curl_easy_perform(handle)
72.     CLOSE FILE download
73.     curl_easy_cleanup(handle)
74.     PRINT "\n\nDownload Complete.\n"
75. END SUB
76.  
77. CLEAR
78.  
79. PRINT "Firefox Download Challenge (Bacon Version) by AIR.\n"
80.  
81. getData$("https://www.mozilla.org/en-US/firefox/new")
82.  
83. regx = RegEx(data$,"data-latest-firefox=.([0-9]+\\.[0-9]+)")
84.  
85. IF regx.count THEN
86.     version_num$ = regx.result$[0]
87.     PRINT version_num$ FORMAT "Firefox Setup %s.exe" TO version$
88. END IF
89.  
90. regx = RegEx(data$,".+href=\"(http.+latest-ssl&amp;os=win64&amp;lang=en-US)\"")
91.  
92. IF regx.count THEN
93.     downloadLink$ = regx.result$[0]
94. END IF
95.  
96. downloadFile(downloadLink$, version$)
97. PRINT
98.  
99.  

AIR.

P.S.  Also like Peter's function, my RegEx function only returns a single (the first) hit.  I'll see about tweaking it so it will return multiple hits like most are used to seeing when using a library like PCRE...