AllBASIC Forum

BASIC User Group => Code Challenges => Topic started by: AIR on October 26, 2018, 03:50:51 PM

Title: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 26, 2018, 03:50:51 PM

New Day, New Code Challenge.

Download the latest 64 bit version of Firefox for Windows.

Requirements:

• Connect to "https://www.mozilla.org/en-US/firefox/new"
• Parse the html to extract the current version number
• Parse the html to extract the download link to the FULL version (there is a "stub" version, we don't want that)
• Clear the screen
• Show a message with the version of Firefox being downloaded.
• Show the download progress as it's happening
• Show a complete message.

The progress should be displayed in the following format:

Quote

    Downloading Latest Firefox (version number goes here)
    Downloaded 34140000 of 44400072 at 11784kb/s

    That is current data downloaded in real time, the total size, and the internet speed each in kb.

    The download progress should display on a single line, updating that line as the download progresses.


I'll be posting a submission using Nim this evening and leave the other languages to whoever wants to submit something....

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 26, 2018, 03:59:57 PM

This look like a natural for SB and cURL.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 26, 2018, 06:14:38 PM

NIM Submission, works on macOS and Linux (on a Raspberry Pi)

to compile:  nim c -d:ssl getFirefox.nim

Code: [Select]

Firefox Download Challenge (Nim Version) by AIR.

Downloading Latest 64Bit Firefox (63.0) for Windows.

Downloaded 42.343MiB of 42.343MiB

Download Complete.

Code: Text

1. import httpclient, htmlparser,xmlparser, xmltree, streams, ospaths, asyncdispatch, strutils
2.  
3. const
4.     ClrLine = "\x1b[0K\r"
5. var
6.     totalsize:BiggestInt
7.  
8. proc ClrScr() =
9.     stdout.write("\x1b[2J\x1b[H")
10.    
11. proc showProgress(total, progress, speed: BiggestInt) {.async.} =
12.     totalsize = total
13.  
14.     stdout.write(ClrLine, "Downloading ", progress.formatSize, " of ", total.formatSize, " at ", (speed div 1000).formatSize(','))
15.  
16.     flushFile(stdout)
17.  
18. proc download(url,filename:string) {.async.} =
19.     var client = newAsyncHttpClient()
20.  
21.     client.onProgressChanged = showProgress
22.  
23.     await client.downloadFile(url, filename)
24.  
25.     stdout.write(ClrLine, "Downloaded ", totalsize.formatSize, " of ", totalsize.formatSize," ".repeat(20))
26.  
27.     flushFile(stdout)
28.  
29.     echo "\n\nDownload Complete.\n"
30.  
31. proc main() =
32.     var
33.         client = newHttpClient()
34.         src = client.getContent("https://www.mozilla.org/en-US/firefox/new")
35.         c = parseHtml( newStringStream(src) )
36.         url: string
37.         version: string
38.  
39.  
40.     for html in c.findAll("html"):
41.         version = html.attr("data-latest-firefox")
42.         break
43.  
44.     for d in c.findAll("li"):
45.         if d.attr("class") == "os_win64":
46.             for e in d.findAll("a"):
47.                 if e.attr("class") == "download-link":
48.                     url = e.attr("href")
49.                     break
50.  
51.     ClrScr()
52.  
53.     echo "Firefox Download Challenge (Nim Version) by AIR.\n"
54.     echo "Downloading Latest 64Bit Firefox (",version,") for Windows.\n"
55.  
56.     waitFor url.download("Firefox Setup " & version & ".exe")
57.  
58. main()
59.  

This would have been so much simpler/clearer if Nim supported xpath syntax, but there ya go....


AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 01:03:53 AM

Here is my first pass at the challenge. I can download the file fine but the cURL progress meter feature doesn't seem to be working for me yet. I may have to resort to the static INFO funtions to build your download status string.

@AIR - When you said ignore the stub URL, did you mean the download file URL it returns or did I get the right link?

Code: ScriptBasic

1. INCLUDE curl.bas
2.  
3. ch = curl::init()
4. curl::option(ch,"URL","https://www.mozilla.org/en-US/firefox/new/")
5. wp = curl::perform(ch)
6.  
7. IF wp LIKE "*data-latest-firefox=\"*\" data-esr-versions*" THEN PRINT JOKER(2),"\n"
8.  
9. IF wp LIKE """*<div id="other-platforms">*<li class="os_win64">*<a href="*"*""" THEN
10.   curl::option(ch,"URL", JOKER(4))
11. END IF
12.  
13. dl_html = curl::perform(ch)
14. IF dl_html LIKE """*href="*"*""" THEN dl_file = JOKER(2)
15.  
16. curl::option(ch,"URL", dl_file)
17. curl::option(ch,"FILE","Firefox_Win64.exe")
18. curl::perform(ch)
19.  
20. curl::finish(ch)
21.  

jrs@jrs-laptop:~/sb/abcc$ time scriba ffdl.sb
63.0

real   0m2.422s
user   0m1.108s
sys   0m0.291s

jrs@jrs-laptop:~/sb/abcc$ ls -l Firefox_Win64.exe
-rw-r--r-- 1 jrs jrs 44400072 Oct 27 01:02 Firefox_Win64.exe
jrs@jrs-laptop:~/sb/abcc$

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 27, 2018, 10:21:53 AM

Quote from: John on October 27, 2018, 01:03:53 AM

@AIR - When you said ignore the stub URL, did you mean the download file URL it returns or did I get the right link?

The stub version is a small program you download that will then download and install Firefox.

What we're after is the full installer, which once you have it doesn't require an internet connection for Firefox to be installed on a system.

Quote from: John

I may have to resort to the static INFO funtions to build your download status string.

I think you're gonna have to update the Curl module, because according to the old module docs CURLOPT_PROGRESSFUNCTION is not implemented:

Quote from: ScriptBasic Documentation

CURLOPT_PROGRESSFUNCTION is used in conjunction with CURLOPT_PROGRESSDATA to specify a progress function that CURLIB library calls from time to time to allow progress indication for the user. This is not implemented in the ScriptBasic interface.

Which means that you won't get real time stats while the download is happening with the Curl module as it currently exists...


AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 10:45:23 AM

Based on your feedback, I have the right installer file and I'm stuck with after the fact download stats without enhancing the libcurl extension module.

On a positive note, a 2.4 second delay before completion and results isn't eternity.

progressfunc.c

Code: C

1. /***************************************************************************
2.  *                                  _   _ ____  _
3.  *  Project                     ___| | | |  _ \| |
4.  *                             / __| | | | |_) | |
5.  *                            | (__| |_| |  _ <| |___
6.  *                             \___|\___/|_| \_\_____|
7.  *
8.  * Copyright (C) 1998 - 2018, Daniel Stenberg, <daniel@haxx.se>, et al.
9.  *
10.  * This software is licensed as described in the file COPYING, which
11.  * you should have received as part of this distribution. The terms
12.  * are also available at https://curl.haxx.se/docs/copyright.html.
13.  *
14.  * You may opt to use, copy, modify, merge, publish, distribute and/or sell
15.  * copies of the Software, and permit persons to whom the Software is
16.  * furnished to do so, under the terms of the COPYING file.
17.  *
18.  * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
19.  * KIND, either express or implied.
20.  *
21.  ***************************************************************************/
22. /* <DESC>
23.  * Use the progress callbacks, old and/or new one depending on available
24.  * libcurl version.
25.  * </DESC>
26.  */
27. #include <stdio.h>
28. #include <curl/curl.h>
29.  
30. #if LIBCURL_VERSION_NUM >= 0x073d00
31. /* In libcurl 7.61.0, support was added for extracting the time in plain
32.    microseconds. Older libcurl versions are stuck in using 'double' for this
33.    information so we complicate this example a bit by supporting either
34.    approach. */
35. #define TIME_IN_US 1  
36. #define TIMETYPE curl_off_t
37. #define TIMEOPT CURLINFO_TOTAL_TIME_T
38. #define MINIMAL_PROGRESS_FUNCTIONALITY_INTERVAL     3000000
39. #else
40. #define TIMETYPE double
41. #define TIMEOPT CURLINFO_TOTAL_TIME
42. #define MINIMAL_PROGRESS_FUNCTIONALITY_INTERVAL     3
43. #endif
44.  
45. #define STOP_DOWNLOAD_AFTER_THIS_MANY_BYTES         6000
46.  
47. struct myprogress {
48.   TIMETYPE lastruntime; /* type depends on version, see above */
49.   CURL *curl;
50. };
51.  
52. /* this is how the CURLOPT_XFERINFOFUNCTION callback works */
53. static int xferinfo(void *p,
54.                     curl_off_t dltotal, curl_off_t dlnow,
55.                     curl_off_t ultotal, curl_off_t ulnow)
56. {
57.   struct myprogress *myp = (struct myprogress *)p;
58.   CURL *curl = myp->curl;
59.   TIMETYPE curtime = 0;
60.  
61.   curl_easy_getinfo(curl, TIMEOPT, &curtime);
62.  
63.   /* under certain circumstances it may be desirable for certain functionality
64.      to only run every N seconds, in order to do this the transaction time can
65.      be used */
66.   if((curtime - myp->lastruntime) >= MINIMAL_PROGRESS_FUNCTIONALITY_INTERVAL) {
67.     myp->lastruntime = curtime;
68. #ifdef TIME_IN_US
69.     fprintf(stderr, "TOTAL TIME: %" CURL_FORMAT_CURL_OFF_T ".%06ld\r\n",
70.             (curtime / 1000000), (long)(curtime % 1000000));
71. #else
72.     fprintf(stderr, "TOTAL TIME: %f \r\n", curtime);
73. #endif
74.   }
75.  
76.   fprintf(stderr, "UP: %" CURL_FORMAT_CURL_OFF_T " of %" CURL_FORMAT_CURL_OFF_T
77.           "  DOWN: %" CURL_FORMAT_CURL_OFF_T " of %" CURL_FORMAT_CURL_OFF_T
78.           "\r\n",
79.           ulnow, ultotal, dlnow, dltotal);
80.  
81.   if(dlnow > STOP_DOWNLOAD_AFTER_THIS_MANY_BYTES)
82.     return 1;
83.   return 0;
84. }
85.  
86. #if LIBCURL_VERSION_NUM < 0x072000
87. /* for libcurl older than 7.32.0 (CURLOPT_PROGRESSFUNCTION) */
88. static int older_progress(void *p,
89.                           double dltotal, double dlnow,
90.                           double ultotal, double ulnow)
91. {
92.   return xferinfo(p,
93.                   (curl_off_t)dltotal,
94.                   (curl_off_t)dlnow,
95.                   (curl_off_t)ultotal,
96.                   (curl_off_t)ulnow);
97. }
98. #endif
99.  
100. int main(void)
101. {
102.   CURL *curl;
103.   CURLcode res = CURLE_OK;
104.   struct myprogress prog;
105.  
106.   curl = curl_easy_init();
107.   if(curl) {
108.     prog.lastruntime = 0;
109.     prog.curl = curl;
110.  
111.     curl_easy_setopt(curl, CURLOPT_URL, "https://example.com/");
112.  
113. #if LIBCURL_VERSION_NUM >= 0x072000
114.     /* xferinfo was introduced in 7.32.0, no earlier libcurl versions will
115.        compile as they won't have the symbols around.
116.  
117.        If built with a newer libcurl, but running with an older libcurl:
118.        curl_easy_setopt() will fail in run-time trying to set the new
119.        callback, making the older callback get used.
120.  
121.        New libcurls will prefer the new callback and instead use that one even
122.        if both callbacks are set. */
123.  
124.     curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, xferinfo);
125.     /* pass the struct pointer into the xferinfo function, note that this is
126.        an alias to CURLOPT_PROGRESSDATA */
127.     curl_easy_setopt(curl, CURLOPT_XFERINFODATA, &prog);
128. #else
129.     curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, older_progress);
130.     /* pass the struct pointer into the progress function */
131.     curl_easy_setopt(curl, CURLOPT_PROGRESSDATA, &prog);
132. #endif
133.  
134.     curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
135.     res = curl_easy_perform(curl);
136.  
137.     if(res != CURLE_OK)
138.       fprintf(stderr, "%s\n", curl_easy_strerror(res));
139.  
140.     /* always cleanup */
141.     curl_easy_cleanup(curl);
142.   }
143.   return (int)res;
144. }
145.  

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 12:57:52 PM

Script BASIC

Code: ScriptBasic

1. ' Firefox Download Challenge (Script BASIC Version) by JRS.
2.  
3. INCLUDE curl.bas
4.  
5. ch = curl::init()
6. curl::option(ch,"URL","https://www.mozilla.org/en-US/firefox/new/")
7. wp = curl::perform(ch)
8.  
9. IF wp LIKE "*data-latest-firefox=\"*\" data-esr-versions*" THEN
10.   version = JOKER(2)
11.   PRINT "Downloading Latest 64Bit Firefox (",version,") for Windows.\n"
12. END IF
13.  
14. IF wp LIKE """*<div id="other-platforms">*<li class="os_win64">*<a href="*"*""" THEN
15.   curl::option(ch,"URL", JOKER(4))
16. END IF
17.  
18. dl_html = curl::perform(ch)
19. IF dl_html LIKE """*href="*"*""" THEN dl_file = JOKER(2)
20.  
21. curl::option(ch,"URL", dl_file)
22. curl::option(ch,"FILE","Firefox_Setup-" & version & ".exe")
23. curl::option(ch,"NOPROGRESS",0)
24. curl::perform(ch)
25.  
26. PRINTNL
27. PRINT "Firefox_Setup-" & version & ".exe Downloaded ",FORMAT("%~##,###,###~ Bytes",curl::info(ch,"SIZE_DOWNLOAD")), _
28.   " at ",FORMAT("%~##,###,###~ Bytes/Second",curl::info(ch,"SPEED_DOWNLOAD")),".\n"
29.  
30. curl::finish(ch)
31.  

jrs@jrs-laptop:~/sb/abcc$ time scriba ffdl.sb
Downloading Latest 64Bit Firefox (63.0) for Windows.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 42.3M  100 42.3M    0     0  24.9M      0  0:00:01  0:00:01 --:--:-- 27.7M

Firefox_Setup-63.0.exe Downloaded 44,400,072 Bytes at 26,133,061 Bytes/Second.

real   0m2.863s
user   0m1.213s
sys   0m0.276s
jrs@jrs-laptop:~/sb/abcc$

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 01:53:57 PM

Curious how big that Nim executable is with all those dependency requirements?

Code: [Select]

import httpclient, htmlparser,xmlparser, xmltree, streams, ospaths, asyncdispatch, strutils


Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 27, 2018, 03:53:21 PM

That's because the core Nim compiler is designed to be fairly lean.

Similar to SB/Python/Ruby/FreePascal/Delphi/Xojo(RealBasic)/Powershell/Lua/ZShell/C#, it has a concept of a "Modules" system whereby additional functionality can be added to a given programming language without having it all baked in, or having to recompile the core to add that functionality.

Not having all functionality baked in also allows one to alter a given module in order to achieve a desired result.  SB's Curl module is a perfect example, where it doesn't currently support Curl Callbacks, but that feature can be added without messing with any of the core stuff.

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 27, 2018, 04:12:57 PM

I was curious how difficult this would be in POWERSHELL, so here's my implementation:

Code: PowerShell

1. Import-Module BitsTransfer
2.  
3. Clear-Host
4.  
5. echo "Firefox Download Challenge (Powershell Version) by AIR.`r`n"
6.  
7. $progressPreference = 'silentlyContinue'
8. $result = Invoke-WebRequest "https://www.mozilla.org/en-US/firefox/new/"
9. $progressPreference = 'Continue'
10.  
11. $FF_Version = ($result | %{ $_ -match 'data-latest-firefox="(.+?)"'}) | %{$Matches[1]}
12.  
13. echo "Downloading Latest 64Bit Firefox ($FF_Version) for Windows.`r`n"
14.  
15. $HTML = $result.ParsedHtml.getElementsByTagName('li')
16.  
17. $hits = $HTML | where {$_.outerHTML -match "class=os_win64"}
18.  
19. $download_link = ($hits | where {$_.outerTEXT -match 'Windows 64-bit'}).innerHTML | %{[regex]::matches($_,'(?<=\").+?(?=\")')[0].value}
20.  
21. Start-BitsTransfer -Source $download_link -Destination "Firefox Setup $FF_VERSION.exe"
22.  
23. echo "Download Complete.`r`n"
24.  
25.  

Note that the "BitsTransfer" module provides a console-based progress bar, so technically this doesn't meet the full requirements of the challenge.  But it was fun to code!

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 05:43:26 PM

Are you able to provide execution times for the submissions? Seeing the output would also be nice.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 27, 2018, 06:52:03 PM

Execution times in this case would be very dependent on your Internet speeds.

For example, my speed is rated at 960+mb/s (roughly 120 MB/s), so the download happens in about a half second for me.  That's a wired connection, WiFi would always be slower.

Also, it would be dependent on how your connection to the website is routed.  I might go through fast routers, and you might go through one that is having packet issues resulting in retries/performance loss.

Besides, this challenge isn't about how FAST you can do it, but HOW you would do it....that's going to be the case with any challenges I throw out there... 8)

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 06:59:36 PM

Maybe posting a poll at the end of a challenge so those lurking can vote on what they think is the best solution.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 27, 2018, 09:24:34 PM

Firefox Download Challenge (NIM Version) by AIR

(https://binarymagic.net/nim_Challenge.gif)

Updated code, removed XML-based parsing and use REGEX instead...

Code: Text

1. import httpclient,re,asyncdispatch,strutils
2.  
3.  
4.  
5. const
6.     ClrLine = "\x1b[0K\r"
7. var
8.     totalsize:BiggestInt
9.  
10. proc ClrScr() =
11.     stdout.write("\x1b[2J\x1b[H")
12.  
13. proc showProgress(total, progress, speed: BiggestInt) {.async.} =
14.     totalsize = total
15.     stdout.write(ClrLine, "Downloading ", progress.formatSize, " of ", total.formatSize, " at ", (speed div 1000).formatSize(','))
16.     flushFile(stdout)
17.  
18. proc download(url,filename:string) {.async.} =
19.     var client = newAsyncHttpClient()
20.     client.onProgressChanged = showProgress
21.     await client.downloadFile(url, filename)
22.     stdout.write(ClrLine, "Downloaded ", totalsize.formatSize, " of ", totalsize.formatSize," ".repeat(20))
23.     flushFile(stdout)
24.     echo "\n\nDownload Complete.\n"
25.    
26. proc main() =
27.     var
28.         client = newHttpClient()
29.         src = client.getContent("https://www.mozilla.org/en-US/firefox/new")
30.         url,version: string
31.         x:int
32.         matches: array[2,string]
33.  
34.     x = src.find(re"data-latest-firefox=.(\d+.\d+).",matches)
35.     version = matches[0]
36.  
37.     x = src.find(re"(http.+product=firefox-latest-ssl.+os=win64.+en-US)",matches)
38.     url = matches[0]
39.  
40.     ClrScr()
41.     echo "Firefox Download Challenge (Nim Version) by AIR.\n"
42.     echo "Downloading Latest 64Bit Firefox (",version,") for Windows.\n"
43.  
44.     waitFor url.download("Firefox Setup " & version & ".exe")
45.  
46.  
47. main()
48.  

C'mon , there's gotta be SOMEONE who would like to give this a try in the language of choice!!  (Tomaaz, I know you're out there!!!  Post it on RetroB if you want, I'm interested in seeing how you do this!)

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 28, 2018, 12:34:41 AM

I bet a code challenge to make REGEX emulate LIKE would be fun. I have avoided regex like the plague. It's time I learn this commonly used function. (so f*cking cryptic)

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 28, 2018, 11:59:42 AM

Quote from: John on October 27, 2018, 12:57:52 PM

Script BASIC

Note: Doesn't meet the challenge requirement for displaying download status live. I'm not sure the user would have time to read it anyways.

How about something like this?


riveraa@nas:~/src$ ./sb64-x86_64.AppImage getFirefox.sb
Downloading Latest 64Bit Firefox (63.0) for Windows.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 42.3M  100 42.3M    0     0  71.2M      0 --:--:-- --:--:-- --:--:-- 81.3M

Firefox_Setup-63.0.exe Downloaded 44,400,072 Bytes at 74,754,643 Bytes/Second.



Add the following to your code before the final CURL::PERFORM:

Code: ScriptBasic

1.     curl::option(ch,"NOPROGRESS",0)

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 28, 2018, 12:09:08 PM

Too cool!

The Script BASIC submission post has been updated.

I had this cURL progress display working in a example of long ago. I just couldn't remember how I did it.

Nice job AIR. This is what sets you apart from everyone else. Your curiosity, experience and being a generous contributor.

FWIW: I'm wondering if what curl::info(ch,"SIZE_DOWNLOAD") is the accumulation of all bytes downloaded since the INIT of the cURL library.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 28, 2018, 05:52:38 PM

I download 'Firefox Setup 63.0.exe' directly from the web site and it matched (size) to what SB / cURL is saying for the download size. This solves the question I had about cURL accumulating its downloaded byte count giving a wrong value. It seems it's the cURL progress function not giving acurate file size info.

Quote

--compressed
(HTTP) Request a compressed response using one of the algorithms curl supports, and save the uncompressed document.  If this option is used and  the  server  sends  an unsupported encoding, curl will report an error.

I wonder If the progress function is seeing the compressed request byte count / rate.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 28, 2018, 09:21:05 PM

Quote from: John on October 28, 2018, 05:52:38 PM

I download 'Firefox Setup 63.0.exe' directly from the web site and it matched (size) to what SB / cURL is saying for the download size. This solves the question I had about cURL accumulating its downloaded byte count giving a wrong value. It seems it's the cURL progress function not giving acurate file size info.

I believe it's a timing issue.  The download completes during the callback cycle, and the value may not be updated.

Quote

--compressed
(HTTP) Request a compressed response using one of the algorithms curl supports, and save the uncompressed document.  If this option is used and  the  server  sends  an unsupported encoding, curl will report an error.

I wonder If the progress function is seeing the compressed request byte count / rate.

You have to actually enable that option (CURLOPT_ACCEPT_ENCODING), does SB have that enabled in the Module by default?

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 28, 2018, 09:32:59 PM

It doesn't look like it based on interface.c for the cURL extension module.

Quote

To aid applications not having to bother about what specific algorithms this particular libcurl build supports, libcurl allows a zero-length string to be set ("") to ask for an Accept-Encoding: header to be used that contains all built-in supported encodings.

Seems nothing equals everything.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 29, 2018, 03:44:20 PM

Quote from: AIR

Tomaaz, I know you're out there!!!  Post it on RetroB if you want, I'm interested in seeing how you do this!

Tomaaz was a member here but deleted his own account. I think you are wasting your time on him. His only forum interest is tormenting Aurel and bitching about everything.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 29, 2018, 04:01:05 PM

My first thought with this challenge was to use SB sockets to connect to get the Firefox download page. That was abruptly halted by Cloudfare. Not like the good old HTTP days.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 29, 2018, 06:24:07 PM

I think you need to send a proper user agent , as well as add SSL support to Sockets.

Bacon has the same issue with it's built in Socket support although Peter coded something that incorporates what's needed as a proof of concept.

I think that most of the older BASICS will have this problem as well.  I know that MBC does, but I'm not gonna fix that since it's essentially dead now....

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 29, 2018, 06:30:07 PM

I wonder if James Fuller is still doing his version of BCX?

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 29, 2018, 07:30:02 PM

I think it would be possible to use wget with a SB SYSTEM or EXECUTE to get the Firefox download page and do the file download with its built in progress display.

The whole challenge could be done with bash, wget and regex.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 29, 2018, 08:24:38 PM

I already did a Bash version for work.....LOL.

Edit:

I originally wrote this to download the latest version for macOS; with a minor tweak it works for windows.

There was much more to this for the Mac, which I've stripped out.

Code: Bash

1. #!/bin/bash
2.  
3. # ff.sh
4. # Script to download the latest version
5. # of Firefox 64bit for Windows
6. #
7. # Written by Armando I. Rivera (AIR)
8. # Timeinc Global Technology Services
9.  
10. XPATH="string(//div[@id='outer-wrapper']/main/div[@id='other-platforms']/div[@class='content']/section[@class='section-other-platforms']/div[@id='download-platform-list-release']/ul[@class='download-platform-list recommended']/li[@class='os_win64']/a/@href)"
11.  
12. VERSION=$(curl -s https://www.mozilla.org/en-US/firefox/new/ | xmllint --html --xpath 'string(/html/@data-latest-firefox)' - 2>/dev/null)
13.  
14. curl -s https://www.mozilla.org/en-US/firefox/new/ | xmllint --html --xpath "$XPATH" - 2>/dev/null | xargs wget -O "Firefox Setup $VERSION.exe" -q --show-progress
15.  

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 29, 2018, 09:56:55 PM

I love it when there is more comment than code.  8)

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 30, 2018, 07:06:37 AM

I think XPATH can be replaced with:

Code: Bash

1. XPATH="string(//div[@id='outer-wrapper']/*/*/*/*/*/*/li[@class='os_win64']/a/@href)"

 ;D ;D 8)

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 30, 2018, 09:02:00 AM

It's amazing what can be done with Bash. Peter has a version of BaCon written in it. 8)

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 30, 2018, 12:46:32 PM

Speaking of Peter/Bacon, here's my BACON Version

I've included a zip file with the updated Curl module, as well as a custom REGEX module that I wrote just for this.

Bacon's native REGEX function doesn't return any matches as strings/text.  You have to use the return value from that function, which is where in the file the hit begins, and use the REGLEN builtin to get the length.  Then you need to feed the source, the REGEX return value, and the REGLEN variable to MID$ to retrieve the actual hit.

My function (called ReGex) returns the actual text.  Like Peter's implementation, it's based on Posix Regex, which has it's own arcane regex syntax.

Interestingly, the regex that I feed to my function doesn't work with Peter's; I haven't looked into why that is since I don't think Peter would want to modify his implementation much (if at all) since a few other Bacon functions seem to rely on it.

Anyway, here's the Bacon source (again, the required modules are in the attached zip file)

Code: Text

1. TRAP LOCAL
2. INCLUDE "curl.bac"
3. INCLUDE "Regex-Plus"
4.  
5.  
6. ' We store our data in this variable
7. DECLARE data$, version_num$
8. DECLARE regx TYPE mREGEX_type
9.  
10. FUNCTION progress_callback(long clientp, double dltotal, double dlnow, double ultotal, double ulnow )
11.     LOCAL totaldotz, dotz, ii TYPE int
12.     LOCAL fractiondownloaded TYPE double
13.     LOCAL tmpStr$
14.  
15.     IF dltotal <= 0.0 THEN RETURN 0
16.     tmpStr$ = CONCAT$((STRING)clientp,"")
17.    
18.     totaldotz = 40
19.     fractiondownloaded = dlnow/dltotal
20.     dotz = ROUND(fractiondownloaded * totaldotz)
21.  
22.     PRINT fractiondownloaded * 100 FORMAT "%3.0f%% [";
23.  
24.     WHILE ii < dotz DO
25.         PRINT "=";
26.         INCR ii
27.     WEND
28.  
29.     WHILE ii < totaldotz DO
30.         PRINT " ";
31.         INCR ii
32.     WEND
33.  
34.     PRINT "] \r";
35.  
36.     RETURN 0
37. END FUNCTION
38.  
39. FUNCTION Save_Data(STRING buffer$, size_t size, size_t nmemb, void *userp)
40.     data$ = CONCAT$(data$, buffer$)
41.     RETURN size*nmemb
42. END FUNCTION
43.  
44. SUB getData$(STRING url) TYPE STRING
45.     LOCAL handle TYPE long
46.     LOCAL success
47.  
48.     handle = curl_easy_init()
49.     curl_easy_setopt(handle, CURLOPT_URL, url)
50.     curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 1)
51.     curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, Save_Data)
52.    
53.     success = curl_easy_perform(handle)
54.     curl_easy_cleanup(handle)
55. END SUB
56.  
57. SUB downloadFile(STRING url, STRING downloadFileName)
58.     LOCAL handle TYPE long
59.     LOCAL success
60.  
61.     PRINT "Downloading Latest 64Bit Firefox (",version_num$,") for Windows.\n"
62.    
63.     handle = curl_easy_init()    
64.     OPEN downloadFileName FOR WRITING AS download
65.     curl_easy_setopt(handle, CURLOPT_URL, url)
66.     curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION,1)
67.     curl_easy_setopt(handle, CURLOPT_WRITEDATA, download)
68.     curl_easy_setopt(handle, CURLOPT_NOPROGRESS,0)
69.     curl_easy_setopt(handle, CURLOPT_PROGRESSFUNCTION,progress_callback)
70.  
71.     success = curl_easy_perform(handle)
72.     CLOSE FILE download
73.     curl_easy_cleanup(handle)
74.     PRINT "\n\nDownload Complete.\n"
75. END SUB
76.  
77. CLEAR
78.  
79. PRINT "Firefox Download Challenge (Bacon Version) by AIR.\n"
80.  
81. getData$("https://www.mozilla.org/en-US/firefox/new")
82.  
83. regx = RegEx(data$,"data-latest-firefox=.([0-9]+\\.[0-9]+)")
84.  
85. IF regx.count THEN
86.     version_num$ = regx.result$[0]
87.     PRINT version_num$ FORMAT "Firefox Setup %s.exe" TO version$
88. END IF
89.  
90. regx = RegEx(data$,".+href=\"(http.+latest-ssl&amp;os=win64&amp;lang=en-US)\"")
91.  
92. IF regx.count THEN
93.     downloadLink$ = regx.result$[0]
94. END IF
95.  
96. downloadFile(downloadLink$, version$)
97. PRINT
98.  
99.  

AIR.

P.S.  Also like Peter's function, my RegEx function only returns a single (the first) hit.  I'll see about tweaking it so it will return multiple hits like most are used to seeing when using a library like PCRE...

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 30, 2018, 10:53:18 PM

BASIC doesn't get any thinner.  ;)

Script BASIC

Code: ScriptBasic

1. ' Firefox Download Challenge (Script BASIC Version) by JRS.
2.  
3. INCLUDE curl.bas
4.  
5. ch = curl::init()
6.  
7. curl::option(ch,"URL","https://www.mozilla.org/en-US/firefox/new/")
8. wp = curl::perform(ch)
9.  
10. IF wp LIKE """*data-latest-firefox="*" data-esr-versions*<div id="other-platforms">*<li class="os_win64">*<a href="*"*""" THEN
11.   PRINT "Downloading Latest 64Bit Firefox (",JOKER(2),") for Windows.\n"
12.   curl::option(ch,"FOLLOWLOCATION", 1)
13.   curl::option(ch,"NOPROGRESS",0)
14.   curl::option(ch,"FILE","Firefox_Setup-" & JOKER(2) & ".exe")
15.   curl::option(ch,"URL", JOKER(6))
16.   curl::perform(ch)
17.   PRINTNL
18.   PRINT "Firefox_Setup-" & JOKER(2) & ".exe Downloaded ",FORMAT("%~##,###,###~ Bytes",curl::info(ch,"SIZE_DOWNLOAD")), _
19.   " at ",FORMAT("%~##,###,###~ Bytes/Second",curl::info(ch,"SPEED_DOWNLOAD")),".\n"
20. ELSE
21.   PRINT "<< ERROR >>\n"
22. END IF
23.  
24. curl::finish(ch)
25.  

$ time scriba ffdl.sb
Downloading Latest 64Bit Firefox (63.0) for Windows.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   130  100   130    0     0    663      0 --:--:-- --:--:-- --:--:--   663
100 42.3M  100 42.3M    0     0  22.7M      0  0:00:01  0:00:01 --:--:-- 35.5M

Firefox_Setup-63.0.exe Downloaded 44,400,072 Bytes at 23,883,847 Bytes/Second.

real   0m2.174s
user   0m1.135s
sys   0m0.271s
$

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: erosolmi on October 30, 2018, 11:10:18 PM

Example in thinBasic.
Attached source code and executable

Ciao
Eros

Code: thinBasic

1. uses "Console"
2. uses "iNet"
3. uses "OS"
4.  
5. printl "Connect to Firefox web site, determine latest Firefox version and download it"
6.  
7. '---Check if computer is connected to internet
8. if INET_GetState Then
9.   '---Get Firefox web site page
10.   string Firefox_Info_Page = INET_UrlGetString("https://www.mozilla.org/en-US/firefox/new/")
11.  
12.   '---Get Firefox available version
13.   string Firefox_Available_Version = grab$(Firefox_Info_Page, "data-latest-firefox=""", """")
14.  
15.   '---Get Win64 direct download url
16.   string Firefox_DownLoad_Url_Win64 = grab$(Firefox_Info_Page, "<div id=""other-platforms"">", "</div>")
17.   Firefox_DownLoad_Url_Win64 = grab$(Firefox_DownLoad_Url_Win64, "<li class=""os_win64"">", "</li>")
18.   Firefox_DownLoad_Url_Win64 = grab$(Firefox_DownLoad_Url_Win64, "<a href=""", """")
19.  
20.   printl "Firefox available version...:", Firefox_Available_Version
21.   printl "Firefox Download url........:", Firefox_DownLoad_Url_Win64
22.  
23.   '---Download
24.   string sLocalFile = APP_SourcePath + "Firefox Setup " + Firefox_Available_Version + ".exe"
25.   printl "Downlaoding to", sLocalFile, "..."
26.   INET_UrlDownload(Firefox_DownLoad_Url_Win64, sLocalFile)
27.   printl "File downlaoded."
28.   printl
29.   printl "Press ESC to end or any other key within 10 secs to execute setup"
30.  
31.   '---Ask if execute setup
32.   string sUserReply = WaitKey(10)
33.   if sUserReply <> "[ESC]" and sUserReply <> "[TIMEOUT]" Then
34.     printl "Executing " + sLocalFile
35.     OS_Shell(sLocalFile, %OS_WNDSTYLE_NORMAL, %OS_SHELL_SYNC)
36.   Else
37.     printl "No execution. Execute manually if needed."
38.   end If
39.  
40. Else
41.   printl "! it seems you are not connected to internet." in %CCOLOR_FLIGHTRED
42. end If
43.  
44. printl "---All done, press a key to end or wait 5 secs---"
45. WaitKey(5)
46.  
47. '[todo] To add async download with any progress info ...
48. '       Install thinBasic and see example at \thinBasic\SampleScripts\INet\Internet_FileDownload.tbasic
49.  

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 30, 2018, 11:24:03 PM

Quote

BASIC doesn't get any thinner.

It worked and brought the real thinBasic to us.  ;D

Thanks for your contribution!

I changed your code block from. =SB to =thinbasic. I normally don't edit others posts but TB highlighting looks much better. I attached the geshi syntax highlighting file for thinBasic I'm using here on the forum. It might be pretty old. If you want to have a peek and post an updated copy that would be great.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 01:49:30 AM

Quote from: John on October 30, 2018, 10:53:18 PM

BASIC doesn't get any thinner.  ;)

Script BASIC

Pretty Tight, John!

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 01:50:46 AM

Quote from: erosolmi on October 30, 2018, 11:10:18 PM

Example in thinBasic.
Attached source code and executable

Ciao
Eros

Very Nice, Eros!!!

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 12:05:14 PM

My PYTHON version, using only modules that come with a standard installation...

Code: Python

1. #!/usr/bin/env python
2.  
3. import urllib2,re,sys
4.  
5. html = urllib2.urlopen('https://mozilla.org/firefox/new').read()
6.  
7. version = re.findall('data-latest-firefox="(\d+\.\d+)',html)[0]
8. download_link = re.findall('.+href="(.+latest-ssl.+os=win64.+US).+',html)[0]
9.  
10. sys.stdout.write ("\33[2J\33[HFirefox Download Challenge (Python Version) by AIR.\n\n")
11. print "Downloading Latest 64Bit Firefox (" + version + ") for Windows.\n"
12.  
13. f = urllib2.urlopen(download_link)
14. with open("Firefox Setup " + version + ".exe", "wb") as dl:
15.     total_size = int(f.info().getheader('Content-Length').strip())
16.     downloaded = 0
17.  
18.     while True:
19.         buffer = f.read(8192)
20.         if not buffer:
21.             print
22.             break
23.         downloaded += len(buffer)
24.         dl.write(buffer)
25.         percent = round( (float(downloaded) / total_size)*100,2)
26.         sys.stdout.write("Downloaded %d of %d bytes (%0.2f%%)\r" % (downloaded, total_size, percent))
27.     print "\nDownload Complete.\n"

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 31, 2018, 01:44:22 PM

So far SB followed by Bash are my leading challenge contenders.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 01:57:37 PM

That's cool, but remember this is not a competition between languages.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 31, 2018, 03:30:34 PM

I agree and it's a great way to see how different languages approach a common task.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 08:07:51 PM

OBJECTIVE C

I wanted to show this, because I think most people who don't use ObjC realize how easy it is to incorporate C code.

Code: Objective-C

1. /*
2.         Download the latest Firefox 64bit for Windows, Objective C version
3.        
4.         Written by AIR.
5.  
6.         This shows how Objective C can leverage C and C libraries
7.         within the same project.
8.  
9.         compile with: gcc -framework Foundation getFirefox.m -lcurl -O2 -o getFirefox
10. */
11.  
12. #import <Foundation/Foundation.h>
13. #import <curl/curl.h>
14.  
15. NSString *version;
16.  
17. int progress_callback(id clientp, double dltotal, double dlnow, double ultotal, double ulnow ) {
18.     printf("Downloaded %d of %d bytes\r",(int)dlnow,(int)dltotal);
19.         return 0;
20. }
21.  
22. void downloadFile(NSString *url, NSString *downloadFileName) {
23.     id handle;
24.     int success;
25.         FILE *fp;
26.  
27.         printf("Downloading Latest 64Bit Firefox (%s) for Windows.\n\n",version.UTF8String);
28.  
29.     handle = curl_easy_init();    
30.         if ((fp = fopen(downloadFileName.UTF8String, "wb+"))) {
31.                 curl_easy_setopt(handle, CURLOPT_URL, url.UTF8String);
32.                 curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION,1);
33.                 curl_easy_setopt(handle, CURLOPT_WRITEDATA, fp);
34.                 curl_easy_setopt(handle, CURLOPT_NOPROGRESS,0);
35.                 curl_easy_setopt(handle, CURLOPT_PROGRESSDATA,downloadFileName.UTF8String);
36.                 curl_easy_setopt(handle, CURLOPT_PROGRESSFUNCTION,progress_callback);
37.        
38.                 success = curl_easy_perform(handle);
39.                 fclose(fp);
40.         }
41.     curl_easy_cleanup(handle);
42.     printf("\n\nDownload of '%s' Complete.\n\n", downloadFileName.UTF8String);
43. }
44.  
45. NSString *regex(NSString *src, NSString *query) {
46.         NSRegularExpression *regex;
47.         NSString *result;
48.         NSArray *matches;
49.  
50.         regex = [NSRegularExpression regularExpressionWithPattern:query options:NSRegularExpressionCaseInsensitive error:nil];
51.         matches = [regex matchesInString:src options:0 range:NSMakeRange(0, [src length])];
52.         result = [src substringWithRange:[(NSTextCheckingResult*)matches[0] rangeAtIndex:1]];
53.  
54.         return result;
55. }
56.  
57.  
58.  
59. int main(int argc, char const *argv[]) {
60.         NSURL *url;
61.         NSString *htmlContent, *downloadLink;
62.  
63.         printf("\33[2J\33[HFirefox Download Challenge (Objective C Version) by AIR.\n\n");
64.  
65.         url = [NSURL URLWithString:@"https://mozilla.org/firefox/new"];
66.  
67.         htmlContent = [NSString stringWithContentsOfURL:url usedEncoding:nil error:nil];
68.  
69.         version = regex(htmlContent,@"data-latest-firefox=.(\\d+\\.\\d+)");
70.         downloadLink = regex(htmlContent,@".+href=.(.+latest-ssl.+win64.+US).+");
71.  
72.         downloadFile(downloadLink,[NSString stringWithFormat:@"Firefox Setup %@.exe",version]);
73.         return 0;
74. }
75.  
76.  

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 31, 2018, 08:13:16 PM

These code examples make me appreciate high level languages like BASIC even more.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 08:27:53 PM

What I like about examples like these is that they can lead to asking "Why didn't I think of that?"

Eros' use of "GRAB$" made me take another look at the Bacon manual.  It's called "INBETWEEN$" in Bacon and works the same (I rewrote my submission for Bacon to test that.  Haven't submitted it though).

It also got me thinking about how to implement something like that myself.  After a little research I was able to put something together in C++ that works. Maybe I'll dust off JADE and see what happens.... ;D

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 31, 2018, 08:37:40 PM

Quote

Maybe I'll dust off JADE and see what happens.... ;D

I would love to see that lost gem resurface.

The SB final version is due to your experimentation with cURL in other languages. Thanks!

40% of the visitors to the forum use Firefox. (dominant browser)

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on November 01, 2018, 01:38:07 AM

Quote from: John on October 31, 2018, 08:37:40 PM

Quote

Maybe I'll dust off JADE and see what happens.... ;D

I would love to see that lost gem resurface.

JADE VERSION

Code: C++

1. #include "jade.h"
2.  
3. /* compile with: g++ -std=c++11 getFirefox.cpp -DUSE_CURL -lcurl -O2 -o getFirefox */
4.  
5. /* Clone my repository for the JADE system itself: git clone https://bitbucket.org/Airr/jade.git */
6.  
7. MAIN
8.     DEF CSTRING HTML$,VERSION$,DL_LINK$, URL$("https://mozilla.org/firefox/new");
9.  
10.     CLS;
11.     PRINT("Firefox Download Challenge (**JADE** Version) by AIR.");
12.     PRINTNL;
13.  
14.     HTML$ = downloadPage(URL$);
15.  
16.     VERSION$ = REGEX(HTML$, "data-latest-firefox=.(\\d+\\.\\d+).");
17.     DL_LINK$ = REGEX(HTML$, ".+href=.(.+latest-ssl.+win64.+US).+");
18.  
19.     PRINT("Downloading Latest 64Bit Firefox ("+VERSION$+") for Windows.");
20.     PRINTNL;
21.  
22.     downloadFile(DL_LINK$,"Firefox Setup " + VERSION$ + ".exe");
23. ENDMAIN
24.  

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on November 01, 2018, 02:13:58 AM

Nice! (cURL pre-processor defines)

JADE has captured the number 2 spot on my list.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: erosolmi on November 01, 2018, 04:23:27 AM

Here a thinBasic ASYNC version with callback function used to determine download progression.
Used only libraries released with thinBasic official setup

Progress data is shown on a pop up box that will be removed when download is finished.

Maybe a little bit complex but worth to post to share ideas.

Code: thinBasic

1. uses "Console"
2. uses "iNet"
3. uses "OS"
4.  
5. printl "Connect to Firefox web site, determine latest Firefox version and download it"
6.  
7. '---Check if computer is connected to internet
8. if INET_GetState Then
9.   '---Get Firefox web site page
10.   string Firefox_Info_Page = INET_UrlGetString("https://www.mozilla.org/en-US/firefox/new/")
11.  
12.   '---Get Firefox available version
13.   string Firefox_Available_Version = grab$(Firefox_Info_Page, "data-latest-firefox=""", """")
14.  
15.   '---Get Win64 direct download url
16.   string Firefox_DownLoad_Url_Win64 = grab$(Firefox_Info_Page, "<div id=""other-platforms"">", "</div>")
17.   Firefox_DownLoad_Url_Win64 = grab$(Firefox_DownLoad_Url_Win64, "<li class=""os_win64"">", "</li>")
18.   Firefox_DownLoad_Url_Win64 = grab$(Firefox_DownLoad_Url_Win64, "<a href=""", """")
19.  
20.   printl "Firefox available version...:", Firefox_Available_Version
21.   printl "Firefox Download url........:", Firefox_DownLoad_Url_Win64
22.  
23.   '---Download
24.   string sLocalFile = APP_SourcePath + "Firefox Setup " + Firefox_Available_Version + ".exe"
25.  
26.   '---Initializes an application's use of the WinINet Internet functions.
27.   dword hInternet = INET_Internet_Open("Download Fireforx", %INTERNET_OPEN_TYPE_DIRECT)
28.   IF hInternet = %NULL THEN
29.     printl "INET_Internet_Open error"
30.     waitkey
31.     stop
32.   END IF
33.  
34.   long MyContext = 1
35.   long hFile = INET_Internet_OpenUrl(hInternet, Firefox_DownLoad_Url_Win64, "", %INTERNET_FLAG_NO_CACHE_WRITE, MyContext)
36.   IF hFile = %NULL THEN
37.     printl "INET_Internet_OpenUrl error. Check url: " & Firefox_DownLoad_Url_Win64
38.     INET_Internet_CloseHandle hInternet
39.     waitkey
40.     stop
41.   END IF
42.  
43. '---Set a callback for hInternet events. Driver will automatically call MyStatusFunction on status change
44.   long bResult = INET_Internet_SetStatusCallBack(hFile, MyStatusFunction)
45.  
46. '---Query remote server about the size of the file we are going to download
47.   long FileSize = INET_Http_QueryInfo(hFile, %HTTP_QUERY_CONTENT_LENGTH)
48.   printl "Real file size in bytes: " & FileSize
49.   IF FileSize = %NULL THEN
50.     printl "INET_Http_QueryInfo(hInternet, %HTTP_QUERY_CONTENT_LENGTH) error. Check url: " & Firefox_DownLoad_Url_Win64
51.     INET_Internet_CloseHandle hInternet
52.     INET_Internet_CloseHandle hFile
53.     waitkey
54.     stop
55.   END IF
56.  
57.    
58. '---Ok we can start a loop for file download. This can be automated in a loop or in a timer function
59. '---Reads the data in %BufferLen bytes chunks
60.   double T0 = timer
61.   double T1
62.  
63.   '---Setup a %BufferLen bytes buffer
64.     long  lBufferLen    = 500000 '500k
65.     string strBuffer    = STRING$(lBufferLen, $SPC)
66.     dword pstrBuffer    = STRPTR(strBuffer)
67.     String strData
68.     dword cbBytesRead  
69.     DWord TotalBytesRead
70.     long  lScreenLine   = 8
71.     long  lScreenPos    = 20
72.  
73.   string sSave = Console_SaveScreen(1, 1, 80, 24)
74.   Console_Box(lScreenPos - 1, lScreenLine - 1, 40, 8, 24, 24, "Downloading ...", 26, %Console_BOX_FLAG_3DOFF)
75.   DO
76.     bResult = INET_Internet_ReadFile(hFile, pstrBuffer, len(strBuffer), cbBytesRead)
77.     IF bResult = 0 OR cbBytesRead = 0 THEN EXIT DO
78.     IF cbBytesRead = LEN(strBuffer) THEN
79.       strData &= strBuffer
80.     ELSE
81.       strData &= LEFT$(strBuffer, cbBytesRead)
82.     END IF
83.     TotalBytesRead += cbBytesRead
84.     T1 = Timer
85.    
86.     PrintAt "Elapsed seconds  : " & LSet$(Format$(T1-T0, "0"), 19)                                         , lScreenPos + 1, lScreenLine + 1, 24
87.     PrintAt "Bytes read so far: " & LSet$(TotalBytesRead, 19)                                              , lScreenPos + 1, lScreenLine + 2, 24
88.     PrintAt "KBytes per second: " & LSet$(Format$((TotalBytesRead / Max(T1-T0, 1))/1024, "0.00"), 19)      , lScreenPos + 1, lScreenLine + 3, 24
89.     PrintAt "Estimated seconds: " & LSet$(Format$((FileSize * (T1-T0))/TotalBytesRead, "0"), 19)           , lScreenPos + 1, lScreenLine + 4, 24
90.     PrintAt "Seconds to go    : " & LSet$(Format$(((FileSize * (T1-T0))/TotalBytesRead)-(T1-T0), "0"), 19) , lScreenPos + 1, lScreenLine + 5, 24
91.    
92.     Console_ProgressBar(1, lScreenPos + 1, lScreenLine + 7, 40-1, 27, 1, FileSize, TotalBytesRead)
93.   LOOP
94.  
95.   Console_RestoreScreen(1, 1, 80, 24, sSave)
96.  
97. '---Closes all used handles
98.   INET_Internet_CloseHandle hInternet
99.   INET_Internet_CloseHandle hFile
100.  
101.  
102.   printl "Saving file: " & sLocalFile
103.   Save_File(sLocalFile, strData)
104.  
105.   printl "File saved."
106.   printl
107.   printl "Press ESC to end or any other key within 10 secs to execute setup"
108.  
109.  
110.   '---Ask if execute setup
111.   string sUserReply = WaitKey(10)
112.   if sUserReply <> "[ESC]" and sUserReply <> "[TIMEOUT]" Then
113.     printl "Executing " + sLocalFile
114.     OS_Shell(sLocalFile, %OS_WNDSTYLE_NORMAL, %OS_SHELL_SYNC)
115.   Else
116.     printl "No execution. Execute manually if needed."
117.   end If
118.  
119. Else
120.   printl "! it seems you are not connected to internet." in %CCOLOR_FLIGHTRED
121. end If
122.  
123. printl "---All done, press a key to end or wait 5 secs---"
124. WaitKey(5)
125.  
126.  
127. '------------------------------------------------------------------
128. ' CallBack function automatically called by Internet driver when
129. ' status change for a specific Internet handle to which a callback
130. ' function has been installed.
131. ' Parameters are mandatory and are defined my Microsoft
132. ' callback template. More info at: http://msdn.microsoft.com/en-us/library/aa385121(VS.85).aspx
133. '------------------------------------------------------------------
134. callback function MyStatusFunction( _
135.                 byval hInternet                 as long , _   '---Mandatory in thinBasic and in API
136.                 byval dwContext                 as long , _   '---Mandatory in thinBasic and in API
137.                 byval dwInternetStatus          as long , _   '---Mandatory in thinBasic and in API
138.                 byval lpvStatusInformation      as long , _   '---Optional in thinBasic, mandatory in API
139.                 byval dwStatusInformationLength as long   _   '---Optional in thinBasic, mandatory in API
140.              ) as long
141. '------------------------------------------------------------------
142.   dim BytesReceived     as dword at 0
143.   static TotBytes       as dword
144.  
145.   static MoreStatusLine as long = 20
146.      
147.   'printat "Data from " & function_name & " callback"          , 1, 17
148.   'printat "  Internet handle: " & hInternet & "  Internet dwContext: " & dwContext, 1, 18
149.  
150.   select case dwInternetStatus
151.     case %INTERNET_STATUS_RECEIVING_RESPONSE
152.  
153.     case %INTERNET_STATUS_RESPONSE_RECEIVED
154.       '---  Successfully received a response from the server.
155.       '---  The lpvStatusInformation parameter points to a DWORD value that contains the number, in bytes, received.
156.       setat(BytesReceived, lpvStatusInformation)
157.       TotBytes += BytesReceived
158.  
159.       '---ATTENTION: TotalBytes received can differ from total bytes of the file to be downloaded
160.  
161.       'printat "  Receiving data (" & dwInternetStatus & " " & INET_Internet_StatusGetDescription(dwInternetStatus) & ") TotBytes: " & TotBytes, 1, 19
162.  
163.     case else
164.       '---In case of other status, just decode it at the bottom.
165.       'printat "  (" & dwInternetStatus & " " & INET_Internet_StatusGetDescription(dwInternetStatus) & ")", 1, MoreStatusLine
166.       incr MoreStatusLine
167.  
168.   end select            
169.  
170.  
171. end function
172.  
173.  

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on November 01, 2018, 09:47:00 AM

Very Cool, Eros!

Question:  Is your Console_Box constructed via WinAPI calls, or are you using a 3rd party library?

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: erosolmi on November 01, 2018, 10:31:36 AM

All developed by me with only Windows console api:
WriteConsoleOutputCharacter
FillConsoleOutputAttribute
ReadConsoleOutputCharacter

Console module was one of the first module I develop in 2004 when I started to work on thinBasic

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on December 12, 2018, 12:19:18 AM

Since I'm working on updating MBC, here's an MBC version:

Code: Text

1. $MODULE "curl.inc"
2.  
3. CLS
4. Dim HTML$,dl_version$,dl_link$,fname$, match as REGEX
5.  
6. HTML$ = downloadPage$("https://www.mozilla.org/en-US/firefox/new")
7.  
8. if bcxregex(HTML$,"data-latest-firefox=.([0-9]+.[0-9]+\.?[0-9]?)",&match) then dl_version$ = match.results[1]
9. if bcxregex(HTML$,".+(http.+ssl[&a-z;=-]+win64[&a-z;=-]+)",&match) then dl_link$ = match.results[1]
10.  
11. Print E"Firefox Download Challenge (MBC Version) by AIR.\n"
12. Print E"Downloading Latest 64Bit Firefox (",dl_version,") for Windows.\n"
13.  
14. downloadFile(dl_link, "Firefox Setup-" & dl_version$ & ".exe")

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on December 12, 2018, 12:43:57 AM

Old challenges never die, they only get more interesting.

Curious. Can you time your MBC and SB Firefox download challenges for a reference?

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on March 05, 2019, 03:46:47 PM

I thought I would mention that one should reinitialize curl between commits. Curl was sending a previous POST buffer with a following GET.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on March 07, 2019, 12:24:46 AM

Go version:

Code: Go

1. package main
2.  
3. import (
4.         "fmt"
5.         "io"
6.         "io/ioutil"
7.         "net/http"
8.         "os"
9.         "regexp"
10.         "strings"
11.  
12.         "github.com/dustin/go-humanize"
13. )
14.  
15. // WriteCounter counts the number of bytes written to it. It implements to the io.Writer
16. // interface and we can pass this into io.TeeReader() which will report progress on each
17. // write cycle.
18. type WriteCounter struct {
19.         Total    uint64
20.         Filesize uint64
21. }
22.  
23. func main() {
24.         clear()
25.         fmt.Printf("Firefox Download Challenge (GO Version) by AIR.\n\n")
26.         url := "https://mozilla.org/firefox/new"
27.  
28.         // Retrieve the data from the url
29.         resp, err := http.Get(url)
30.         chkError(err)
31.  
32.         // Auto close connection
33.         defer resp.Body.Close()
34.  
35.         // reads html as a slice (array) of bytes
36.         html, err := ioutil.ReadAll(resp.Body)
37.         chkError(err)
38.  
39.         // regex to extract current version
40.         re, err := regexp.Compile(`data-latest-firefox="([0-9]+\.[0-9]+?\.[0-9]+)"`)
41.         chkError(err)
42.  
43.         // Regex to extract the download link
44.         re2, err := regexp.Compile(`.+href="(.+latest-ssl.+os=win64.+US).+`)
45.         chkError(err)
46.  
47.         // Perform search for version and download link
48.         ffVersion := re.FindStringSubmatch(string(html))
49.         ffLink := re2.FindStringSubmatch(string(html))
50.  
51.         // Download the Windows installer
52.         err = DownloadFile("Firefox Setup "+ffVersion[1]+".exe", ffLink[1])
53.         chkError(err)
54.  
55.         fmt.Println("\nDownload Complete.")
56. }
57.  
58. func chkError(e error) {
59.         if e != nil {
60.                 panic(e)
61.         }
62. }
63.  
64. // Clears screen (Linux/macOS)
65. func clear() {
66.         fmt.Print("\033[2J\033[H")
67. }
68.  
69. // DownloadFile will download a url to a local file. It's efficient because it will
70. // write as it downloads and not load the whole file into memory. We pass an io.TeeReader
71. // into Copy() to report progress on the download.
72. func DownloadFile(filepath string, url string) error {
73.  
74.         // Create the file
75.         out, err := os.Create(filepath)
76.         if err != nil {
77.                 return err
78.         }
79.         defer out.Close()
80.  
81.         // Get the data
82.         resp, err := http.Get(url)
83.         if err != nil {
84.                 return err
85.         }
86.         defer resp.Body.Close()
87.  
88.         // Create our progress reporter (with reported Filesize set) and pass it to be used alongside our writer
89.         counter := &WriteCounter{Filesize: uint64(resp.ContentLength)}
90.  
91.         _, err = io.Copy(out, io.TeeReader(resp.Body, counter))
92.         if err != nil {
93.                 return err
94.         }
95.  
96.         // The progress outputs on a single line so print a new line once it's finished downloading
97.         fmt.Print("\n")
98.  
99.         return nil
100. }
101.  
102. // Callback used by WriteCounter
103. func (wc *WriteCounter) Write(p []byte) (int, error) {
104.         n := len(p)
105.         wc.Total += uint64(n)
106.  
107.         wc.PrintProgress()
108.         return n, nil
109. }
110.  
111. // PrintProgress displays the download status
112. func (wc WriteCounter) PrintProgress() {
113.         // Clear the line by using a character return to go back to the start and remove
114.         // the remaining characters by filling it with spaces
115.         fmt.Printf("\r%s", strings.Repeat(" ", 30))
116.  
117.         // Return again and print current status of download
118.         // We use the humanize package to print the status in a human-readable way (e.g. 10 MB)
119.         fmt.Printf("\rDownloaded %s of %s", humanize.Bytes(wc.Total), humanize.Bytes(wc.Filesize))
120.  
121. }
122.  

Output:
Firefox Download Challenge (GO Version) by AIR.

Downloaded 46 MB of 46 MB

Download Complete.

AIR.