AllBASIC

BASIC User Group => Code Challenges => Topic started by: AIR on October 26, 2018, 03:50:51 pm

Title: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 26, 2018, 03:50:51 pm
New Day, New Code Challenge.

Download the latest 64 bit version of Firefox for Windows.

Requirements:


The progress should be displayed in the following format:
Quote
    Downloading Latest Firefox (version number goes here)
    Downloaded 34140000 of 44400072 at 11784kb/s

    That is current data downloaded in real time, the total size, and the internet speed each in kb.

    The download progress should display on a single line, updating that line as the download progresses.


I'll be posting a submission using Nim this evening and leave the other languages to whoever wants to submit something....

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 26, 2018, 03:59:57 pm
This look like a natural for SB and cURL.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 26, 2018, 06:14:38 pm
NIM Submission, works on macOS and Linux (on a Raspberry Pi)

to compile:  nim c -d:ssl getFirefox.nim

Code: [Select]
Firefox Download Challenge (Nim Version) by AIR.

Downloading Latest 64Bit Firefox (63.0) for Windows.

Downloaded 42.343MiB of 42.343MiB

Download Complete.

Code: Text
  1. import httpclient, htmlparser,xmlparser, xmltree, streams, ospaths, asyncdispatch, strutils
  2.  
  3. const
  4.     ClrLine = "\x1b[0K\r"
  5. var
  6.     totalsize:BiggestInt
  7.  
  8. proc ClrScr() =
  9.     stdout.write("\x1b[2J\x1b[H")
  10.    
  11. proc showProgress(total, progress, speed: BiggestInt) {.async.} =
  12.     totalsize = total
  13.  
  14.     stdout.write(ClrLine, "Downloading ", progress.formatSize, " of ", total.formatSize, " at ", (speed div 1000).formatSize(','))
  15.  
  16.     flushFile(stdout)
  17.  
  18. proc download(url,filename:string) {.async.} =
  19.     var client = newAsyncHttpClient()
  20.  
  21.     client.onProgressChanged = showProgress
  22.  
  23.     await client.downloadFile(url, filename)
  24.  
  25.     stdout.write(ClrLine, "Downloaded ", totalsize.formatSize, " of ", totalsize.formatSize," ".repeat(20))
  26.  
  27.     flushFile(stdout)
  28.  
  29.     echo "\n\nDownload Complete.\n"
  30.  
  31. proc main() =
  32.     var
  33.         client = newHttpClient()
  34.         src = client.getContent("https://www.mozilla.org/en-US/firefox/new")
  35.         c = parseHtml( newStringStream(src) )
  36.         url: string
  37.         version: string
  38.  
  39.  
  40.     for html in c.findAll("html"):
  41.         version = html.attr("data-latest-firefox")
  42.         break
  43.  
  44.     for d in c.findAll("li"):
  45.         if d.attr("class") == "os_win64":
  46.             for e in d.findAll("a"):
  47.                 if e.attr("class") == "download-link":
  48.                     url = e.attr("href")
  49.                     break
  50.  
  51.     ClrScr()
  52.  
  53.     echo "Firefox Download Challenge (Nim Version) by AIR.\n"
  54.     echo "Downloading Latest 64Bit Firefox (",version,") for Windows.\n"
  55.  
  56.     waitFor url.download("Firefox Setup " & version & ".exe")
  57.  
  58. main()
  59.  


This would have been so much simpler/clearer if Nim supported xpath syntax, but there ya go....


AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 01:03:53 am
Here is my first pass at the challenge. I can download the file fine but the cURL progress meter feature doesn't seem to be working for me yet. I may have to resort to the static INFO funtions to build your download status string.

@AIR - When you said ignore the stub URL, did you mean the download file URL it returns or did I get the right link?
Code: Script BASIC
  1. INCLUDE curl.bas
  2.  
  3. ch = curl::init()
  4. curl::option(ch,"URL","https://www.mozilla.org/en-US/firefox/new/")
  5. wp = curl::perform(ch)
  6.  
  7. IF wp LIKE "*data-latest-firefox=\"*\" data-esr-versions*" THEN PRINT JOKER(2),"\n"
  8.  
  9. IF wp LIKE """*<div id="other-platforms">*<li class="os_win64">*<a href="*"*""" THEN
  10.   curl::option(ch,"URL", JOKER(4))
  11. END IF
  12.  
  13. dl_html = curl::perform(ch)
  14. IF dl_html LIKE """*href="*"*""" THEN dl_file = JOKER(2)
  15.  
  16. curl::option(ch,"URL", dl_file)
  17. curl::option(ch,"FILE","Firefox_Win64.exe")
  18. curl::perform(ch)
  19.  
  20. curl::finish(ch)
  21.  


jrs@jrs-laptop:~/sb/abcc$ time scriba ffdl.sb
63.0

real   0m2.422s
user   0m1.108s
sys   0m0.291s

jrs@jrs-laptop:~/sb/abcc$ ls -l Firefox_Win64.exe
-rw-r--r-- 1 jrs jrs 44400072 Oct 27 01:02 Firefox_Win64.exe
jrs@jrs-laptop:~/sb/abcc$

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 27, 2018, 10:21:53 am

@AIR - When you said ignore the stub URL, did you mean the download file URL it returns or did I get the right link?


The stub version is a small program you download that will then download and install Firefox.

What we're after is the full installer, which once you have it doesn't require an internet connection for Firefox to be installed on a system.

Quote from: John
I may have to resort to the static INFO funtions to build your download status string.

I think you're gonna have to update the Curl module, because according to the old module docs CURLOPT_PROGRESSFUNCTION is not implemented:

Quote from: ScriptBasic Documentation
CURLOPT_PROGRESSFUNCTION is used in conjunction with CURLOPT_PROGRESSDATA to specify a progress function that CURLIB library calls from time to time to allow progress indication for the user. This is not implemented in the ScriptBasic interface.

Which means that you won't get real time stats while the download is happening with the Curl module as it currently exists...


AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 10:45:23 am
Based on your feedback, I have the right installer file and I'm stuck with after the fact download stats without enhancing the libcurl extension module.

On a positive note, a 2.4 second delay before completion and results isn't eternity.

progressfunc.c
Code: C
  1. /***************************************************************************
  2.  *                                  _   _ ____  _
  3.  *  Project                     ___| | | |  _ \| |
  4.  *                             / __| | | | |_) | |
  5.  *                            | (__| |_| |  _ <| |___
  6.  *                             \___|\___/|_| \_\_____|
  7.  *
  8.  * Copyright (C) 1998 - 2018, Daniel Stenberg, <daniel@haxx.se>, et al.
  9.  *
  10.  * This software is licensed as described in the file COPYING, which
  11.  * you should have received as part of this distribution. The terms
  12.  * are also available at https://curl.haxx.se/docs/copyright.html.
  13.  *
  14.  * You may opt to use, copy, modify, merge, publish, distribute and/or sell
  15.  * copies of the Software, and permit persons to whom the Software is
  16.  * furnished to do so, under the terms of the COPYING file.
  17.  *
  18.  * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
  19.  * KIND, either express or implied.
  20.  *
  21.  ***************************************************************************/
  22. /* <DESC>
  23.  * Use the progress callbacks, old and/or new one depending on available
  24.  * libcurl version.
  25.  * </DESC>
  26.  */
  27. #include <stdio.h>
  28. #include <curl/curl.h>
  29.  
  30. #if LIBCURL_VERSION_NUM >= 0x073d00
  31. /* In libcurl 7.61.0, support was added for extracting the time in plain
  32.    microseconds. Older libcurl versions are stuck in using 'double' for this
  33.    information so we complicate this example a bit by supporting either
  34.    approach. */
  35. #define TIME_IN_US 1  
  36. #define TIMETYPE curl_off_t
  37. #define TIMEOPT CURLINFO_TOTAL_TIME_T
  38. #define MINIMAL_PROGRESS_FUNCTIONALITY_INTERVAL     3000000
  39. #else
  40. #define TIMETYPE double
  41. #define TIMEOPT CURLINFO_TOTAL_TIME
  42. #define MINIMAL_PROGRESS_FUNCTIONALITY_INTERVAL     3
  43. #endif
  44.  
  45. #define STOP_DOWNLOAD_AFTER_THIS_MANY_BYTES         6000
  46.  
  47. struct myprogress {
  48.   TIMETYPE lastruntime; /* type depends on version, see above */
  49.   CURL *curl;
  50. };
  51.  
  52. /* this is how the CURLOPT_XFERINFOFUNCTION callback works */
  53. static int xferinfo(void *p,
  54.                     curl_off_t dltotal, curl_off_t dlnow,
  55.                     curl_off_t ultotal, curl_off_t ulnow)
  56. {
  57.   struct myprogress *myp = (struct myprogress *)p;
  58.   CURL *curl = myp->curl;
  59.   TIMETYPE curtime = 0;
  60.  
  61.   curl_easy_getinfo(curl, TIMEOPT, &curtime);
  62.  
  63.   /* under certain circumstances it may be desirable for certain functionality
  64.      to only run every N seconds, in order to do this the transaction time can
  65.      be used */
  66.   if((curtime - myp->lastruntime) >= MINIMAL_PROGRESS_FUNCTIONALITY_INTERVAL) {
  67.     myp->lastruntime = curtime;
  68. #ifdef TIME_IN_US
  69.     fprintf(stderr, "TOTAL TIME: %" CURL_FORMAT_CURL_OFF_T ".%06ld\r\n",
  70.             (curtime / 1000000), (long)(curtime % 1000000));
  71. #else
  72.     fprintf(stderr, "TOTAL TIME: %f \r\n", curtime);
  73. #endif
  74.   }
  75.  
  76.   fprintf(stderr, "UP: %" CURL_FORMAT_CURL_OFF_T " of %" CURL_FORMAT_CURL_OFF_T
  77.           "  DOWN: %" CURL_FORMAT_CURL_OFF_T " of %" CURL_FORMAT_CURL_OFF_T
  78.           "\r\n",
  79.           ulnow, ultotal, dlnow, dltotal);
  80.  
  81.   if(dlnow > STOP_DOWNLOAD_AFTER_THIS_MANY_BYTES)
  82.     return 1;
  83.   return 0;
  84. }
  85.  
  86. #if LIBCURL_VERSION_NUM < 0x072000
  87. /* for libcurl older than 7.32.0 (CURLOPT_PROGRESSFUNCTION) */
  88. static int older_progress(void *p,
  89.                           double dltotal, double dlnow,
  90.                           double ultotal, double ulnow)
  91. {
  92.   return xferinfo(p,
  93.                   (curl_off_t)dltotal,
  94.                   (curl_off_t)dlnow,
  95.                   (curl_off_t)ultotal,
  96.                   (curl_off_t)ulnow);
  97. }
  98. #endif
  99.  
  100. int main(void)
  101. {
  102.   CURL *curl;
  103.   CURLcode res = CURLE_OK;
  104.   struct myprogress prog;
  105.  
  106.   curl = curl_easy_init();
  107.   if(curl) {
  108.     prog.lastruntime = 0;
  109.     prog.curl = curl;
  110.  
  111.     curl_easy_setopt(curl, CURLOPT_URL, "https://example.com/");
  112.  
  113. #if LIBCURL_VERSION_NUM >= 0x072000
  114.     /* xferinfo was introduced in 7.32.0, no earlier libcurl versions will
  115.        compile as they won't have the symbols around.
  116.  
  117.        If built with a newer libcurl, but running with an older libcurl:
  118.        curl_easy_setopt() will fail in run-time trying to set the new
  119.        callback, making the older callback get used.
  120.  
  121.        New libcurls will prefer the new callback and instead use that one even
  122.        if both callbacks are set. */
  123.  
  124.     curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, xferinfo);
  125.     /* pass the struct pointer into the xferinfo function, note that this is
  126.        an alias to CURLOPT_PROGRESSDATA */
  127.     curl_easy_setopt(curl, CURLOPT_XFERINFODATA, &prog);
  128. #else
  129.     curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, older_progress);
  130.     /* pass the struct pointer into the progress function */
  131.     curl_easy_setopt(curl, CURLOPT_PROGRESSDATA, &prog);
  132. #endif
  133.  
  134.     curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
  135.     res = curl_easy_perform(curl);
  136.  
  137.     if(res != CURLE_OK)
  138.       fprintf(stderr, "%s\n", curl_easy_strerror(res));
  139.  
  140.     /* always cleanup */
  141.     curl_easy_cleanup(curl);
  142.   }
  143.   return (int)res;
  144. }
  145.  
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 12:57:52 pm
Script BASIC

Code: Script BASIC
  1. ' Firefox Download Challenge (Script BASIC Version) by JRS.
  2.  
  3. INCLUDE curl.bas
  4.  
  5. ch = curl::init()
  6. curl::option(ch,"URL","https://www.mozilla.org/en-US/firefox/new/")
  7. wp = curl::perform(ch)
  8.  
  9. IF wp LIKE "*data-latest-firefox=\"*\" data-esr-versions*" THEN
  10.   version = JOKER(2)
  11.   PRINT "Downloading Latest 64Bit Firefox (",version,") for Windows.\n"
  12. END IF
  13.  
  14. IF wp LIKE """*<div id="other-platforms">*<li class="os_win64">*<a href="*"*""" THEN
  15.   curl::option(ch,"URL", JOKER(4))
  16. END IF
  17.  
  18. dl_html = curl::perform(ch)
  19. IF dl_html LIKE """*href="*"*""" THEN dl_file = JOKER(2)
  20.  
  21. curl::option(ch,"URL", dl_file)
  22. curl::option(ch,"FILE","Firefox_Setup-" & version & ".exe")
  23. curl::option(ch,"NOPROGRESS",0)
  24. curl::perform(ch)
  25.  
  26. PRINTNL
  27. PRINT "Firefox_Setup-" & version & ".exe Downloaded ",FORMAT("%~##,###,###~ Bytes",curl::info(ch,"SIZE_DOWNLOAD")), _
  28.   " at ",FORMAT("%~##,###,###~ Bytes/Second",curl::info(ch,"SPEED_DOWNLOAD")),".\n"
  29.  
  30. curl::finish(ch)
  31.  


jrs@jrs-laptop:~/sb/abcc$ time scriba ffdl.sb
Downloading Latest 64Bit Firefox (63.0) for Windows.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 42.3M  100 42.3M    0     0  24.9M      0  0:00:01  0:00:01 --:--:-- 27.7M

Firefox_Setup-63.0.exe Downloaded 44,400,072 Bytes at 26,133,061 Bytes/Second.

real   0m2.863s
user   0m1.213s
sys   0m0.276s
jrs@jrs-laptop:~/sb/abcc$

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 01:53:57 pm
Curious how big that Nim executable is with all those dependency requirements?

Code: [Select]
import httpclient, htmlparser,xmlparser, xmltree, streams, ospaths, asyncdispatch, strutils
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 27, 2018, 03:53:21 pm
That's because the core Nim compiler is designed to be fairly lean.

Similar to SB/Python/Ruby/FreePascal/Delphi/Xojo(RealBasic)/Powershell/Lua/ZShell/C#, it has a concept of a "Modules" system whereby additional functionality can be added to a given programming language without having it all baked in, or having to recompile the core to add that functionality.

Not having all functionality baked in also allows one to alter a given module in order to achieve a desired result.  SB's Curl module is a perfect example, where it doesn't currently support Curl Callbacks, but that feature can be added without messing with any of the core stuff.

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 27, 2018, 04:12:57 pm
I was curious how difficult this would be in POWERSHELL, so here's my implementation:

Code: PowerShell
  1. Import-Module BitsTransfer
  2.  
  3. Clear-Host
  4.  
  5. echo "Firefox Download Challenge (Powershell Version) by AIR.`r`n"
  6.  
  7. $progressPreference = 'silentlyContinue'
  8. $result = Invoke-WebRequest "https://www.mozilla.org/en-US/firefox/new/"
  9. $progressPreference = 'Continue'
  10.  
  11. $FF_Version = ($result | %{ $_ -match 'data-latest-firefox="(.+?)"'}) | %{$Matches[1]}
  12.  
  13. echo "Downloading Latest 64Bit Firefox ($FF_Version) for Windows.`r`n"
  14.  
  15. $HTML = $result.ParsedHtml.getElementsByTagName('li')
  16.  
  17. $hits = $HTML | where {$_.outerHTML -match "class=os_win64"}
  18.  
  19. $download_link = ($hits | where {$_.outerTEXT -match 'Windows 64-bit'}).innerHTML | %{[regex]::matches($_,'(?<=\").+?(?=\")')[0].value}
  20.  
  21. Start-BitsTransfer -Source $download_link -Destination "Firefox Setup $FF_VERSION.exe"
  22.  
  23. echo "Download Complete.`r`n"
  24.  
  25.  

Note that the "BitsTransfer" module provides a console-based progress bar, so technically this doesn't meet the full requirements of the challenge.  But it was fun to code!

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 05:43:26 pm
Are you able to provide execution times for the submissions? Seeing the output would also be nice.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 27, 2018, 06:52:03 pm
Execution times in this case would be very dependent on your Internet speeds.

For example, my speed is rated at 960+mb/s (roughly 120 MB/s), so the download happens in about a half second for me.  That's a wired connection, WiFi would always be slower.

Also, it would be dependent on how your connection to the website is routed.  I might go through fast routers, and you might go through one that is having packet issues resulting in retries/performance loss.

Besides, this challenge isn't about how FAST you can do it, but HOW you would do it....that's going to be the case with any challenges I throw out there... 8)

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 27, 2018, 06:59:36 pm
Maybe posting a poll at the end of a challenge so those lurking can vote on what they think is the best solution.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 27, 2018, 09:24:34 pm
Firefox Download Challenge (NIM Version) by AIR

(https://binarymagic.net/nim_Challenge.gif)

Updated code, removed XML-based parsing and use REGEX instead...

Code: Text
  1. import httpclient,re,asyncdispatch,strutils
  2.  
  3.  
  4.  
  5. const
  6.     ClrLine = "\x1b[0K\r"
  7. var
  8.     totalsize:BiggestInt
  9.  
  10. proc ClrScr() =
  11.     stdout.write("\x1b[2J\x1b[H")
  12.  
  13. proc showProgress(total, progress, speed: BiggestInt) {.async.} =
  14.     totalsize = total
  15.     stdout.write(ClrLine, "Downloading ", progress.formatSize, " of ", total.formatSize, " at ", (speed div 1000).formatSize(','))
  16.     flushFile(stdout)
  17.  
  18. proc download(url,filename:string) {.async.} =
  19.     var client = newAsyncHttpClient()
  20.     client.onProgressChanged = showProgress
  21.     await client.downloadFile(url, filename)
  22.     stdout.write(ClrLine, "Downloaded ", totalsize.formatSize, " of ", totalsize.formatSize," ".repeat(20))
  23.     flushFile(stdout)
  24.     echo "\n\nDownload Complete.\n"
  25.    
  26. proc main() =
  27.     var
  28.         client = newHttpClient()
  29.         src = client.getContent("https://www.mozilla.org/en-US/firefox/new")
  30.         url,version: string
  31.         x:int
  32.         matches: array[2,string]
  33.  
  34.     x = src.find(re"data-latest-firefox=.(\d+.\d+).",matches)
  35.     version = matches[0]
  36.  
  37.     x = src.find(re"(http.+product=firefox-latest-ssl.+os=win64.+en-US)",matches)
  38.     url = matches[0]
  39.  
  40.     ClrScr()
  41.     echo "Firefox Download Challenge (Nim Version) by AIR.\n"
  42.     echo "Downloading Latest 64Bit Firefox (",version,") for Windows.\n"
  43.  
  44.     waitFor url.download("Firefox Setup " & version & ".exe")
  45.  
  46.  
  47. main()
  48.  

C'mon , there's gotta be SOMEONE who would like to give this a try in the language of choice!!  (Tomaaz, I know you're out there!!!  Post it on RetroB if you want, I'm interested in seeing how you do this!)

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 28, 2018, 12:34:41 am
I bet a code challenge to make REGEX emulate LIKE would be fun. I have avoided regex like the plague. It's time I learn this commonly used function. (so f*cking cryptic)
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 28, 2018, 11:59:42 am
Script BASIC

Note: Doesn't meet the challenge requirement for displaying download status live. I'm not sure the user would have time to read it anyways.


How about something like this?


riveraa@nas:~/src$ ./sb64-x86_64.AppImage getFirefox.sb
Downloading Latest 64Bit Firefox (63.0) for Windows.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 42.3M  100 42.3M    0     0  71.2M      0 --:--:-- --:--:-- --:--:-- 81.3M

Firefox_Setup-63.0.exe Downloaded 44,400,072 Bytes at 74,754,643 Bytes/Second.



Add the following to your code before the final CURL::PERFORM:

Code: Script BASIC
  1.     curl::option(ch,"NOPROGRESS",0)

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 28, 2018, 12:09:08 pm
Too cool!

The Script BASIC submission post has been updated.

I had this cURL progress display working in a example of long ago. I just couldn't remember how I did it.

Nice job AIR. This is what sets you apart from everyone else. Your curiosity, experience and being a generous contributor.

FWIW: I'm wondering if what curl::info(ch,"SIZE_DOWNLOAD") is the accumulation of all bytes downloaded since the INIT of the cURL library.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 28, 2018, 05:52:38 pm
I download 'Firefox Setup 63.0.exe' directly from the web site and it matched (size) to what SB / cURL is saying for the download size. This solves the question I had about cURL accumulating its downloaded byte count giving a wrong value. It seems it's the cURL progress function not giving acurate file size info.

Quote
--compressed
(HTTP) Request a compressed response using one of the algorithms curl supports, and save the uncompressed document.  If this option is used and  the  server  sends  an unsupported encoding, curl will report an error.

I wonder If the progress function is seeing the compressed request byte count / rate.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 28, 2018, 09:21:05 pm
I download 'Firefox Setup 63.0.exe' directly from the web site and it matched (size) to what SB / cURL is saying for the download size. This solves the question I had about cURL accumulating its downloaded byte count giving a wrong value. It seems it's the cURL progress function not giving acurate file size info.

I believe it's a timing issue.  The download completes during the callback cycle, and the value may not be updated.

Quote
--compressed
(HTTP) Request a compressed response using one of the algorithms curl supports, and save the uncompressed document.  If this option is used and  the  server  sends  an unsupported encoding, curl will report an error.

I wonder If the progress function is seeing the compressed request byte count / rate.

You have to actually enable that option (CURLOPT_ACCEPT_ENCODING), does SB have that enabled in the Module by default?

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 28, 2018, 09:32:59 pm
It doesn't look like it based on interface.c for the cURL extension module.

Quote

To aid applications not having to bother about what specific algorithms this particular libcurl build supports, libcurl allows a zero-length string to be set ("") to ask for an Accept-Encoding: header to be used that contains all built-in supported encodings.

Seems nothing equals everything.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 29, 2018, 03:44:20 pm
Quote from: AIR
Tomaaz, I know you're out there!!!  Post it on RetroB if you want, I'm interested in seeing how you do this!

Tomaaz was a member here but deleted his own account. I think you are wasting your time on him. His only forum interest is tormenting Aurel and bitching about everything.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 29, 2018, 04:01:05 pm
My first thought with this challenge was to use SB sockets to connect to get the Firefox download page. That was abruptly halted by Cloudfare. Not like the good old HTTP days.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 29, 2018, 06:24:07 pm
I think you need to send a proper user agent , as well as add SSL support to Sockets.

Bacon has the same issue with it's built in Socket support although Peter coded something that incorporates what's needed as a proof of concept.

I think that most of the older BASICS will have this problem as well.  I know that MBC does, but I'm not gonna fix that since it's essentially dead now....

AIR.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 29, 2018, 06:30:07 pm
I wonder if James Fuller is still doing his version of BCX?
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 29, 2018, 07:30:02 pm
I think it would be possible to use wget with a SB SYSTEM or EXECUTE to get the Firefox download page and do the file download with its built in progress display.

The whole challenge could be done with bash, wget and regex.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 29, 2018, 08:24:38 pm
I already did a Bash version for work.....LOL.

Edit:

I originally wrote this to download the latest version for macOS; with a minor tweak it works for windows.

There was much more to this for the Mac, which I've stripped out.

Code: Bash
  1. #!/bin/bash
  2.  
  3. # ff.sh
  4. # Script to download the latest version
  5. # of Firefox 64bit for Windows
  6. #
  7. # Written by Armando I. Rivera (AIR)
  8. # Timeinc Global Technology Services
  9.  
  10. XPATH="string(//div[@id='outer-wrapper']/main/div[@id='other-platforms']/div[@class='content']/section[@class='section-other-platforms']/div[@id='download-platform-list-release']/ul[@class='download-platform-list recommended']/li[@class='os_win64']/a/@href)"
  11.  
  12. VERSION=$(curl -s https://www.mozilla.org/en-US/firefox/new/ | xmllint --html --xpath 'string(/html/@data-latest-firefox)' - 2>/dev/null)
  13.  
  14. curl -s https://www.mozilla.org/en-US/firefox/new/ | xmllint --html --xpath "$XPATH" - 2>/dev/null | xargs wget -O "Firefox Setup $VERSION.exe" -q --show-progress
  15.  
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 29, 2018, 09:56:55 pm
I love it when there is more comment than code.  8)
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 30, 2018, 07:06:37 am
I think XPATH can be replaced with:

Code: Bash
  1. XPATH="string(//div[@id='outer-wrapper']/*/*/*/*/*/*/li[@class='os_win64']/a/@href)"

 ;D ;D 8)

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 30, 2018, 09:02:00 am
It's amazing what can be done with Bash. Peter has a version of BaCon written in it. 8)
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 30, 2018, 12:46:32 pm
Speaking of Peter/Bacon, here's my BACON Version

I've included a zip file with the updated Curl module, as well as a custom REGEX module that I wrote just for this.

Bacon's native REGEX function doesn't return any matches as strings/text.  You have to use the return value from that function, which is where in the file the hit begins, and use the REGLEN builtin to get the length.  Then you need to feed the source, the REGEX return value, and the REGLEN variable to MID$ to retrieve the actual hit.

My function (called ReGex) returns the actual text.  Like Peter's implementation, it's based on Posix Regex, which has it's own arcane regex syntax.

Interestingly, the regex that I feed to my function doesn't work with Peter's; I haven't looked into why that is since I don't think Peter would want to modify his implementation much (if at all) since a few other Bacon functions seem to rely on it.

Anyway, here's the Bacon source (again, the required modules are in the attached zip file)

Code: Text
  1. TRAP LOCAL
  2. INCLUDE "curl.bac"
  3. INCLUDE "Regex-Plus"
  4.  
  5.  
  6. ' We store our data in this variable
  7. DECLARE data$, version_num$
  8. DECLARE regx TYPE mREGEX_type
  9.  
  10. FUNCTION progress_callback(long clientp, double dltotal, double dlnow, double ultotal, double ulnow )
  11.     LOCAL totaldotz, dotz, ii TYPE int
  12.     LOCAL fractiondownloaded TYPE double
  13.     LOCAL tmpStr$
  14.  
  15.     IF dltotal <= 0.0 THEN RETURN 0
  16.     tmpStr$ = CONCAT$((STRING)clientp,"")
  17.    
  18.     totaldotz = 40
  19.     fractiondownloaded = dlnow/dltotal
  20.     dotz = ROUND(fractiondownloaded * totaldotz)
  21.  
  22.     PRINT fractiondownloaded * 100 FORMAT "%3.0f%% [";
  23.  
  24.     WHILE ii < dotz DO
  25.         PRINT "=";
  26.         INCR ii
  27.     WEND
  28.  
  29.     WHILE ii < totaldotz DO
  30.         PRINT " ";
  31.         INCR ii
  32.     WEND
  33.  
  34.     PRINT "] \r";
  35.  
  36.     RETURN 0
  37. END FUNCTION
  38.  
  39. FUNCTION Save_Data(STRING buffer$, size_t size, size_t nmemb, void *userp)
  40.     data$ = CONCAT$(data$, buffer$)
  41.     RETURN size*nmemb
  42. END FUNCTION
  43.  
  44. SUB getData$(STRING url) TYPE STRING
  45.     LOCAL handle TYPE long
  46.     LOCAL success
  47.  
  48.     handle = curl_easy_init()
  49.     curl_easy_setopt(handle, CURLOPT_URL, url)
  50.     curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 1)
  51.     curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, Save_Data)
  52.    
  53.     success = curl_easy_perform(handle)
  54.     curl_easy_cleanup(handle)
  55. END SUB
  56.  
  57. SUB downloadFile(STRING url, STRING downloadFileName)
  58.     LOCAL handle TYPE long
  59.     LOCAL success
  60.  
  61.     PRINT "Downloading Latest 64Bit Firefox (",version_num$,") for Windows.\n"
  62.    
  63.     handle = curl_easy_init()    
  64.     OPEN downloadFileName FOR WRITING AS download
  65.     curl_easy_setopt(handle, CURLOPT_URL, url)
  66.     curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION,1)
  67.     curl_easy_setopt(handle, CURLOPT_WRITEDATA, download)
  68.     curl_easy_setopt(handle, CURLOPT_NOPROGRESS,0)
  69.     curl_easy_setopt(handle, CURLOPT_PROGRESSFUNCTION,progress_callback)
  70.  
  71.     success = curl_easy_perform(handle)
  72.     CLOSE FILE download
  73.     curl_easy_cleanup(handle)
  74.     PRINT "\n\nDownload Complete.\n"
  75. END SUB
  76.  
  77. CLEAR
  78.  
  79. PRINT "Firefox Download Challenge (Bacon Version) by AIR.\n"
  80.  
  81. getData$("https://www.mozilla.org/en-US/firefox/new")
  82.  
  83. regx = RegEx(data$,"data-latest-firefox=.([0-9]+\\.[0-9]+)")
  84.  
  85. IF regx.count THEN
  86.     version_num$ = regx.result$[0]
  87.     PRINT version_num$ FORMAT "Firefox Setup %s.exe" TO version$
  88. END IF
  89.  
  90. regx = RegEx(data$,".+href=\"(http.+latest-ssl&amp;os=win64&amp;lang=en-US)\"")
  91.  
  92. IF regx.count THEN
  93.     downloadLink$ = regx.result$[0]
  94. END IF
  95.  
  96. downloadFile(downloadLink$, version$)
  97. PRINT
  98.  
  99.  

AIR.

P.S.  Also like Peter's function, my RegEx function only returns a single (the first) hit.  I'll see about tweaking it so it will return multiple hits like most are used to seeing when using a library like PCRE...
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 30, 2018, 10:53:18 pm
BASIC doesn't get any thinner.  ;)

Script BASIC

Code: Script BASIC
  1. ' Firefox Download Challenge (Script BASIC Version) by JRS.
  2.  
  3. INCLUDE curl.bas
  4.  
  5. ch = curl::init()
  6.  
  7. curl::option(ch,"URL","https://www.mozilla.org/en-US/firefox/new/")
  8. wp = curl::perform(ch)
  9.  
  10. IF wp LIKE """*data-latest-firefox="*" data-esr-versions*<div id="other-platforms">*<li class="os_win64">*<a href="*"*""" THEN
  11.   PRINT "Downloading Latest 64Bit Firefox (",JOKER(2),") for Windows.\n"
  12.   curl::option(ch,"FOLLOWLOCATION", 1)
  13.   curl::option(ch,"NOPROGRESS",0)
  14.   curl::option(ch,"FILE","Firefox_Setup-" & JOKER(2) & ".exe")
  15.   curl::option(ch,"URL", JOKER(6))
  16.   curl::perform(ch)
  17.   PRINTNL
  18.   PRINT "Firefox_Setup-" & JOKER(2) & ".exe Downloaded ",FORMAT("%~##,###,###~ Bytes",curl::info(ch,"SIZE_DOWNLOAD")), _
  19.   " at ",FORMAT("%~##,###,###~ Bytes/Second",curl::info(ch,"SPEED_DOWNLOAD")),".\n"
  20. ELSE
  21.   PRINT "<< ERROR >>\n"
  22. END IF
  23.  
  24. curl::finish(ch)
  25.  


$ time scriba ffdl.sb
Downloading Latest 64Bit Firefox (63.0) for Windows.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   130  100   130    0     0    663      0 --:--:-- --:--:-- --:--:--   663
100 42.3M  100 42.3M    0     0  22.7M      0  0:00:01  0:00:01 --:--:-- 35.5M

Firefox_Setup-63.0.exe Downloaded 44,400,072 Bytes at 23,883,847 Bytes/Second.

real   0m2.174s
user   0m1.135s
sys   0m0.271s
$
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: erosolmi on October 30, 2018, 11:10:18 pm
Example in thinBasic.
Attached source code and executable

Ciao
Eros

Code: thinBasic
  1. uses "Console"
  2. uses "iNet"
  3. uses "OS"
  4.  
  5. printl "Connect to Firefox web site, determine latest Firefox version and download it"
  6.  
  7. '---Check if computer is connected to internet
  8. if INET_GetState Then
  9.   '---Get Firefox web site page
  10.   string Firefox_Info_Page = INET_UrlGetString("https://www.mozilla.org/en-US/firefox/new/")
  11.  
  12.   '---Get Firefox available version
  13.   string Firefox_Available_Version = grab$(Firefox_Info_Page, "data-latest-firefox=""", """")
  14.  
  15.   '---Get Win64 direct download url
  16.   string Firefox_DownLoad_Url_Win64 = grab$(Firefox_Info_Page, "<div id=""other-platforms"">", "</div>")
  17.   Firefox_DownLoad_Url_Win64 = grab$(Firefox_DownLoad_Url_Win64, "<li class=""os_win64"">", "</li>")
  18.   Firefox_DownLoad_Url_Win64 = grab$(Firefox_DownLoad_Url_Win64, "<a href=""", """")
  19.  
  20.   printl "Firefox available version...:", Firefox_Available_Version
  21.   printl "Firefox Download url........:", Firefox_DownLoad_Url_Win64
  22.  
  23.   '---Download
  24.   string sLocalFile = APP_SourcePath + "Firefox Setup " + Firefox_Available_Version + ".exe"
  25.   printl "Downlaoding to", sLocalFile, "..."
  26.   INET_UrlDownload(Firefox_DownLoad_Url_Win64, sLocalFile)
  27.   printl "File downlaoded."
  28.   printl
  29.   printl "Press ESC to end or any other key within 10 secs to execute setup"
  30.  
  31.   '---Ask if execute setup
  32.   string sUserReply = WaitKey(10)
  33.   if sUserReply <> "[ESC]" and sUserReply <> "[TIMEOUT]" Then
  34.     printl "Executing " + sLocalFile
  35.     OS_Shell(sLocalFile, %OS_WNDSTYLE_NORMAL, %OS_SHELL_SYNC)
  36.   Else
  37.     printl "No execution. Execute manually if needed."
  38.   end If
  39.  
  40. Else
  41.   printl "! it seems you are not connected to internet." in %CCOLOR_FLIGHTRED
  42. end If
  43.  
  44. printl "---All done, press a key to end or wait 5 secs---"
  45. WaitKey(5)
  46.  
  47. '[todo] To add async download with any progress info ...
  48. '       Install thinBasic and see example at \thinBasic\SampleScripts\INet\Internet_FileDownload.tbasic
  49.  

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 30, 2018, 11:24:03 pm
Quote
BASIC doesn't get any thinner.

It worked and brought the real thinBasic to us.  ;D

Thanks for your contribution!

I changed your code block from. =SB to =thinbasic. I normally don't edit others posts but TB highlighting looks much better. I attached the geshi syntax highlighting file for thinBasic I'm using here on the forum. It might be pretty old. If you want to have a peek and post an updated copy that would be great.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 01:49:30 am
BASIC doesn't get any thinner.  ;)

Script BASIC

Pretty Tight, John!

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 01:50:46 am
Example in thinBasic.
Attached source code and executable

Ciao
Eros


Very Nice, Eros!!!

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 12:05:14 pm
My PYTHON version, using only modules that come with a standard installation...

Code: Python
  1. #!/usr/bin/env python
  2.  
  3. import urllib2,re,sys
  4.  
  5. html = urllib2.urlopen('https://mozilla.org/firefox/new').read()
  6.  
  7. version = re.findall('data-latest-firefox="(\d+\.\d+)',html)[0]
  8. download_link = re.findall('.+href="(.+latest-ssl.+os=win64.+US).+',html)[0]
  9.  
  10. sys.stdout.write ("\33[2J\33[HFirefox Download Challenge (Python Version) by AIR.\n\n")
  11. print "Downloading Latest 64Bit Firefox (" + version + ") for Windows.\n"
  12.  
  13. f = urllib2.urlopen(download_link)
  14. with open("Firefox Setup " + version + ".exe", "wb") as dl:
  15.     total_size = int(f.info().getheader('Content-Length').strip())
  16.     downloaded = 0
  17.  
  18.     while True:
  19.         buffer = f.read(8192)
  20.         if not buffer:
  21.             print
  22.             break
  23.         downloaded += len(buffer)
  24.         dl.write(buffer)
  25.         percent = round( (float(downloaded) / total_size)*100,2)
  26.         sys.stdout.write("Downloaded %d of %d bytes (%0.2f%%)\r" % (downloaded, total_size, percent))
  27.     print "\nDownload Complete.\n"

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 31, 2018, 01:44:22 pm
So far SB followed by Bash are my leading challenge contenders.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 01:57:37 pm
That's cool, but remember this is not a competition between languages.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 31, 2018, 03:30:34 pm
I agree and it's a great way to see how different languages approach a common task.


Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 08:07:51 pm
OBJECTIVE C

I wanted to show this, because I think most people who don't use ObjC realize how easy it is to incorporate C code.

Code: Objective-C
  1. /*
  2.         Download the latest Firefox 64bit for Windows, Objective C version
  3.        
  4.         Written by AIR.
  5.  
  6.         This shows how Objective C can leverage C and C libraries
  7.         within the same project.
  8.  
  9.         compile with: gcc -framework Foundation getFirefox.m -lcurl -O2 -o getFirefox
  10. */
  11.  
  12. #import <Foundation/Foundation.h>
  13. #import <curl/curl.h>
  14.  
  15. NSString *version;
  16.  
  17. int progress_callback(id clientp, double dltotal, double dlnow, double ultotal, double ulnow ) {
  18.     printf("Downloaded %d of %d bytes\r",(int)dlnow,(int)dltotal);
  19.         return 0;
  20. }
  21.  
  22. void downloadFile(NSString *url, NSString *downloadFileName) {
  23.     id handle;
  24.     int success;
  25.         FILE *fp;
  26.  
  27.         printf("Downloading Latest 64Bit Firefox (%s) for Windows.\n\n",version.UTF8String);
  28.  
  29.     handle = curl_easy_init();    
  30.         if ((fp = fopen(downloadFileName.UTF8String, "wb+"))) {
  31.                 curl_easy_setopt(handle, CURLOPT_URL, url.UTF8String);
  32.                 curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION,1);
  33.                 curl_easy_setopt(handle, CURLOPT_WRITEDATA, fp);
  34.                 curl_easy_setopt(handle, CURLOPT_NOPROGRESS,0);
  35.                 curl_easy_setopt(handle, CURLOPT_PROGRESSDATA,downloadFileName.UTF8String);
  36.                 curl_easy_setopt(handle, CURLOPT_PROGRESSFUNCTION,progress_callback);
  37.        
  38.                 success = curl_easy_perform(handle);
  39.                 fclose(fp);
  40.         }
  41.     curl_easy_cleanup(handle);
  42.     printf("\n\nDownload of '%s' Complete.\n\n", downloadFileName.UTF8String);
  43. }
  44.  
  45. NSString *regex(NSString *src, NSString *query) {
  46.         NSRegularExpression *regex;
  47.         NSString *result;
  48.         NSArray *matches;
  49.  
  50.         regex = [NSRegularExpression regularExpressionWithPattern:query options:NSRegularExpressionCaseInsensitive error:nil];
  51.         matches = [regex matchesInString:src options:0 range:NSMakeRange(0, [src length])];
  52.         result = [src substringWithRange:[(NSTextCheckingResult*)matches[0] rangeAtIndex:1]];
  53.  
  54.         return result;
  55. }
  56.  
  57.  
  58.  
  59. int main(int argc, char const *argv[]) {
  60.         NSURL *url;
  61.         NSString *htmlContent, *downloadLink;
  62.  
  63.         printf("\33[2J\33[HFirefox Download Challenge (Objective C Version) by AIR.\n\n");
  64.  
  65.         url = [NSURL URLWithString:@"https://mozilla.org/firefox/new"];
  66.  
  67.         htmlContent = [NSString stringWithContentsOfURL:url usedEncoding:nil error:nil];
  68.  
  69.         version = regex(htmlContent,@"data-latest-firefox=.(\\d+\\.\\d+)");
  70.         downloadLink = regex(htmlContent,@".+href=.(.+latest-ssl.+win64.+US).+");
  71.  
  72.         downloadFile(downloadLink,[NSString stringWithFormat:@"Firefox Setup %@.exe",version]);
  73.         return 0;
  74. }
  75.  
  76.  

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 31, 2018, 08:13:16 pm
These code examples make me appreciate high level languages like BASIC even more.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on October 31, 2018, 08:27:53 pm
What I like about examples like these is that they can lead to asking "Why didn't I think of that?"

Eros' use of "GRAB$" made me take another look at the Bacon manual.  It's called "INBETWEEN$" in Bacon and works the same (I rewrote my submission for Bacon to test that.  Haven't submitted it though).

It also got me thinking about how to implement something like that myself.  After a little research I was able to put something together in C++ that works. Maybe I'll dust off JADE and see what happens.... ;D

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on October 31, 2018, 08:37:40 pm
Quote
Maybe I'll dust off JADE and see what happens.... ;D

I would love to see that lost gem resurface.

The SB final version is due to your experimentation with cURL in other languages. Thanks!

40% of the visitors to the forum use Firefox. (dominant browser)
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on November 01, 2018, 01:38:07 am
Quote
Maybe I'll dust off JADE and see what happens.... ;D

I would love to see that lost gem resurface.

JADE VERSION

Code: C++
  1. #include "jade.h"
  2.  
  3. /* compile with: g++ -std=c++11 getFirefox.cpp -DUSE_CURL -lcurl -O2 -o getFirefox */
  4.  
  5. /* Clone my repository for the JADE system itself: git clone https://bitbucket.org/Airr/jade.git */
  6.  
  7. MAIN
  8.     DEF CSTRING HTML$,VERSION$,DL_LINK$, URL$("https://mozilla.org/firefox/new");
  9.  
  10.     CLS;
  11.     PRINT("Firefox Download Challenge (**JADE** Version) by AIR.");
  12.     PRINTNL;
  13.  
  14.     HTML$ = downloadPage(URL$);
  15.  
  16.     VERSION$ = REGEX(HTML$, "data-latest-firefox=.(\\d+\\.\\d+).");
  17.     DL_LINK$ = REGEX(HTML$, ".+href=.(.+latest-ssl.+win64.+US).+");
  18.  
  19.     PRINT("Downloading Latest 64Bit Firefox ("+VERSION$+") for Windows.");
  20.     PRINTNL;
  21.  
  22.     downloadFile(DL_LINK$,"Firefox Setup " + VERSION$ + ".exe");
  23. ENDMAIN
  24.  

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on November 01, 2018, 02:13:58 am
Nice! (cURL pre-processor defines)

JADE has captured the number 2 spot on my list.

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: erosolmi on November 01, 2018, 04:23:27 am
Here a thinBasic ASYNC version with callback function used to determine download progression.
Used only libraries released with thinBasic official setup

Progress data is shown on a pop up box that will be removed when download is finished.

Maybe a little bit complex but worth to post to share ideas.

Code: thinBasic
  1. uses "Console"
  2. uses "iNet"
  3. uses "OS"
  4.  
  5. printl "Connect to Firefox web site, determine latest Firefox version and download it"
  6.  
  7. '---Check if computer is connected to internet
  8. if INET_GetState Then
  9.   '---Get Firefox web site page
  10.   string Firefox_Info_Page = INET_UrlGetString("https://www.mozilla.org/en-US/firefox/new/")
  11.  
  12.   '---Get Firefox available version
  13.   string Firefox_Available_Version = grab$(Firefox_Info_Page, "data-latest-firefox=""", """")
  14.  
  15.   '---Get Win64 direct download url
  16.   string Firefox_DownLoad_Url_Win64 = grab$(Firefox_Info_Page, "<div id=""other-platforms"">", "</div>")
  17.   Firefox_DownLoad_Url_Win64 = grab$(Firefox_DownLoad_Url_Win64, "<li class=""os_win64"">", "</li>")
  18.   Firefox_DownLoad_Url_Win64 = grab$(Firefox_DownLoad_Url_Win64, "<a href=""", """")
  19.  
  20.   printl "Firefox available version...:", Firefox_Available_Version
  21.   printl "Firefox Download url........:", Firefox_DownLoad_Url_Win64
  22.  
  23.   '---Download
  24.   string sLocalFile = APP_SourcePath + "Firefox Setup " + Firefox_Available_Version + ".exe"
  25.  
  26.   '---Initializes an application's use of the WinINet Internet functions.
  27.   dword hInternet = INET_Internet_Open("Download Fireforx", %INTERNET_OPEN_TYPE_DIRECT)
  28.   IF hInternet = %NULL THEN
  29.     printl "INET_Internet_Open error"
  30.     waitkey
  31.     stop
  32.   END IF
  33.  
  34.   long MyContext = 1
  35.   long hFile = INET_Internet_OpenUrl(hInternet, Firefox_DownLoad_Url_Win64, "", %INTERNET_FLAG_NO_CACHE_WRITE, MyContext)
  36.   IF hFile = %NULL THEN
  37.     printl "INET_Internet_OpenUrl error. Check url: " & Firefox_DownLoad_Url_Win64
  38.     INET_Internet_CloseHandle hInternet
  39.     waitkey
  40.     stop
  41.   END IF
  42.  
  43. '---Set a callback for hInternet events. Driver will automatically call MyStatusFunction on status change
  44.   long bResult = INET_Internet_SetStatusCallBack(hFile, MyStatusFunction)
  45.  
  46. '---Query remote server about the size of the file we are going to download
  47.   long FileSize = INET_Http_QueryInfo(hFile, %HTTP_QUERY_CONTENT_LENGTH)
  48.   printl "Real file size in bytes: " & FileSize
  49.   IF FileSize = %NULL THEN
  50.     printl "INET_Http_QueryInfo(hInternet, %HTTP_QUERY_CONTENT_LENGTH) error. Check url: " & Firefox_DownLoad_Url_Win64
  51.     INET_Internet_CloseHandle hInternet
  52.     INET_Internet_CloseHandle hFile
  53.     waitkey
  54.     stop
  55.   END IF
  56.  
  57.    
  58. '---Ok we can start a loop for file download. This can be automated in a loop or in a timer function
  59. '---Reads the data in %BufferLen bytes chunks
  60.   double T0 = timer
  61.   double T1
  62.  
  63.   '---Setup a %BufferLen bytes buffer
  64.     long  lBufferLen    = 500000 '500k
  65.     string strBuffer    = STRING$(lBufferLen, $SPC)
  66.     dword pstrBuffer    = STRPTR(strBuffer)
  67.     String strData
  68.     dword cbBytesRead  
  69.     DWord TotalBytesRead
  70.     long  lScreenLine   = 8
  71.     long  lScreenPos    = 20
  72.  
  73.   string sSave = Console_SaveScreen(1, 1, 80, 24)
  74.   Console_Box(lScreenPos - 1, lScreenLine - 1, 40, 8, 24, 24, "Downloading ...", 26, %Console_BOX_FLAG_3DOFF)
  75.   DO
  76.     bResult = INET_Internet_ReadFile(hFile, pstrBuffer, len(strBuffer), cbBytesRead)
  77.     IF bResult = 0 OR cbBytesRead = 0 THEN EXIT DO
  78.     IF cbBytesRead = LEN(strBuffer) THEN
  79.       strData &= strBuffer
  80.     ELSE
  81.       strData &= LEFT$(strBuffer, cbBytesRead)
  82.     END IF
  83.     TotalBytesRead += cbBytesRead
  84.     T1 = Timer
  85.    
  86.     PrintAt "Elapsed seconds  : " & LSet$(Format$(T1-T0, "0"), 19)                                         , lScreenPos + 1, lScreenLine + 1, 24
  87.     PrintAt "Bytes read so far: " & LSet$(TotalBytesRead, 19)                                              , lScreenPos + 1, lScreenLine + 2, 24
  88.     PrintAt "KBytes per second: " & LSet$(Format$((TotalBytesRead / Max(T1-T0, 1))/1024, "0.00"), 19)      , lScreenPos + 1, lScreenLine + 3, 24
  89.     PrintAt "Estimated seconds: " & LSet$(Format$((FileSize * (T1-T0))/TotalBytesRead, "0"), 19)           , lScreenPos + 1, lScreenLine + 4, 24
  90.     PrintAt "Seconds to go    : " & LSet$(Format$(((FileSize * (T1-T0))/TotalBytesRead)-(T1-T0), "0"), 19) , lScreenPos + 1, lScreenLine + 5, 24
  91.    
  92.     Console_ProgressBar(1, lScreenPos + 1, lScreenLine + 7, 40-1, 27, 1, FileSize, TotalBytesRead)
  93.   LOOP
  94.  
  95.   Console_RestoreScreen(1, 1, 80, 24, sSave)
  96.  
  97. '---Closes all used handles
  98.   INET_Internet_CloseHandle hInternet
  99.   INET_Internet_CloseHandle hFile
  100.  
  101.  
  102.   printl "Saving file: " & sLocalFile
  103.   Save_File(sLocalFile, strData)
  104.  
  105.   printl "File saved."
  106.   printl
  107.   printl "Press ESC to end or any other key within 10 secs to execute setup"
  108.  
  109.  
  110.   '---Ask if execute setup
  111.   string sUserReply = WaitKey(10)
  112.   if sUserReply <> "[ESC]" and sUserReply <> "[TIMEOUT]" Then
  113.     printl "Executing " + sLocalFile
  114.     OS_Shell(sLocalFile, %OS_WNDSTYLE_NORMAL, %OS_SHELL_SYNC)
  115.   Else
  116.     printl "No execution. Execute manually if needed."
  117.   end If
  118.  
  119. Else
  120.   printl "! it seems you are not connected to internet." in %CCOLOR_FLIGHTRED
  121. end If
  122.  
  123. printl "---All done, press a key to end or wait 5 secs---"
  124. WaitKey(5)
  125.  
  126.  
  127. '------------------------------------------------------------------
  128. ' CallBack function automatically called by Internet driver when
  129. ' status change for a specific Internet handle to which a callback
  130. ' function has been installed.
  131. ' Parameters are mandatory and are defined my Microsoft
  132. ' callback template. More info at: http://msdn.microsoft.com/en-us/library/aa385121(VS.85).aspx
  133. '------------------------------------------------------------------
  134. callback function MyStatusFunction( _
  135.                 byval hInternet                 as long , _   '---Mandatory in thinBasic and in API
  136.                 byval dwContext                 as long , _   '---Mandatory in thinBasic and in API
  137.                 byval dwInternetStatus          as long , _   '---Mandatory in thinBasic and in API
  138.                 byval lpvStatusInformation      as long , _   '---Optional in thinBasic, mandatory in API
  139.                 byval dwStatusInformationLength as long   _   '---Optional in thinBasic, mandatory in API
  140.              ) as long
  141. '------------------------------------------------------------------
  142.   dim BytesReceived     as dword at 0
  143.   static TotBytes       as dword
  144.  
  145.   static MoreStatusLine as long = 20
  146.      
  147.   'printat "Data from " & function_name & " callback"          , 1, 17
  148.   'printat "  Internet handle: " & hInternet & "  Internet dwContext: " & dwContext, 1, 18
  149.  
  150.   select case dwInternetStatus
  151.     case %INTERNET_STATUS_RECEIVING_RESPONSE
  152.  
  153.     case %INTERNET_STATUS_RESPONSE_RECEIVED
  154.       '---  Successfully received a response from the server.
  155.       '---  The lpvStatusInformation parameter points to a DWORD value that contains the number, in bytes, received.
  156.       setat(BytesReceived, lpvStatusInformation)
  157.       TotBytes += BytesReceived
  158.  
  159.       '---ATTENTION: TotalBytes received can differ from total bytes of the file to be downloaded
  160.  
  161.       'printat "  Receiving data (" & dwInternetStatus & " " & INET_Internet_StatusGetDescription(dwInternetStatus) & ") TotBytes: " & TotBytes, 1, 19
  162.  
  163.     case else
  164.       '---In case of other status, just decode it at the bottom.
  165.       'printat "  (" & dwInternetStatus & " " & INET_Internet_StatusGetDescription(dwInternetStatus) & ")", 1, MoreStatusLine
  166.       incr MoreStatusLine
  167.  
  168.   end select            
  169.  
  170.  
  171. end function
  172.  
  173.  

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on November 01, 2018, 09:47:00 am
Very Cool, Eros!

Question:  Is your Console_Box constructed via WinAPI calls, or are you using a 3rd party library?

AIR.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: erosolmi on November 01, 2018, 10:31:36 am
All developed by me with only Windows console api:
WriteConsoleOutputCharacter
FillConsoleOutputAttribute
ReadConsoleOutputCharacter

Console module was one of the first module I develop in 2004 when I started to work on thinBasic
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on December 12, 2018, 12:19:18 am
Since I'm working on updating MBC, here's an MBC version:

Code: Text
  1. $MODULE "curl.inc"
  2.  
  3. CLS
  4. Dim HTML$,dl_version$,dl_link$,fname$, match as REGEX
  5.  
  6. HTML$ = downloadPage$("https://www.mozilla.org/en-US/firefox/new")
  7.  
  8. if bcxregex(HTML$,"data-latest-firefox=.([0-9]+.[0-9]+\.?[0-9]?)",&match) then dl_version$ = match.results[1]
  9. if bcxregex(HTML$,".+(http.+ssl[&a-z;=-]+win64[&a-z;=-]+)",&match) then dl_link$ = match.results[1]
  10.  
  11. Print E"Firefox Download Challenge (MBC Version) by AIR.\n"
  12. Print E"Downloading Latest 64Bit Firefox (",dl_version,") for Windows.\n"
  13.  
  14. downloadFile(dl_link, "Firefox Setup-" & dl_version$ & ".exe")

Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on December 12, 2018, 12:43:57 am
Old challenges never die, they only get more interesting.

Curious. Can you time your MBC and SB Firefox download challenges for a reference?
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: John on March 05, 2019, 03:46:47 pm
I thought I would mention that one should reinitialize curl between commits. Curl was sending a previous POST buffer with a following GET.
Title: Re: Programatically download the latest 64 bit version of Firefox for Windows.
Post by: AIR on March 07, 2019, 12:24:46 am
Go version:

Code: Go
  1. package main
  2.  
  3. import (
  4.         "fmt"
  5.         "io"
  6.         "io/ioutil"
  7.         "net/http"
  8.         "os"
  9.         "regexp"
  10.         "strings"
  11.  
  12.         "github.com/dustin/go-humanize"
  13. )
  14.  
  15. // WriteCounter counts the number of bytes written to it. It implements to the io.Writer
  16. // interface and we can pass this into io.TeeReader() which will report progress on each
  17. // write cycle.
  18. type WriteCounter struct {
  19.         Total    uint64
  20.         Filesize uint64
  21. }
  22.  
  23. func main() {
  24.         clear()
  25.         fmt.Printf("Firefox Download Challenge (GO Version) by AIR.\n\n")
  26.         url := "https://mozilla.org/firefox/new"
  27.  
  28.         // Retrieve the data from the url
  29.         resp, err := http.Get(url)
  30.         chkError(err)
  31.  
  32.         // Auto close connection
  33.         defer resp.Body.Close()
  34.  
  35.         // reads html as a slice (array) of bytes
  36.         html, err := ioutil.ReadAll(resp.Body)
  37.         chkError(err)
  38.  
  39.         // regex to extract current version
  40.         re, err := regexp.Compile(`data-latest-firefox="([0-9]+\.[0-9]+?\.[0-9]+)"`)
  41.         chkError(err)
  42.  
  43.         // Regex to extract the download link
  44.         re2, err := regexp.Compile(`.+href="(.+latest-ssl.+os=win64.+US).+`)
  45.         chkError(err)
  46.  
  47.         // Perform search for version and download link
  48.         ffVersion := re.FindStringSubmatch(string(html))
  49.         ffLink := re2.FindStringSubmatch(string(html))
  50.  
  51.         // Download the Windows installer
  52.         err = DownloadFile("Firefox Setup "+ffVersion[1]+".exe", ffLink[1])
  53.         chkError(err)
  54.  
  55.         fmt.Println("\nDownload Complete.")
  56. }
  57.  
  58. func chkError(e error) {
  59.         if e != nil {
  60.                 panic(e)
  61.         }
  62. }
  63.  
  64. // Clears screen (Linux/macOS)
  65. func clear() {
  66.         fmt.Print("\033[2J\033[H")
  67. }
  68.  
  69. // DownloadFile will download a url to a local file. It's efficient because it will
  70. // write as it downloads and not load the whole file into memory. We pass an io.TeeReader
  71. // into Copy() to report progress on the download.
  72. func DownloadFile(filepath string, url string) error {
  73.  
  74.         // Create the file
  75.         out, err := os.Create(filepath)
  76.         if err != nil {
  77.                 return err
  78.         }
  79.         defer out.Close()
  80.  
  81.         // Get the data
  82.         resp, err := http.Get(url)
  83.         if err != nil {
  84.                 return err
  85.         }
  86.         defer resp.Body.Close()
  87.  
  88.         // Create our progress reporter (with reported Filesize set) and pass it to be used alongside our writer
  89.         counter := &WriteCounter{Filesize: uint64(resp.ContentLength)}
  90.  
  91.         _, err = io.Copy(out, io.TeeReader(resp.Body, counter))
  92.         if err != nil {
  93.                 return err
  94.         }
  95.  
  96.         // The progress outputs on a single line so print a new line once it's finished downloading
  97.         fmt.Print("\n")
  98.  
  99.         return nil
  100. }
  101.  
  102. // Callback used by WriteCounter
  103. func (wc *WriteCounter) Write(p []byte) (int, error) {
  104.         n := len(p)
  105.         wc.Total += uint64(n)
  106.  
  107.         wc.PrintProgress()
  108.         return n, nil
  109. }
  110.  
  111. // PrintProgress displays the download status
  112. func (wc WriteCounter) PrintProgress() {
  113.         // Clear the line by using a character return to go back to the start and remove
  114.         // the remaining characters by filling it with spaces
  115.         fmt.Printf("\r%s", strings.Repeat(" ", 30))
  116.  
  117.         // Return again and print current status of download
  118.         // We use the humanize package to print the status in a human-readable way (e.g. 10 MB)
  119.         fmt.Printf("\rDownloaded %s of %s", humanize.Bytes(wc.Total), humanize.Bytes(wc.Filesize))
  120.  
  121. }
  122.  

Output:
Firefox Download Challenge (GO Version) by AIR.

Downloaded 46 MB of 46 MB

Download Complete.


AIR.