PoshCode Logo PowerShell Code Repository

Get-WebFile 3.6 (modification of post by Joel Bennett view diff)
diff | embed code: <script type="text/javascript" src="http://PoshCode.org/embed/417"></script>download | new post

An upgrade to my wget script which can output the downloaded html to the pipeline. Get-WebFile can download text or binary files, automatically determine file names, and present a progress report for large files…

  1. ## Get-WebFile (aka wget for PowerShell)
  2. ##############################################################################################################
  3. ## Downloads a file or page from the web
  4. ## History:
  5. ## v3.6 - Add -Passthru switch to output TEXT files
  6. ## v3.5 - Add -Quiet switch to turn off the progress reports ...
  7. ## v3.4 - Add progress report for files which don't report size
  8. ## v3.3 - Add progress report for files which report their size
  9. ## v3.2 - Use the pure Stream object because StreamWriter is based on TextWriter:
  10. ##        it was messing up binary files, and making mistakes with extended characters in text
  11. ## v3.1 - Unwrap the filename when it has quotes around it
  12. ## v3   - rewritten completely using HttpWebRequest + HttpWebResponse to figure out the file name, if possible
  13. ## v2   - adds a ton of parsing to make the output pretty
  14. ##        added measuring the scripts involved in the command, (uses Tokenizer)
  15. ##############################################################################################################
  16. function Get-WebFile {
  17.    param(
  18.       $url = (Read-Host "The URL to download"),
  19.       $fileName = $null,
  20.       [switch]$Passthru,
  21.       [switch]$quiet
  22.    )
  23.    
  24.    $req = [System.Net.HttpWebRequest]::Create($url);
  25.    $res = $req.GetResponse();
  26.  
  27.    if($fileName -and !(Split-Path $fileName)) {
  28.       $fileName = Join-Path (Get-Location -PSProvider "FileSystem") $fileName
  29.    }
  30.    elseif((!$Passthru -and ($fileName -eq $null)) -or (($fileName -ne $null) -and (Test-Path -PathType "Container" $fileName)))
  31.    {
  32.       [string]$fileName = ([regex]'(?i)filename=(.*)$').Match( $res.Headers["Content-Disposition"] ).Groups[1].Value
  33.       $fileName = $fileName.trim("\/""'")
  34.       if(!$fileName) {
  35.          $fileName = $res.ResponseUri.Segments[-1]
  36.          $fileName = $fileName.trim("\/")
  37.          if(!$fileName) {
  38.             $fileName = Read-Host "Please provide a file name"
  39.          }
  40.          $fileName = $fileName.trim("\/")
  41.          if(!([IO.FileInfo]$fileName).Extension) {
  42.             $fileName = $fileName + "." + $res.ContentType.Split(";")[0].Split("/")[1]
  43.          }
  44.       }
  45.       $fileName = Join-Path (Get-Location -PSProvider "FileSystem") $fileName
  46.    }
  47.    if($Passthru) {
  48.       $encoding = [System.Text.Encoding]::GetEncoding( $res.CharacterSet )
  49.       [string]$output = ""
  50.    }
  51.  
  52.    if($res.StatusCode -eq 200) {
  53.       [int]$goal = $res.ContentLength
  54.       $reader = $res.GetResponseStream()
  55.       if($fileName) {
  56.          $writer = new-object System.IO.FileStream $fileName, "Create"
  57.       }
  58.       [byte[]]$buffer = new-object byte[] 4096
  59.       [int]$total = [int]$count = 0
  60.       do
  61.       {
  62.          $count = $reader.Read($buffer, 0, $buffer.Length);
  63.          if($fileName) {
  64.             $writer.Write($buffer, 0, $count);
  65.          }
  66.          if($Passthru){
  67.             $output += $encoding.GetString($buffer,0,$count)
  68.          } elseif(!$quiet) {
  69.             $total += $count
  70.             if($goal -gt 0) {
  71.                Write-Progress "Downloading $url" "Saving $total of $goal" -id 0 -percentComplete (($total/$goal)*100)
  72.             } else {
  73.                Write-Progress "Downloading $url" "Saving $total bytes..." -id 0
  74.             }
  75.          }
  76.       } while ($count -gt 0)
  77.      
  78.       $reader.Close()
  79.       if($fileName) {
  80.          $writer.Flush()
  81.          $writer.Close()
  82.       }
  83.       if($Passthru){
  84.          $output
  85.       }
  86.    }
  87.    $res.Close();
  88.    if($fileName) {
  89.       ls $fileName
  90.    }
  91. }

Submit a correction or amendment below (click here to make a fresh posting)
After submitting an amendment, you'll be able to view the differences between the old and new posts easily.

Syntax highlighting:


Remember me