But, Wait! There’s More! Toying with PowerShell write speed.

Why hello old friend. It’s been a while. Sometimes life happens…

… and then you remember you actually still have a blog!

I have a random task in front of me at work where I have to do a lot of text manipulation. Obviously I’m going to PowerShell the crap out of this, but as I start framing out the script I realize I’m going to be writing a lot of strings. There is no speed requirements to this operation, but because I’m weird like that, I wanted to figure out what’s the fastest way to write a whole bunch of data to a file.

Here’s the wee little script I wrote to play around:

$arrDataToWrite=@()
$charSet="aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ0123456789".ToCharArray()

$StringLength = "400"
$FileLength = "10000"
$arrTestFile="C:\temp\arraytestfile.txt"
$lineTestFile="C:\temp\LineTestFile.txt"

#test one. Build array, output array in one shot.
Measure-Command {
  write-host "Test: create array of characters and output entire array"
  for($i=0; $i -le $FileLength; $i++){
    $arrDataToWrite+=$(-join $($charSet |get-random -Count $StringLength))
  }
  $arrDataToWrite |Out-File $arrTestFile
}

#test two. Iterate through, outputting as we go.
Measure-command{
  write-host "Test2: create characters and output inline"
  for($line=0; $line -le $FileLength; $line++){
      $(-join $($charSet |get-random -Count $StringLength))|Out-File $lineTestFile -Append
  }
}

Methodology was pretty simple.

• First test: build one big array full full of strings of random characters. After building the array, output the whole array to file
• Second test: build strings of random characters and use Out-File to append the strings as we loop

The results as you can see are pretty clear. After several test runs of varying sizes, appending strings to the file in-line is pretty consistently ~3x slower than the array method.  This makes sense when you think about the operations. In test two, every time I write to the file I have to access/open the file, write the data, close the file. If I’m counting right, that’s 3x the number of file operations… and that loop takes 3x longer…. huh…

Obviously it’s an over-simplification, but there are vastly more activities taking place in test two, hence the much longer execution time. Case closed, send this to print!

However, in the famous words of Ron Popeil: But wait! There’s more!

I typically do a litte of digging around before posting thoughts. Call it pragmatic, call it worrying, call it imposter syndrome, but reality is I just don’t want to stick my foot in my mouth. So I’m reading a few articles when I come across the article titled Slow Code: Top 5 Ways to Make Your PowerShell Scripts Run Faster from Ashely McGlone.

Sure enough, half way down this informative article  is a heading that reads “appending to arrays”. Well, that’s what we’re working on here! So I give this technique a shot, and voila my fastest run just got 50% faster! Here’s the code from test1 further optimized

#But wait, there's more!
#test three: Build the dataset within the loop and assign it to the array in one shot
$arrTestFile2="C:\temp\arraytestfile_fast.txt"
Measure-Command{
  write-host "Test3: assign all data to array in one shot and output entire array"
  $arrDataToWrite_Fast = for($j=0; $j -le $FileLength; $j++){
    $(-join $($charSet |get-random -Count $StringLength))
  }
  $arrDataToWrite_Fast |Out-File $arrTestFile2
}

As Mr. McGlone states in the linked article, the For loop runs and stores data in memory. By assigning that For loop to the array in one shot, you only have one expensive array operation rather than ‘N’ number of array operations.

As I stated at the outset, there’s no real reason behind this activity other than learning and having fun. To that end, I hope you keep learning, keep scripting and keep having fun.

Oh and Ashley, if you should by chance come across this, thanks for your article and for helping me learn something new!