PowerShell background jobs unlock scripting performance

What is a job in PowerShell?

In all cases, a job is a task that PowerShell runs asynchronously, meaning that it runs the command or script without affecting the prompt's availability. PowerShell has several background job types:

• RemoteJob. Commands and scripts that run through a remote session.
• ThreadJob. Commands and scripts that run in a separate thread of the same process as the parent PowerShell process.
• BackgroundJob. Commands and scripts that run in a separate process from the parent PowerShell process.

The choice between a ThreadJob and a BackgroundJob comes down to process isolation and performance. A script running in the same process, even as a different thread, can affect the parent process. If a ThreadJob encounters a terminating error, then it can affect the parent process. While this is not the case with a BackgroundJob, running in a separate process does add some overhead, including how objects are handled since you cannot receive a live object from a BackgroundJob.

Generally, because BackgroundJobs are more extensive, they are the recommended job type and are what are examined for the rest of this article.

The basics of managing PowerShell background jobs

PowerShell background job management is done with cmdlets that are part of the Microsoft.PowerShell.Core snap-in.

The easiest way to see the available commands is to query with Get-Command to get all commands that end with -Job, using the Format-Table alias for cleaner output.

Get-Command *-Job | ft -a

Here is the output from PowerShell 7.5.

And here is the output for Windows PowerShell.

Upon closer examination, you see Resume-Job and Suspend-Job are no longer available in PowerShell 7.5, but this tutorial provides examples that work with Windows PowerShell and PowerShell 7.

One of the great things about PowerShell is you can infer from cmdlet names what they do. Want to start a job? Let's try Start-Job.

Start-Job -ScriptBlock {Write-Host 'this command runs in a background job'}

There is no Write-Host output, but rather a job object. The background job runs asynchronously and requires specific commands to check on its status and find its output. If you didn't assign the output of that command to a variable, as in this example, pay attention to the Id. Use Get-Job to find the job information.

Get-Job -Id 4

Now, you can see the state marked as Completed.

To receive the data from the job, check that the HasMoreData property is True, indicating available output. Use Receive-Job to collect that data, assigning the output to a variable.

$jobOutput = Receive-Job -Id 4

In this case, the data was simply output from Write-Host. The $jobOutput variable is empty.

If you need to wait for a job to complete, which command can you use? Let's try Wait-Job.

Wait-Job -Id 4

PowerShell pauses until the job completes. In this case, it returns immediately. However, since you are in PowerShell, use the pipeline to pass the output of each command to the next command to make a nice one-liner.

Start-Job -ScriptBlock {Write-Host 'Pipeline'} | Wait-Job | Receive-Job

The Wait-Job cmdlet is useful when you need a specific job to complete before the next command.

Finally, to remove the job, use Remove-Job.

Remove-Job -Id 4

Any remaining jobs are removed when the PowerShell session closes.

How background jobs differ from Start-Process

If you think background jobs are a fancy wrapper around Start-Process, you aren't completely wrong. While they both run processes that are separate from the main PowerShell session, background jobs can continue to interact with the parent session to check the status and perform other actions. Start-Process does not have the same functionality.

Replicating the earlier example, try the following -- replacing pwsh for powershell in Windows PowerShell.

Start-Process pwsh -ArgumentList '-c Write-Host "this command runs in a background job"'

Running that command opens and closes the PowerShell console in a flash, leaving no indication if it worked.

Reproducing background jobs with Start-Process requires significant work to construct a redirect of standard input/output from the process and then perform object serialization that loads back into the original parent PowerShell process. All this effort is redundant to recreate the functionality that already exists in background jobs. You might prefer Start-Process to a background job for process isolation from the parent PowerShell session when launching a GUI application or running a process with different credentials.

How to use background jobs in scripts

Converting your PowerShell code to run as a background job might require some effort. Background jobs do not provide interactive input to the parent session. For example, checking the job in the following command gives a state of Blocked.

Start-Job -ScriptBlock {Read-Host 'Data please: ';Read-Host 'More data: '}

To pass data to a job, you can use the -ArgumentList parameter and a scriptblock that has a param() block.

Start-Job -ScriptBlock {param($string)Write-Host $string} -ArgumentList 'Output from job'

This screenshot shows the output from the string in the -ArgumentList parameter was passed and handled by the job.

Error handling also needs to be considered. If a stopping error occurs, the job fails. For example, if you run the following simple script and then query the job, it returns a Failed state.

Start-Job -ScriptBlock {Throw 'error'}
Get-Job -Id 14

Therefore, it is important to handle any errors and to only force the script to stop on errors if that is the desired behavior.

One approach to make this process significantly easier is to debug the script in an interactive session, paying attention to input, output and errors, and then add steps to log to a file to track execution information. The logging can include simple checkpoints to denote which sections of the script were executed or can take a more comprehensive approach that logs errors and object properties.

How to handle data from PowerShell background jobs

PowerShell is not a strongly typed programming language. This might not be a problem, but it is something to be aware of when working across different execution contexts. Objects returned from background jobs go through a serialization and deserialization process, which affects the functionality.

Receiving data from a background job is almost as easy as receiving data from any other cmdlet in PowerShell. Use Receive-Job to collect data output. For example, the following PowerShell command gets the file object.

Start-Job -ScriptBlock {Get-Item C:\tmp\test.txt} | Wait-Job | Receive-Job

The returned object looks like a legitimate FileInfo object, but double-check with the following code, which checks the type.

$file = Start-Job -ScriptBlock {Get-Item C:\tmp\test.txt} | Wait-Job | Receive-Job
$file.GetType()

If you do the same thing in the interactive session, then you get the expected type.

$file = Get-Item C:\tmp\test.txt
$file.GetType()

The results differ because PowerShell serializes the object and then deserializes the object when returning it from a background job. The properties or data are preserved, but the methods -- actions or functions invoked with parentheses -- are not available.

For example, these commands work in an interactive session but don't work if you receive the FileInfo object from a job, as shown here.

A background job is ideal for use with data-only objects, such as PSCustomObject.

How to scale PowerShell background jobs

While running a single PowerShell task in the background is helpful, the real benefit comes by running several background jobs concurrently, especially when time is of the essence.

For example, maybe you need to run gpupdate to apply Group Policy changes on all your servers ASAP. With a foreach loop and no background jobs, this task runs on each server sequentially as in the following PowerShell script.

$servers = Get-ADComputer -filter {OperatingSystem -like "*Windows Server*"}
foreach ($server in $servers) {
Invoke-Command -ComputerName $Server.Name -ScriptBlock {
gpupdate.exe /force
}
}

The script processes each server sequentially, performing the update on a server before it moves to the next. For an enterprise with hundreds or thousands of servers, this procedure could take a long time.

However, with background jobs, PowerShell can complete this quickly.

The first idea might be to use the -AsJob parameter on Invoke-Command and end up with the following.

$servers = Get-ADComputer -filter {OperatingSystem -like "*Windows Server*"}
foreach ($server in $servers) {
Invoke-Command -ComputerName $Server.Name -AsJob -ScriptBlock {
gpupdate.exe /force
}
}

Due to job overhead and depending on the number of servers, this script could overload the local machine. You can manage this by monitoring the number of jobs running with the following command:

(Get-Job -State Running).Count

The number of jobs that can be run simultaneously varies depending on the host system, but a good place to start for a simple script like this might be 20. Let's update the script accordingly and have it sleep for five seconds if the count is 20 or greater to prevent system overload.

$servers = Get-ADComputer -filter {OperatingSystem -like "*Windows Server*"}
foreach ($server in $servers) {
while ((Get-Job -State "Running").Count -ge 20) {
Start-Sleep -Seconds 5
}
Invoke-Command -ComputerName $Server.Name -AsJob -ScriptBlock {
gpupdate.exe /force
}
}

Now, you have a solution to apply Group Policy changes to all servers in AD in a quick and scalable fashion. Before running a script like this in your environment, properly source your servers. For example, you may have an organizational unit that you can use with the -SearchBase parameter to target specific servers rather than all of them.

Be aware that using Invoke-Command with the -AsJob parameter technically initiates a RemoteJob, not a BackgroundJob. However, a RemoteJob is handled in the same way as a BackgroundJob, so Get-Job, Wait-Job and Receive-Job still work with this method.

How to work with data at scale with background jobs

After running many jobs, it can be daunting to track the state of each of them, especially in the previous example where the script ran a command on every server in AD. To track status on a per-job basis, you can give each of them meaningful names in several ways:

• -Name parameter in Start-Job.
• -JobName parameter in Invoke-Command.
• Location property on the job itself.

If you are running a group of jobs that target specific resources, folders in file shares or IP addresses on a network, then you can track that information by using the -Name parameter in Start-Job.

$fileShares = Get-Content C:\temp\fileshares.txt
foreach ($share in $fileShares) {
Start-Job -Name $share -ScriptBlock {
param($share)
# do something with $share
} -ArgumentList $share
}

Running Get-Job shows the name of the share for each job.

Or, if you are working with a RemoteJob, as in the gpupdate example, you could either use the -JobName parameter or the Location property. For example, when running a job on a remote system, the name of the remote host in the Location property appears after running the following PowerShell command.

Invoke-Command -AsJob -ScriptBlock {write-host 'yes'} -ComputerName dc.domain.local

That enables you to easily run a report on the status of each job in a scalable fashion, such as the following example.

$report = while ((Get-Job -HasMoreData $true).Count -gt 0) {
foreach ($job in (Get-Job -HasMoreData $true)) {
[pscustomobject]@{
Share = $job.Name
JobState = $job.State
Data = Receive-Job -Job $job
}
}
}

If you have long-running jobs, then you might need to wait for them to finish before running the report.

Anthony Howell is an IT strategist with extensive experience in infrastructure and automation technologies. His expertise includes PowerShell, DevOps, cloud computing, and working in both Windows and Linux environments.