Culling the commands log

[ncarmichael]

I use the commands a lot for checking disk space, certficates and 101 other jobs, this has left me with a log in JC of over 18,000 items.

What methods have you found for culling the logs of the old (and the successfull) so that all I am left with is bad news I need to action?

I was thinking of some powershell that I could fire off (as a scheduled command) that works withteh api to delete everything over a certain age with a 0 (success) code

I have tried raising a helpdesk ticket but the code supplied so far hasn't worked

[ncarmichael · Accepted solution]

Are you saying my log is overweight  ? 🙂

I did some debugging and have found so far that

* Remove-JCCommandResult doesn't limit what it returns meaning the dataset it downloads to my machine is all thirtyfive thousand items meaning it risks a timeout and is slow

* responseTime is not a datetime object so needs to be cast as such to be able to do a comparison

the comparitor should be greater-than not less-than

I have hacked your code to something that (sort of) works (I am not a programmer so its admittedly ugly)

Could you see if you programmers have a way to limit the resultset being sent from the server and have them turn my horrible hacking into something others could use?

# only work on logs younger than X days 
$days = 2 # can be 2 days or more 

# Set my key
#notshowingyouthat

$backtrackDays = (get-date).adddays(-$days)

Write-Output "$(get-date) Will cull any items that exited without error and occured after $backtrackDays"

# Gathering the cmd results for purging
$rawresults = Get-JCCommandResult 

Write-Output "$(get-date) Have gathered $($rawresults.Count) raw items"

$results = $rawresults | where {($_.exitCode -eq 0) -and (([System.DateTime]::ParseExact($_.responseTime.substring(0,$_.responseTime.indexof("T")),'yyyy-mm-dd',$null)) -gt $backtrackDays)} | select _id,responsetime


Write-Output "$(get-date) Have filtered them to  $($results.Count) items to cull"

# Removing
foreach ($r in $results){
    $count += 1 
    write-host "removing $($r._id), $count out of $($results.Count) "
    Remove-JCCommandResult -CommandResultID $r._id -force -Verbose
    
}

[shawnsong]

Hey Michael,

JC cmd results will store up to 30 days of logs, and we do have pwsh cmd to purge them, here is a quick example how to do it, based on the days you set for cut line:

# Set a date of back tracking
$backtrackDays = (get-date).adddays(-20)


$results = Get-JCCommandResult  | where {($_.exitcode -eq 0) -and ($_.responseTime -lt $backtrackDays)} | select _id,responsetime
foreach ($r in $results){

    Remove-JCCommandResult -CommandResultID $r._id
}

 you can run this on schedule from anywhere with JC pwsh module installed. 

[ncarmichael]

Hi, Unfortunatly that command just exited after sometime, without error and without affecting the number of results items. I assume it was a timeout as my command results currently number almost thirty-thousand.

I tried looking in the API documentation for the raw commands but without luck.

[shawnsong]

Hi Michael,

Good point about the amount of cmd results might clogged the run - I would suggest tweaking the `backtrackDays` variable to probably do 1 or 2 days to see if you can get a success. 

# set the days
$days = 1 # can be 2 days or more 
$backtrackDays = (get-date).adddays(-$days)

[ncarmichael]

We are one step closer as I do get an error message now "A parameter cannot be found that matches parameter name 'moduleBannerMessageCount'"

Presumably a dependancy is missing but its not something Google knows about (maybe I should ask chatgpt instead 🙂 )

[shawnsong]

I haven't see that error yet probably due to my tiny amount of results 😀 but I have made some tweaks by having the number of yours in mind, you may try the revision here 

# Set the days
$days = 1 # can be 2 days or more 
$backtrackDays = (get-date).adddays(-$days)

# Gathering the cmd results for purging
$results = Get-JCCommandResult | where {($_.exitcode -eq 0) -and ($_.responseTime -lt $backtrackDays)} | select _id,responsetime

# Removing
foreach ($r in $results){
    $count += 1 
    write-host "removing $($r._id), $count out of $($results.Count) "
    Remove-JCCommandResult -CommandResultID $r._id -force -Verbose
    
}

You may paste the output here if there is any error occurs too (hopefully not!). 

[ncarmichael · Accepted solution]

Are you saying my log is overweight  ? 🙂

I did some debugging and have found so far that

* Remove-JCCommandResult doesn't limit what it returns meaning the dataset it downloads to my machine is all thirtyfive thousand items meaning it risks a timeout and is slow

* responseTime is not a datetime object so needs to be cast as such to be able to do a comparison

the comparitor should be greater-than not less-than

I have hacked your code to something that (sort of) works (I am not a programmer so its admittedly ugly)

Could you see if you programmers have a way to limit the resultset being sent from the server and have them turn my horrible hacking into something others could use?

# only work on logs younger than X days 
$days = 2 # can be 2 days or more 

# Set my key
#notshowingyouthat

$backtrackDays = (get-date).adddays(-$days)

Write-Output "$(get-date) Will cull any items that exited without error and occured after $backtrackDays"

# Gathering the cmd results for purging
$rawresults = Get-JCCommandResult 

Write-Output "$(get-date) Have gathered $($rawresults.Count) raw items"

$results = $rawresults | where {($_.exitCode -eq 0) -and (([System.DateTime]::ParseExact($_.responseTime.substring(0,$_.responseTime.indexof("T")),'yyyy-mm-dd',$null)) -gt $backtrackDays)} | select _id,responsetime


Write-Output "$(get-date) Have filtered them to  $($results.Count) items to cull"

# Removing
foreach ($r in $results){
    $count += 1 
    write-host "removing $($r._id), $count out of $($results.Count) "
    Remove-JCCommandResult -CommandResultID $r._id -force -Verbose
    
}

[shawnsong]

ah, it seems the `responseTime` attribute of the cmd results you got has a different format when comparing with an in-place (get-date) - i'm running all these on Mac, it seems the datetime format is comparable despite they are showing in different format,  are you running pwsh on Linux? 

Btw, Nice tweak there for making it work for you! 

[ncarmichael]

Apart from a few "appliances" running AWS-Linux and the directors arm-based Mac's we are Windows all the way.

Is there any parameters we can put into Get-JCCommandResult  (for pass though to your backend) so that a limited amount of rows are returned so that the the script can work on "chunks" at a time. 

[shawnsong]

Thanks for the insights Michael. Win & MacOS shouldn't be the causing this error in this case.

you can try this cmdlet `Get-JcSdkCommandResult -paginate:$false` to get a maximum 100 results in response, you might want to tweak the attribute in the for_loop "_id" to "id" instead.

btw, what is your pwsh module version where you get the time format error? I heard this is known bug has been fixed in version 2.1.3 (January 06, 2023). 

[ncarmichael]

I am using 2.1.3 and will try what you say.

In my own scripts (that I send out using JC "commands") I use a declartion that downloads any modules it needs, is there a standard declartion for scripts using jc module that would also make sure it's on the latest version?

Something like this for example

$mod="jumpcloud"
if ( -not (get-installedModule $mod -ErrorAction SilentlyContinue)) { install-module $mod -force } else {update-module $mod}

[ncarmichael]

So my latest version is as follows, but I think there maybe some caching going on as it seems to always be the same 100 results

# only work on logs younger than X days 
$days = 30 # can be 2 days or more 

# Set my key
#notshowingyouthat

$mod="jumpcloud"
if ( -not (get-installedModule $mod -ErrorAction SilentlyContinue)) { install-module $mod -force } else {update-module $mod}

$backtrackDays = (get-date).adddays(-$days)

Write-Output "$(get-date) Will cull any items that exited without error and occured after $backtrackDays"

# Gathering the cmd results for purging
$rawresults = Get-JcSdkCommandResult -paginate:$false 

Write-Output "$(get-date) Have gathered $($rawresults.Count) raw items"

$results = $rawresults | where {($_.exitCode -eq 0) -and ($_.responseTime -gt $backtrackDays)} | select id,responsetime


Write-Output "$(get-date) Have filtered them to  $($results.Count) items to cull"

# Removing
foreach ($r in $results){
    $count += 1
    $del=$r.id	
    write-host "removing $del, $count out of $($results.Count) "
    write-host "Remove-JCCommandResult -CommandResultID $del -force"
	# -Verbose
    
}

[shawnsong]

I planted connect-jconline in my pwsh profile so every time the console relaunches it will bump up the version if there is a new release. 😉 To me the easiest way to keep up-to-date but sure there are many other ways to do it. 

so back to your latest mod of the script - it seems the deletion is not in motion when i look at the for-loop, thus the first page of 100 results remains:

    write-host "Remove-JCCommandResult -CommandResultID $del -force"

specifically this line will just print out the " Remove-JCComanndResult" cmd instead of executing it. maybe you can try without "write-host". 

[ncarmichael · Accepted solution]

Thankyou, the log is now small and only gives me bad news 🙂

I have JC run it as a command every hour, the final script is

# remove, in a chunk of no more than 100 items any old and successfully command log entries
# only reach back to X days in the past 
$days = 30 # can be 2 days or more 

# Set my key
$JumpCloudOrgId = "put my org id here"
$JumpCloudApiKey = "put my api key here"

Write-Output "installing dependancys if not there, otherwise upgrade"
$mod = "jumpcloud"
if (-not (Get-InstalledModule $mod -ErrorAction SilentlyContinue)) { Install-Module $mod -Force } else { Update-Module $mod }

Write-Output "connect to JC (this will also trigger an upgrade )"
Connect-JCOnline -Force -JumpCloudApiKey $JumpCloudApiKey -JumpCloudOrgId $JumpCloudOrgId -JCEnvironment "production"

#turn days to specific date
$backtrackDays = (Get-Date).adddays(- $days)

Write-Output "$(get-date) Will cull any items that exited without error and occured after $backtrackDays"

Write-Output "Gathering the cmd results for purging, this will be no more than be 100 items"
$rawresults = Get-JcSdkCommandResult -paginate:$false

Write-Output "$(get-date) Have gathered $($rawresults.Count) raw items"

$results = $rawresults | Where-Object { ($_.exitCode -eq 0) -and ($_.responseTime -gt $backtrackDays) } | Select-Object id,responsetime

Write-Output "$(get-date) Have filtered them to  $($results.Count) items to cull"

# Removing
foreach ($r in $results) {
  $count += 1
  $del = $r.id
  Write-Host "removing $del, $count out of $($results.Count) "
  Remove-JCCommandResult -CommandResultID $del -Force
  #add -Verbose to the previous line for more info

}

[shawnsong]

Awesome! Great to see we worked this out 😉

[ncarmichael]

Sadly I have just come back from seven days away from work to find 23k log entires waiting for me 😞

I suspect after running a few times it ends up with 100 failed tasks which it then gets stuck because it doesn't clear any of them.

I tried adding a date clause so that if is very old (even if its failed) it still gets culled but I think the date comparison needs some work 😞

[shawnsong]

can you share the error msg or the failed cmd results ?

[ncarmichael]

---

@shawnsong wrote:

can you share the error msg or the failed cmd results ?

It's simply as follows, I think the code will need to be written to go through each "record" 1 by 1 (in the for next loop) and report its "workings out" so we can understand its logic

Gathering the cmd results for purging, this will be no more than be 100 items
02/21/2023 12:35:52 Have gathered 100 raw items
02/21/2023 12:35:52 Have filtered them to 0 items to cull

[shawnsong]

That helps, thanks. 

you can simply just compare $results VS $rawresults, and you can adjust the $days at the very top to probably just 1 in the context  there are 100 results in just 1 day, and start culling then keep up. 

[ncarmichael]

I have broke out the comparison into in the loop to see what is going on, what is failing is the date comparison, the API is providing the date in the "27 February 2023 09:42:42" format which powershell doe's not recognise as datetime so does a comparison as text which then does not give the required result 😞

I am not sure where to go with this next as my powershell experiments to convert it to datetime have failed.

Is there a parameter that we could perhaps pass to the API either output the date as a datetime or chose a format that is easy for powershell to convert?

[shawnsong]

Actually the "get-jcsdkcommandresult" is returning a date time value for attribute "responseTime" - used for comparison here

$rawresults = Get-JcSdkCommandResult -paginate:$false
$results = $rawresults | Where-Object { ($_.exitCode -eq 0) -and ($_.responseTime -gt $backtrackDays) } | Select-Object id,responsetime

 I have double checked ^ the final version script and confirmed we are comparing the date time data:

And may I know are you on JC lounge slack workspace? if yes, let me know your name thus I can reach out, or you can ping me too. 

[ncarmichael]

Hi,

I have joined slack as requested.

My latest discovery is sometimes the date returned is null (perhaps because the job is still running or crashed out  rather than terminated cleanly) so I need to add handling for that into the script

[shawnsong]

I have responded to you as well in our slack lounge 😉

That's interesting, the unfinished cmds (in cmd queue) should not show up in cmd results, but yeah if you have null value in $rawresults.ResponseTime we can simply just filter it out like this: 

$results = $rawresults | Where-Object { ($_.exitCode -eq 0) -and ($null -ne $_.responseTime) -and ($_.responseTime -gt $backtrackDays) } | Select-Object id,responsetime