โ01-11-2023 04:09 PM
I use the commands a lot for checking disk space, certficates and 101 other jobs, this has left me with a log in JC of over 18,000 items.
What methods have you found for culling the logs of the old (and the successfull) so that all I am left with is bad news I need to action?
I was thinking of some powershell that I could fire off (as a scheduled command) that works withteh api to delete everything over a certain age with a 0 (success) code
I have tried raising a helpdesk ticket but the code supplied so far hasn't worked
Solved! Go to Solution.
โ01-20-2023 07:40 AM - edited โ01-20-2023 11:33 AM
Are you saying my log is overweight ? ๐
I did some debugging and have found so far that
* Remove-JCCommandResult doesn't limit what it returns meaning the dataset it downloads to my machine is all thirtyfive thousand items meaning it risks a timeout and is slow
* responseTime is not a datetime object so needs to be cast as such to be able to do a comparison
the comparitor should be greater-than not less-than
I have hacked your code to something that (sort of) works (I am not a programmer so its admittedly ugly)
Could you see if you programmers have a way to limit the resultset being sent from the server and have them turn my horrible hacking into something others could use?
# only work on logs younger than X days
$days = 2 # can be 2 days or more
# Set my key
#notshowingyouthat
$backtrackDays = (get-date).adddays(-$days)
Write-Output "$(get-date) Will cull any items that exited without error and occured after $backtrackDays"
# Gathering the cmd results for purging
$rawresults = Get-JCCommandResult
Write-Output "$(get-date) Have gathered $($rawresults.Count) raw items"
$results = $rawresults | where {($_.exitCode -eq 0) -and (([System.DateTime]::ParseExact($_.responseTime.substring(0,$_.responseTime.indexof("T")),'yyyy-mm-dd',$null)) -gt $backtrackDays)} | select _id,responsetime
Write-Output "$(get-date) Have filtered them to $($results.Count) items to cull"
# Removing
foreach ($r in $results){
$count += 1
write-host "removing $($r._id), $count out of $($results.Count) "
Remove-JCCommandResult -CommandResultID $r._id -force -Verbose
}
โ01-25-2023 07:01 AM
Thankyou, the log is now small and only gives me bad news ๐
I have JC run it as a command every hour, the final script is
# remove, in a chunk of no more than 100 items any old and successfully command log entries
# only reach back to X days in the past
$days = 30 # can be 2 days or more
# Set my key
$JumpCloudOrgId = "put my org id here"
$JumpCloudApiKey = "put my api key here"
Write-Output "installing dependancys if not there, otherwise upgrade"
$mod = "jumpcloud"
if (-not (Get-InstalledModule $mod -ErrorAction SilentlyContinue)) { Install-Module $mod -Force } else { Update-Module $mod }
Write-Output "connect to JC (this will also trigger an upgrade )"
Connect-JCOnline -Force -JumpCloudApiKey $JumpCloudApiKey -JumpCloudOrgId $JumpCloudOrgId -JCEnvironment "production"
#turn days to specific date
$backtrackDays = (Get-Date).adddays(- $days)
Write-Output "$(get-date) Will cull any items that exited without error and occured after $backtrackDays"
Write-Output "Gathering the cmd results for purging, this will be no more than be 100 items"
$rawresults = Get-JcSdkCommandResult -paginate:$false
Write-Output "$(get-date) Have gathered $($rawresults.Count) raw items"
$results = $rawresults | Where-Object { ($_.exitCode -eq 0) -and ($_.responseTime -gt $backtrackDays) } | Select-Object id,responsetime
Write-Output "$(get-date) Have filtered them to $($results.Count) items to cull"
# Removing
foreach ($r in $results) {
$count += 1
$del = $r.id
Write-Host "removing $del, $count out of $($results.Count) "
Remove-JCCommandResult -CommandResultID $del -Force
#add -Verbose to the previous line for more info
}
โ01-12-2023 07:20 PM
Hey Michael,
JC cmd results will store up to 30 days of logs, and we do have pwsh cmd to purge them, here is a quick example how to do it, based on the days you set for cut line:
# Set a date of back tracking
$backtrackDays = (get-date).adddays(-20)
$results = Get-JCCommandResult | where {($_.exitcode -eq 0) -and ($_.responseTime -lt $backtrackDays)} | select _id,responsetime
foreach ($r in $results){
Remove-JCCommandResult -CommandResultID $r._id
}
you can run this on schedule from anywhere with JC pwsh module installed.
โ01-16-2023 07:08 AM
Hi, Unfortunatly that command just exited after sometime, without error and without affecting the number of results items. I assume it was a timeout as my command results currently number almost thirty-thousand.
I tried looking in the API documentation for the raw commands but without luck.
โ01-18-2023 03:01 AM
Hi Michael,
Good point about the amount of cmd results might clogged the run - I would suggest tweaking the `backtrackDays` variable to probably do 1 or 2 days to see if you can get a success.
# set the days
$days = 1 # can be 2 days or more
$backtrackDays = (get-date).adddays(-$days)
โ01-18-2023 04:44 AM - edited โ01-18-2023 06:52 AM
We are one step closer as I do get an error message now "A parameter cannot be found that matches parameter name 'moduleBannerMessageCount'"
Presumably a dependancy is missing but its not something Google knows about (maybe I should ask chatgpt instead ๐ )
โ01-19-2023 08:50 PM
I haven't see that error yet probably due to my tiny amount of results ๐ but I have made some tweaks by having the number of yours in mind, you may try the revision here
# Set the days
$days = 1 # can be 2 days or more
$backtrackDays = (get-date).adddays(-$days)
# Gathering the cmd results for purging
$results = Get-JCCommandResult | where {($_.exitcode -eq 0) -and ($_.responseTime -lt $backtrackDays)} | select _id,responsetime
# Removing
foreach ($r in $results){
$count += 1
write-host "removing $($r._id), $count out of $($results.Count) "
Remove-JCCommandResult -CommandResultID $r._id -force -Verbose
}
You may paste the output here if there is any error occurs too (hopefully not!).
โ01-20-2023 07:40 AM - edited โ01-20-2023 11:33 AM
Are you saying my log is overweight ? ๐
I did some debugging and have found so far that
* Remove-JCCommandResult doesn't limit what it returns meaning the dataset it downloads to my machine is all thirtyfive thousand items meaning it risks a timeout and is slow
* responseTime is not a datetime object so needs to be cast as such to be able to do a comparison
the comparitor should be greater-than not less-than
I have hacked your code to something that (sort of) works (I am not a programmer so its admittedly ugly)
Could you see if you programmers have a way to limit the resultset being sent from the server and have them turn my horrible hacking into something others could use?
# only work on logs younger than X days
$days = 2 # can be 2 days or more
# Set my key
#notshowingyouthat
$backtrackDays = (get-date).adddays(-$days)
Write-Output "$(get-date) Will cull any items that exited without error and occured after $backtrackDays"
# Gathering the cmd results for purging
$rawresults = Get-JCCommandResult
Write-Output "$(get-date) Have gathered $($rawresults.Count) raw items"
$results = $rawresults | where {($_.exitCode -eq 0) -and (([System.DateTime]::ParseExact($_.responseTime.substring(0,$_.responseTime.indexof("T")),'yyyy-mm-dd',$null)) -gt $backtrackDays)} | select _id,responsetime
Write-Output "$(get-date) Have filtered them to $($results.Count) items to cull"
# Removing
foreach ($r in $results){
$count += 1
write-host "removing $($r._id), $count out of $($results.Count) "
Remove-JCCommandResult -CommandResultID $r._id -force -Verbose
}
โ01-20-2023 07:17 PM
ah, it seems the `responseTime` attribute of the cmd results you got has a different format when comparing with an in-place (get-date) - i'm running all these on Mac, it seems the datetime format is comparable despite they are showing in different format, are you running pwsh on Linux?
Btw, Nice tweak there for making it work for you!
โ01-23-2023 04:12 AM
Apart from a few "appliances" running AWS-Linux and the directors arm-based Mac's we are Windows all the way.
Is there any parameters we can put into Get-JCCommandResult (for pass though to your backend) so that a limited amount of rows are returned so that the the script can work on "chunks" at a time.
โ01-23-2023 07:32 PM
Thanks for the insights Michael. Win & MacOS shouldn't be the causing this error in this case.
you can try this cmdlet `Get-JcSdkCommandResult -paginate:$false` to get a maximum 100 results in response, you might want to tweak the attribute in the for_loop "_id" to "id" instead.
btw, what is your pwsh module version where you get the time format error? I heard this is known bug has been fixed in version 2.1.3 (January 06, 2023).
โ01-24-2023 04:42 AM - edited โ01-24-2023 05:48 AM
I am using 2.1.3 and will try what you say.
In my own scripts (that I send out using JC "commands") I use a declartion that downloads any modules it needs, is there a standard declartion for scripts using jc module that would also make sure it's on the latest version?
Something like this for example
$mod="jumpcloud"
if ( -not (get-installedModule $mod -ErrorAction SilentlyContinue)) { install-module $mod -force } else {update-module $mod}
โ01-24-2023 11:27 AM - edited โ01-24-2023 12:08 PM
So my latest version is as follows, but I think there maybe some caching going on as it seems to always be the same 100 results
# only work on logs younger than X days
$days = 30 # can be 2 days or more
# Set my key
#notshowingyouthat
$mod="jumpcloud"
if ( -not (get-installedModule $mod -ErrorAction SilentlyContinue)) { install-module $mod -force } else {update-module $mod}
$backtrackDays = (get-date).adddays(-$days)
Write-Output "$(get-date) Will cull any items that exited without error and occured after $backtrackDays"
# Gathering the cmd results for purging
$rawresults = Get-JcSdkCommandResult -paginate:$false
Write-Output "$(get-date) Have gathered $($rawresults.Count) raw items"
$results = $rawresults | where {($_.exitCode -eq 0) -and ($_.responseTime -gt $backtrackDays)} | select id,responsetime
Write-Output "$(get-date) Have filtered them to $($results.Count) items to cull"
# Removing
foreach ($r in $results){
$count += 1
$del=$r.id
write-host "removing $del, $count out of $($results.Count) "
write-host "Remove-JCCommandResult -CommandResultID $del -force"
# -Verbose
}
โ01-24-2023 07:47 PM
I planted connect-jconline in my pwsh profile so every time the console relaunches it will bump up the version if there is a new release. ๐ To me the easiest way to keep up-to-date but sure there are many other ways to do it.
so back to your latest mod of the script - it seems the deletion is not in motion when i look at the for-loop, thus the first page of 100 results remains:
write-host "Remove-JCCommandResult -CommandResultID $del -force"
specifically this line will just print out the " Remove-JCComanndResult" cmd instead of executing it. maybe you can try without "write-host".
โ01-25-2023 07:01 AM
Thankyou, the log is now small and only gives me bad news ๐
I have JC run it as a command every hour, the final script is
# remove, in a chunk of no more than 100 items any old and successfully command log entries
# only reach back to X days in the past
$days = 30 # can be 2 days or more
# Set my key
$JumpCloudOrgId = "put my org id here"
$JumpCloudApiKey = "put my api key here"
Write-Output "installing dependancys if not there, otherwise upgrade"
$mod = "jumpcloud"
if (-not (Get-InstalledModule $mod -ErrorAction SilentlyContinue)) { Install-Module $mod -Force } else { Update-Module $mod }
Write-Output "connect to JC (this will also trigger an upgrade )"
Connect-JCOnline -Force -JumpCloudApiKey $JumpCloudApiKey -JumpCloudOrgId $JumpCloudOrgId -JCEnvironment "production"
#turn days to specific date
$backtrackDays = (Get-Date).adddays(- $days)
Write-Output "$(get-date) Will cull any items that exited without error and occured after $backtrackDays"
Write-Output "Gathering the cmd results for purging, this will be no more than be 100 items"
$rawresults = Get-JcSdkCommandResult -paginate:$false
Write-Output "$(get-date) Have gathered $($rawresults.Count) raw items"
$results = $rawresults | Where-Object { ($_.exitCode -eq 0) -and ($_.responseTime -gt $backtrackDays) } | Select-Object id,responsetime
Write-Output "$(get-date) Have filtered them to $($results.Count) items to cull"
# Removing
foreach ($r in $results) {
$count += 1
$del = $r.id
Write-Host "removing $del, $count out of $($results.Count) "
Remove-JCCommandResult -CommandResultID $del -Force
#add -Verbose to the previous line for more info
}
โ01-25-2023 11:04 PM
Awesome! Great to see we worked this out ๐
โ02-20-2023 07:54 AM
Sadly I have just come back from seven days away from work to find 23k log entires waiting for me ๐
I suspect after running a few times it ends up with 100 failed tasks which it then gets stuck because it doesn't clear any of them.
I tried adding a date clause so that if is very old (even if its failed) it still gets culled but I think the date comparison needs some work ๐
โ02-20-2023 11:41 PM
can you share the error msg or the failed cmd results ?
โ02-21-2023 07:42 AM - edited โ02-21-2023 07:45 AM
@shawnsong wrote:can you share the error msg or the failed cmd results ?
It's simply as follows, I think the code will need to be written to go through each "record" 1 by 1 (in the for next loop) and report its "workings out" so we can understand its logic
Gathering the cmd results for purging, this will be no more than be 100 items
02/21/2023 12:35:52 Have gathered 100 raw items
02/21/2023 12:35:52 Have filtered them to 0 items to cull
โ02-22-2023 01:47 AM
That helps, thanks.
you can simply just compare $results VS $rawresults, and you can adjust the $days at the very top to probably just 1 in the context there are 100 results in just 1 day, and start culling then keep up.
โ02-27-2023 03:43 AM
I have broke out the comparison into in the loop to see what is going on, what is failing is the date comparison, the API is providing the date in the "27 February 2023 09:42:42" format which powershell doe's not recognise as datetime so does a comparison as text which then does not give the required result ๐
I am not sure where to go with this next as my powershell experiments to convert it to datetime have failed.
Is there a parameter that we could perhaps pass to the API either output the date as a datetime or chose a format that is easy for powershell to convert?
โ02-27-2023 05:47 AM
Actually the "get-jcsdkcommandresult" is returning a date time value for attribute "responseTime" - used for comparison here
$rawresults = Get-JcSdkCommandResult -paginate:$false
$results = $rawresults | Where-Object { ($_.exitCode -eq 0) -and ($_.responseTime -gt $backtrackDays) } | Select-Object id,responsetime
I have double checked ^ the final version script and confirmed we are comparing the date time data:
And may I know are you on JC lounge slack workspace? if yes, let me know your name thus I can reach out, or you can ping me too.
โ03-09-2023 05:54 AM
Hi,
I have joined slack as requested.
My latest discovery is sometimes the date returned is null (perhaps because the job is still running or crashed out rather than terminated cleanly) so I need to add handling for that into the script
โ03-09-2023 08:01 PM - edited โ03-09-2023 08:01 PM
I have responded to you as well in our slack lounge ๐
That's interesting, the unfinished cmds (in cmd queue) should not show up in cmd results, but yeah if you have null value in $rawresults.ResponseTime we can simply just filter it out like this:
$results = $rawresults | Where-Object { ($_.exitCode -eq 0) -and ($null -ne $_.responseTime) -and ($_.responseTime -gt $backtrackDays) } | Select-Object id,responsetime
New to the site? Take a look at these additional resources:
Ready to join us? You can register here.