locked #logging Possible Enhancement for the ALL.TXT file #logging


Steve Miller
 

Could WSJT-X be coded so that a new ALL.TXT file is created new when a threshold of number of records is exceeded? My current ALL.TXT file is too large for NotePad++ to open (870mb) so it is basically a throw away; I'll have to delete it and start a new file.

If the number of records threshold was implemented, at the start of WSJT-X - check if the ALL.TXT file exceeds the records threshold. If so, grab the last record in the file - get the date and time stamp of that last record and name change the current ALL.TXT to ALL_20220530_182530.TXT or similar. Then start a fresh ALL.TXT file again.

Just a thought.

Steve, K8SJM


Michael Black
 

I think the log file should be done by month with an appropriate name format that shows the month.
e.g. 202205_ALL.TXT
Mike W9MDB

On Monday, May 30, 2022, 04:19:16 PM CDT, Steve Miller <k8sjm@...> wrote:

Could WSJT-X be coded so that a new ALL.TXT file is created new when a threshold of number of records is exceeded? My current ALL.TXT file is too large for NotePad++ to open (870mb) so it is basically a throw away; I'll have to delete it and start a new file.

If the number of records threshold was implemented, at the start of WSJT-X - check if the ALL.TXT file exceeds the records threshold. If so, grab the last record in the file - get the date and time stamp of that last record and name change the current ALL.TXT to ALL_20220530_182530.TXT or similar. Then start a fresh ALL.TXT file again.

Just a thought.

Steve, K8SJM


Bruce - K5WBM
 

I rename my all.txt file at the end of the year to keep the file size reasonable i.e all 2021 information is in the 2021 all.txt file. That also allows you to go back and look for a particular qso by year if needed.

Bruce - K5WBM

On May 30, 2022, at 4:19 PM, Steve Miller <k8sjm@...> wrote:

Could WSJT-X be coded so that a new ALL.TXT file is created new when a threshold of number of records is exceeded? My current ALL.TXT file is too large for NotePad++ to open (870mb) so it is basically a throw away; I'll have to delete it and start a new file.

If the number of records threshold was implemented, at the start of WSJT-X - check if the ALL.TXT file exceeds the records threshold. If so, grab the last record in the file - get the date and time stamp of that last record and name change the current ALL.TXT to ALL_20220530_182530.TXT or similar. Then start a fresh ALL.TXT file again.

Just a thought.

Steve, K8SJM





Martin G0HDB
 

On Mon, May 30, 2022 at 10:19 PM, Steve Miller wrote:

Could WSJT-X be coded so that a new ALL.TXT file is created new when a
threshold of number of records is exceeded? My current ALL.TXT file is too
large for NotePad++ to open (870mb) so it is basically a throw away; I'll have
to delete it and start a new file.
[Snipped[

Just a thought.
Steve, K8SJM
Steve:

In recent years I've got into the regular habit of renaming my ALL.TXT file, using a format of 'ALL-TXT for 2022-April.txt', at the end of each month so that a new file is automatically created by WSJT-X when the app is next started, typically at the start of the next month. This ensures that I've got the full historical record of all my WSJT-X Tx'ing and Rx'ing that can be searched whenever I need to check, for example, why a QSO wasn't confirmed in LoTW or eQSL.

Also, I use an app called 'EditPad Lite' for opening and editing all my .TXT and .ADI files; it doesn't baulk at opening a legacy ALL.TXT file that's over 200Mbytes in size so it might be worth seeing if it'll handle your 870Mb file.

--
Martin G0HDB


Tim Dawson
 

Myself, I really don't care . . . don't use bad editing tools and the size becomes irrelevant, or, worst case, use something like split to break it up as (if) needed . . .

On May 30, 2022 4:21:02 PM CDT, "Michael Black via groups.io" <mdblack98@...> wrote:
I think the log file should be done by month with an appropriate name format that shows the month.
e.g. 202205_ALL.TXT
Mike W9MDB




On Monday, May 30, 2022, 04:19:16 PM CDT, Steve Miller <k8sjm@...> wrote:

Could WSJT-X be coded so that a new ALL.TXT file is created new when a threshold of number of records is exceeded? My current ALL.TXT file is too large for NotePad++ to open (870mb) so it is basically a throw away; I'll have to delete it and start a new file.

If the number of records threshold was implemented, at the start of WSJT-X - check if the ALL.TXT file exceeds the records threshold. If so, grab the last record in the file - get the date and time stamp of that last record and name change the current ALL.TXT to ALL_20220530_182530.TXT or similar. Then start a fresh ALL.TXT file again.

Just a thought.

Steve, K8SJM










--
Sent from my Android device with K-9 Mail. Please excuse my brevity.


Joe Subich, W4TV
 

On 2022-05-30 5:21 PM, Michael Black via groups.io wrote:
I think the log file should be done by month with an appropriate name
format that shows the month. e.g. 202205_ALL.TXT
Exactly what one of the derivative programs does ... much easier to
search.

73,

... Joe, W4TV


On 2022-05-30 5:21 PM, Michael Black via groups.io wrote:
I think the log file should be done by month with an appropriate name format that shows the month.
e.g. 202205_ALL.TXT
Mike W9MDB
On Monday, May 30, 2022, 04:19:16 PM CDT, Steve Miller <k8sjm@...> wrote:
Could WSJT-X be coded so that a new ALL.TXT file is created new when a threshold of number of records is exceeded? My current ALL.TXT file is too large for NotePad++ to open (870mb) so it is basically a throw away; I'll have to delete it and start a new file.
If the number of records threshold was implemented, at the start of WSJT-X - check if the ALL.TXT file exceeds the records threshold. If so, grab the last record in the file - get the date and time stamp of that last record and name change the current ALL.TXT to ALL_20220530_182530.TXT or similar. Then start a fresh ALL.TXT file again.
Just a thought.
Steve, K8SJM


Chuck Moore <wd4hxg@...>
 

FIle editors such as 'Ultra-Edit' and others are available which will open files larger than what Notepad can handle. Ultra-Edit has a 30 day trial period and thenyou have to pay for a license. Others are shareware.After reading multiple messages on this, I decided toperform a closeout of the WSJT-X file by renaming it monthly and allowing WSJT-X to start a new AllText file.WD4HXG ChuckOn May 30, 2022, at 8:21 PM, Martin G0HDB <marting0hdb@...> wrote:On Mon, May 30, 2022 at 10:19 PM, Steve Miller wrote:Could WSJT-X be coded so that a new ALL.TXT file is created new when athreshold of number of records is exceeded? My current ALL.TXT file is toolarge for NotePad++ to open (870mb) so it is basically a throw away; I'll haveto delete it and start a new file.[Snipped[Just a thought.Steve, K8SJMSteve:In recent years I've got into the regular habit of renaming my ALL.TXT file, using a format of 'ALL-TXT for 2022-April.txt', at the end of each month so that a new file is automatically created by WSJT-X when the app is next started, typically at the start of the next month. This ensures that I've got the full historical record of all my WSJT-X Tx'ing and Rx'ing that can be searched whenever I need to check, for example, why a QSO wasn't confirmed in LoTW or eQSL.Also, I use an app called 'EditPad Lite' for opening and editing all my .TXT and .ADI files; it doesn't baulk at opening a legacy ALL.TXT file that's over 200Mbytes in size so it might be worth seeing if it'll handle your 870Mb file.-- Martin G0HDB


Reino Talarmo
 

at the end of each month so that a new file is automatically created by WSJT-X when the app is next started, typically at the start of the next month.
Hi Martin,

I am running an antenna experiment and record ALL.txt 24/7. I just rename ALL.txt in fly e.g. to ALL-220531_test1.txt. At the next decoding instance wsjt-x generates a new ALL.txt. I have not found any collision ever, keep my fingers crossed!

Hi Steve,

If you still have that huge (870mb) file in your computer and want to recover it in smaller junks, then you could make a simple Python program to split it by the number of row into smaller files. I did it by reading the big file line by line so very little memory was used.

73, Reino OH3mA


Alan G4ZFQ
 

On 30/05/2022 18:50, Steve Miller wrote:
Could WSJT-X be coded so that a new ALL.TXT file is created new when a threshold of number of records is exceeded? My current ALL.TXT file is too large for NotePad++ to open (870mb) so it is basically a throw away; I'll have to delete it and start a new file.

There are several ways the wsjtx_syslog.log file may be handled. This is personalised by a wsjtx_log_config.ini file.
Maybe the ALL_TXT file could be handled this way?

Several files in the log directory can get uneccesarily big.

Note the 870MB file can be opened and copy/pasted from another editor.
Maybe this one, I'm not sure if it has a size limit? https://groups.io/g/ProgramsByW2JDB
Some listed here https://stackoverflow.com/questions/159521/text-editor-to-open-big-giant-huge-large-text-files, Google finds a lot .
Maybe someone knows a good one. UltraEdit offers a 30 days trial. Notepad++ is free.

73 Alan G4ZFQ


Bobby Chandler
 

I use Editpad Lite to reduce the file size. It allows me to cut all lines that DON't have my call. If my file gets to 30MB or so, I run it and cut it down.

Bobby/N4AU

--
n4au@...
n4au@...


Sam Birnbaum
 

Hi Steve,
The program that I wrote (and is free) AllText.exe can process any size All.Txt file.It does not load the file into memory. Given the callsigns and the date of the qsoin question it will find the records in less than 0.5 seconds regardless of the sizeof the file. The program can be run even while wsjt-x is actively adding recordsto the all.txt file. 
It is a Windows only executable and is available for free at Programsbyw2jdb@... just need to subscribe, go to files and download it.  
All programs found at ProgramsByW2JDB@groups.io are free.
73,  

Sam W2JDB

-----Original Message-----
From: Steve Miller <k8sjm@...>
To: main@WSJTX.groups.io <main@WSJTX.groups.io>
Sent: Mon, May 30, 2022 2:50 pm
Subject: [WSJTX] #logging Possible Enhancement for the ALL.TXT file

Could WSJT-X be coded so that a new ALL.TXT file is created new when a threshold of number of records is exceeded? My current ALL.TXT file is too large for NotePad++ to open (870mb) so it is basically a throw away; I'll have to delete it and start a new file.

If the number of records threshold was implemented, at the start of WSJT-X - check if the ALL.TXT file exceeds the records threshold. If so, grab the last record in the file - get the date and time stamp of that last record and name change the current ALL.TXT to ALL_20220530_182530.TXT or similar. Then start a fresh ALL.TXT file again.

Just a thought.

Steve, K8SJM


Stu Holzer (K8ST)
 

Maybe there could be an option to just turn it off the ALL file if it isn't wanted or needed ...


Karza
 

On 1.6.2022 14.19, Stu Holzer (K8ST) wrote:
Maybe there could be an option to just turn it off the ALL file if it isn't wanted or needed ...
If you are on Linux, you can accomplish  this by linking  ALL.TXT to the  'bit bucket' /dev/null

73's de Kari, oh2gqc


Steve Miller
 

Thank you all for the responses to my enhancement post.
After reading some of the responses, I can refine my request to these 3 options in regards to the ALL.TXT file.

1) Write all activity as it currently does today. Manually manage the file for historical purposes.
2) Write only my calls 'send' and 'receive' activity to the file. File size would never be an issue based on your own call activity.
3) Don't create or write to this file at all.

Thoughts?


Joe Subich, W4TV
 

Not a good idea. I've several times gone to ALL.TXT to confirm a
QSO when the "final 73" etc. was lost in QRM/fade, etc. I'll get
a card that's not in my log, check ALL.TXT and find both reports
are there ... good QSO.

73,

... Joe, W4TV

On 2022-06-01 11:06 AM, Karza wrote:
On 1.6.2022 14.19, Stu Holzer (K8ST) wrote:
Maybe there could be an option to just turn it off the ALL file if it isn't wanted or needed ...
If you are on Linux, you can accomplish  this by linking  ALL.TXT to the  'bit bucket' /dev/null
73's de Kari, oh2gqc


Joe Subich, W4TV
 

On 2022-06-01 1:26 PM, Steve Miller wrote:

Thoughts?
I would recommend an option to duplicate the behavior of one of the
derivative programs ... start a new ALL.TXT (ALL_YYYY-MM-DD.txt) file
at 00:00z on the first of each month.

Not only does that keep the size of each file generally manageable with
Notebook (no special software), it makes finding a particular callsign
or time period much easier.

73,

... Joe, W4TV


On 2022-06-01 1:26 PM, Steve Miller wrote:
Thank you all for the responses to my enhancement post.
After reading some of the responses, I can refine my request to these 3 options in regards to the ALL.TXT file.
1) Write all activity as it currently does today. Manually manage the file for historical purposes.
2) Write only my calls 'send' and 'receive' activity to the file. File size would never be an issue based on your own call activity.
3) Don't create or write to this file at all.
Thoughts?


Dave Garber
 

so in other words, make it do what jtdx does??? serching old all texts
become very difficult

I find it easier to just edit with notepad, keep current month or 2 in
all.txt, but cut and paste older info to a renamed all.tx, as in "all
01-01-22 - 06-01-22.txt"

if the lost log entry has a date, I can find which txt file to search


Dave Garber
VE3WEJ / VE3IE

On Thu, Jun 2, 2022 at 1:28 AM Joe Subich, W4TV <lists@...> wrote:

On 2022-06-01 1:26 PM, Steve Miller wrote:
>
> Thoughts?

I would recommend an option to duplicate the behavior of one of the
derivative programs ... start a new ALL.TXT (ALL_YYYY-MM-DD.txt) file
at 00:00z on the first of each month.

Not only does that keep the size of each file generally manageable with
Notebook (no special software), it makes finding a particular callsign
or time period much easier.

73,

... Joe, W4TV


On 2022-06-01 1:26 PM, Steve Miller wrote:
Thank you all for the responses to my enhancement post.
After reading some of the responses, I can refine my request to these 3
options in regards to the ALL.TXT file.

1) Write all activity as it currently does today. Manually manage the
file for historical purposes.
2) Write only my calls 'send' and 'receive' activity to the file. File
size would never be an issue based on your own call activity.
3) Don't create or write to this file at all.

Thoughts?






Herb Blue - WB8ASI
 

Steve, I"ve gotten in the habit of using Notebook+ about each month to strip out just the lines in the ALL.TXT file that contain my callsign. I then paste this condensed file into an Annual collection of these monthly condensed files, and have it saved elsewhere on my computer. Erase everything in the ALL.TXT file and start fresh for the new month. FIle size is no problem, and much easier for do lookups. Only takes a couple minutes of cut and paste. Herb WB8ASI


Eugene Morgan
 

Given the various opinions on option for the ALL.TXT file I'd suggest that the developers leave the handling of the ALL.TXT file as it is and focus on more important aspects of the program.

For my part I just wrote a very simple program that will extract all instances of my call sign from the ALL.TXT file and writes those lines to another file. In developing this simple utility I have a couple of questions. Assume I'm writing this program for you:

1. Do you want to also include those lines where you are calling CQ? Yes or No?
2. Would you want it to automatically over write the existing ALL.TXT file with a new version of the ALL.TXT file that includes only those lines where your call is found? Yes or No?

My thinking is that if your ALL.TXT only included lines with your call sign you could reduce the size of the ALL.TXT file considerably. Yes you would need to run this utility manually once in a while. There's no configuration file, no registry entry. It can automatically locate the ALL.TXT file in the WSJT-X folder and it would be for Windows only.

Gene


d_ziolkowski
 

Every couple months I go to the ALL.txt file and rename it date_ALL.txt.
Then I move it to an archive_ALL folder. A new ALL.txt file is
automatically generated.

The hardest part about this procedure is remembering to do it.
I'm sure a script/bat file triggered by CRON/win_task-scheduler, can do
this,but I never got around to it.

Thanks Dan KC2STA

On Thu, Jun 2, 2022 at 9:28 AM Dave Garber <ve3wej@...> wrote:

so in other words, make it do what jtdx does??? serching old all texts
become very difficult

I find it easier to just edit with notepad, keep current month or 2 in
all.txt, but cut and paste older info to a renamed all.tx, as in "all
01-01-22 - 06-01-22.txt"

if the lost log entry has a date, I can find which txt file to search


Dave Garber
VE3WEJ / VE3IE


On Thu, Jun 2, 2022 at 1:28 AM Joe Subich, W4TV <lists@...> wrote:

On 2022-06-01 1:26 PM, Steve Miller wrote:
>
> Thoughts?

I would recommend an option to duplicate the behavior of one of the
derivative programs ... start a new ALL.TXT (ALL_YYYY-MM-DD.txt) file
at 00:00z on the first of each month.

Not only does that keep the size of each file generally manageable with
Notebook (no special software), it makes finding a particular callsign
or time period much easier.

73,

... Joe, W4TV


On 2022-06-01 1:26 PM, Steve Miller wrote:
Thank you all for the responses to my enhancement post.
After reading some of the responses, I can refine my request to these 3
options in regards to the ALL.TXT file.

1) Write all activity as it currently does today. Manually manage the
file for historical purposes.
2) Write only my calls 'send' and 'receive' activity to the file. File
size would never be an issue based on your own call activity.
3) Don't create or write to this file at all.

Thoughts?









--
Dan Ziolkowski KC2STA
SKCC #4290T
Ubuntu LINUX