TDM_Log Readme.txt

INTRODUCTION

The purpose of this module is to help a developer discover why records are being mysteriously deleted or overwritten. It does this by logging the key fields and other important information when a specific datafile is saved or deleted. This solution is only suitable for character-mode DataFlex.

This solution falls under the category of "desperate times call for desperate measures". You really should only do this if you have exhausted every other avenue available to solve the problem. The good news is that every time I've used this technique, I've been able to find the problem as soon as a user caused the problem to occur.

The overall scheme of this technique is to insert some code into an FMAC macro that saves and deletes records The code will log each time a record is created or updated for a specific file. Because the code is inserted at the macro level, every source file, module, package, and include file anywhere in the application that uses SAVE, SAVERECORD and DELETE on the monitored file will have the transaction logged.

This solution is compatible with every known technique for saving and deleting records:

• Save filename
• Save (integer)
• Save indirect_file.recnum
• SaveRecord filename
• SaveRecord (integer)
• SaveRecord indirect_file.recnum
• Delete filename
• Delete (integer)
• Delete indirect_file.recnum

Other macros, such as the ENTER macro, will need to be slightly modified to log saves in those macros too. If none of your programs use an ENTER macro, then you can skip that part.

This solution is designed to require a minimum number of changes to the source code. Three lines are required to be inserted at the top of each file. The lines are simple enough that a DataFlex program could be written to do this insertion, or if you're handy with PERL, AWK or your editor's macro language, you could use any of those techniques to insert those lines too.

WHAT IS THE RESULT?

The log file will contain a list of SAVE and DELETE transactions that occurred in chronological order. An example of what the log will look like is:

invoice, S, 01/27/2004, 1315, 4115, jim, 12345, 5, 1500

In the above line, 'invoice' is the name of the program that saved ('S') the record #1500 on 01/27/2004 at 1:13pm (1315). The user's name was jim and the save occurred in the invoice program on line 4115 (see invoice.prn to correlate this number with an actual source code line number). The record affected (its key fields) are 12345 and 5.

Later, if users complain that a record with a key 12345-5 is missing, you can search for it in the log file and find the above record. Now that you know it was saved to record 1500, you can then search for other transactions that occurred to record 1500. You might find this line:

invmaint, D, 01/27/2004, 1317, 1277, sally, 12345, 5, 1500

It looks like Sally accidentally or otherwise deleted that record at 1:17pm (1317) in the invmaint program. The exact line number that caused the deletion is line number 1277. You can use the INVMAINT.PRN file to find out which source line was actually involved with the deletion. Mystery solved.

Or maybe you'll see:

inv_lkup, S, 01/28/2004, 1015, 5523, joe, 12377, 2, 1500

That line says that the inv_lkup program overwrote record 1500 with line item 12377-2 at 10:15am the next day. I don't know why a lookup program would be doing that, but line 5523 in the INV_LKUP.PRN file is the place to start trying to figure out how it happened.

The key points here are that once you know exactly which record stored the now missing line item, you can then find out what other programs edited that record and which one deleted or overwrote it.

INSTALLATION:

First, make a backup of your flex.cfl, FMAC. and filelist.cfg files. The FMAC. file is a text file (source file) that contains the definitions of all of the DataFlex commands. The flex.cfl file is the precompiled version of the FMAC. file.

Next, make a backup of all your source and *.flx files. You will be recompiling every program throughout your application that opens the datafile you want to monitor.

Create a new datafile called TDM_LOG using the TDM_LOG.DEF file provided in the .zip file.

The schema for the TDM_LOG file is:

After you've created the TDM_LOG file, modify the KEY1 and KEY2 fields to match the data type and length of the key fields in the file you want to monitor. If the file you want to monitor has more than 2 key fields, just add them in using the same pattern as above. If the file only has one key field, you can just leave KEY2 alone.

The log file doesn't need any indexes. Because of that, you don't need to worry about the maximum number of records setting.

CHANGING THE FMAC

Open your FMAC file using your favorite editor.

In DataFlex 3.2, there are five commands that save a record: SAVE, SAVERECORD, API_SAVE, OLD_SAVERECORD and OLD_SAVE. I'll assume that you aren't using OLD_SAVERECORD or OLD_SAVE. If you are, then you'll need to modify those macros as well.

SAVE and SAVERECORD use API_SAVE and API_SAVERECORD respectively. API_SAVE uses API_SAVERECORD (by using SAVERECORD as a surrogate). API_SAVERECORD uses a command called VARIABLE_FILE_COMMAND. That's the macro that you will actually modify.

In the VARIABLE_FILE_COMMAND macro, add this line at the top of the macro:

TDM_DELETE_LOG !1 !2

and add this line at the bottom of the macro:

TDM_SAVE_LOG !1 !2

The VARIABLE_FILE_COMMAND macro should look like:

#COMMAND VARIABLE_FILE_COMMAND
  TDM_DELETE_LOG !1 !2
  // This supports both Indirect_file and Indirect_file.Recnum (JJT)
  #IFSAME !2.RECNUM |FS0,0& |FN0,0& |FD0,0& |FS0,0&.RECNUM
    !A [] !1 |CI0 !3
  #ELSE
    #IFCLASS !2 SNDIECWVF
        !A [] !1 !2 !3
    #ELSE
        DECLARE_DATAFILE !2
        !A [] !1 !2.FILE_NUMBER !3
    #ENDIF
  #ENDIF
  TDM_SAVE_LOG !1 !2
#ENDCOMMAND

In the ENTER macro, there is one location that needs to be changed. Find the lines in the ENTER macro that look like:

//[NOT ERR] DELETE !1 // delete records
!A [~|127] $C3 !1.RECNUM

Uncomment the first line and DELETE the second line so that they look like this:

[NOT ERR] DELETE !1 // delete records

You need to delete the second line because it contains a !A. The !A will cause the compiler to go bonkers when it's inside of a comment, so you must delete the line to prevent any compile-time problems.

Save the FMAC file and pack it using dfpack:

DFPACK FMAC.

This will generate a new flex.cfl file. You can verify that it worked by checking the time/date stamp on your flex.cfl file.

TESTING THIS SOLUTION

A sample program, along with the expected results, is available in the TESTING subdirectory included in the .ZIP file that came with this module. Once you've modified the FMAC and installed the TEST1 datafile into your filelist.cfg, you should be able to compile and run the t1.src program without errors using the TDM_LOG.PKG provided in the distribution .ZIP file. The distribution TDM_LOG.PKG assumes that TEST1 is installed as datafile #42. If you decide to install TEST1 someplace else, you will need to follow the below instructions for changing TDM_LOG.PKG.

CHANGING OTHER MACRO FILES:

If there are custom macros in your project, you might need to modify some of those files as well. Look for any macro lines that use $C2 (save) or $C3 (delete) and change those lines to use the new SAVE and DELETE commands.

GET FILE INFORMATION:

Next, use your favorite programming editor to open the .FD file for the datafile you want to monitor. If the datafile's name is LINEITEM, you'll want to open lineitem.fd.

Look for the definitions of the datafile's key fields. They will look something like:

#REPLACE LINEITEM.HEADER |FS142,1 #REPLACE LINEITEM.LINENUM |FN142,2

The first line says that LINEITEM.HEADER is defined as an ASCII type (|FS), that it is file number 142 and that it's the first field in the datafile.

MODIFY TDM_LOG.PKG:

Once you've located the key field information, open the TDM_LOG.PKG file.

Search for "CHANGE-THIS-LINE" inside of the TDM_LOG.PKG file. There are four lines that need to be changed.

The first line looks like:

DEFINE TDM_TARGET_FILE FOR 42 // CHANGE-THIS-LINE

Change the 42 in that line to the file number of the file you want to monitor. Using the above example for lineitem, you would change 42 to 142.

The next line that needs to be changed looks like:

[TDM_LOG_THIS] MOVE "XXX" TO TDM_LOG.USER // CHANGE-THIS-LINE

This line records the user's identification to the log file. If you don't have user IDs or you don't care about recording that, you can replace "XXX" with "". A typical way this line would be changed might look like:

[TDM_LOG_THIS] GET_ENVIRONMENT "USER_ID" TO TDM_LOG.USER

The important point here is that however you get the user's ID, it must be guarded with the [TDM_LOG_THIS] indicator to prevent that line from executing when it doesn't need to execute. If you're not comfortable with macro writing, then just leave this line alone and don't worry about trying to get the user ID. It's more important to find where the problem is occurring than finding out who to blame. Also, modifying a macro with anything more complicated than the above two examples can be a very tricky business unless you really know what you're doing.

The next line that needs to be changed looks like:

[TDM_LOG_THIS] MOVE |FS42,1 TO TDM_LOG.KEY1 // CHANGE-THIS-LINE

In that line, change the |FS42,1 to whatever your first key field is. The above example is |FS142,1 so you'll change the line to look like:

[TDM_LOG_THIS] MOVE |FS142,1 TO TDM_LOG.KEY1 // CHANGE-THIS-LINE

Similarly, the next line that needs to be changed looks like

[TDM_LOG_THIS] MOVE |FS42,2 TO TDM_LOG.KEY2 // CHANGE-THIS-LINE

and you'll make a similar change of |FS42,2 to |FS142,2. If you only had one key field, you will need to comment the above line so that the line will still be there when you want to monitor a different datafile.

If you have additional key fields, follow the above pattern for as many key fields as you have.

Finally, the last line to change is:

[TDM_LOG_THIS] MOVE |FN42,0 TO TDM_LOG.REC // CHANGE-THIS-LINE

Change only the file number (42 in this example) to whatever file number you used in the first change. Using the above example, you will change this line to:

[TDM_LOG_THIS] MOVE |FN142,0 TO TDM_LOG.REC // CHANGE-THIS-LINE

because 142 is the file number for the lineitem file.

Save the TDM_LOG.PKG file.

MODIFY SOURCE FILES:

At the top of every source file, add these three lines. These lines can appear before any image definitions, so you can just blindly add them at the top of every program:

#REPLACE TDM_PROGRAM_NAME "invoice" 
USE TDM_LOG.PKG 
OPEN tdm_log

The first line has the name of the program and it's the only line that will be different for every program. The other two lines, the USE and OPEN lines, will be the same in every program.

Recompile all of your programs and test them. As you save and delete records in the file that's being monitored, you should see lines being added to the TDM_LOG datafile.

There will be a slight performance penalty when using this technique, even in programs that are recompiled and don't even open the file being monitored. This is because every save needs to check if it's being monitored or not.

UNINSTALLING:

Restore the backed-up SRC and FLX files. Restore the backed-up filelist.cfg, fmac and flex.cfl files too. Delete the TDM_LOG.* files.

LIMITATIONS:

This solution is not compatible with alias files. If the file has an alias file that is also making saves and deletes, this solution can be modified to handle it. I've personally never encountered that problem so I've never had to code it.

This solution assumes that the program doesn't make use of clever techniques that set the file-mode of all opened files to be read-only. In particular, the TDM_LOG file is opened at the top of the program and it must be writable whenever a record in the monitored datafile is saved.

CONCLUSION:

I hope you find this solution helpful in solving what would otherwise be an almost impossible problem to find. If you do successfully find a problem using this technique, drop me an email and let me know.

Return to Log All Save and Delete Transactions Article

Curtis Krauskopf is a software engineer and the president of The Database Managers (www.decompile.com). He has been writing code professionally for over 25 years. His prior projects include multiple web e-commerce applications, decompilers for the DataFlex language, aircraft simulators, an automated Y2K conversion program for over 3,000,000 compiled DataFlex programs, and inventory control projects. Curtis has spoken at many domestic and international DataFlex developer conferences and has been published in FlexLines Online, JavaPro Magazine, C/C++ Users Journal and C++ Builder Developer's Journal.

Other Popular DataFlex topics at The Database Managers, Inc.:

• DataFlex Programming Tips
• DataFlex Freeware
• Timing an Event in DataFlex (article)
• All About Flexkeys
• WebApp Server Programming Tips
• Financial application in DataFlex
• Links to other DataFlex websites
• Log all Save and Delete Transactions